A 3D point cloud semantic segmentation method for large scene and multi-target environment

Abstract

This invention discloses a 3D point cloud semantic segmentation method for large-scale, multi-object environments. The method includes: searching the original input point cloud using a K-nearest neighbor search operator; encoding the obtained bilateral input features using local feature coding blocks; processing the fused bilateral encoded features using an attention pooling module; adding and fusing the input features and the obtained local aggregated features to obtain output features; upsampling the output features of the coding layers at all resolutions using a decoder with an embedded multi-scale feature fusion module, and then fusing the upsampled features at other resolutions to obtain a feature representation with the same resolution as the original input point cloud; and converting the feature representation with the same resolution as the input point cloud into a semantic segmentation result using a segmentation prediction head and outputting it. This invention overcomes the limitations of existing technologies in terms of local semantic feature representation capabilities and cross-scale information fusion capabilities, achieving accurate and efficient semantic segmentation of 3D point clouds in large-scale, multi-object environments.

Description

Technical Field

[0001] This invention belongs to the fields of geographic science and computer vision technology, and relates to a 3D point cloud semantic segmentation method for large-scale, multi-target environments. Background Technology

[0002] With the rapid development of computer vision technology, the demand for processing and analyzing 3D data is growing rapidly. Point cloud data, as an important form of 3D data, shows broad application prospects in fields such as autonomous driving, virtual reality, and robot navigation. In these applications, semantic segmentation of point cloud data plays a crucial role, enabling precise differentiation and identification of different objects or regions within the point cloud data, providing detailed information descriptions of the 3D environment.

[0003] 3D point cloud semantic segmentation is an important research direction in the field of computer vision, aiming to classify points or regions in a point cloud into different semantic categories. Although significant progress has been made in semantic segmentation techniques for 2D images, semantic segmentation of 3D point cloud data still faces many challenges, such as data sparsity, noise, and irregularity.

[0004] Large-scale point clouds, composed of millions or even billions of points collected from the real world, are extremely complex, which greatly limits the development of segmentation techniques for large-scale complex scene point clouds. Currently, most 3D point cloud semantic segmentation methods are derived from image semantic segmentation techniques, but due to limitations in the ability to represent local semantic features and fuse cross-scale information, these methods are inadequate when dealing with large scenes and multi-object environments. Summary of the Invention

[0005] The technical problem to be solved by this invention is to overcome the limitations of the local semantic feature representation capability and cross-scale information fusion capability in the existing technology, and to achieve accurate and efficient semantic segmentation of 3D point clouds in large-scale and multi-target environments.

[0006] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution.

[0007] This invention provides a 3D point cloud semantic segmentation method for large-scale, multi-object environments, comprising: To acquire point clouds, the K-nearest neighbor (KNN) search operator of each coding layer of the encoder is used to search the feature space and geometric space of the point cloud of this coding layer. This yields the neighborhood semantic feature representation of the point cloud and the geometric information ignoring the absolute coordinates of the neighborhood, which are used as the bilateral input features of each coding layer. By utilizing the local feature encoding blocks of the local feature learner of each encoding layer of the encoder, the bilateral input features are feature-encoded to obtain bilateral encoded features. The bilateral encoded features are fused through feature addition operations; By utilizing the attention pooling module of the encoder's local feature learner, the fused bilateral encoded features are processed to obtain the local aggregated features of the point cloud in this encoding layer. The semantic features of the bilateral input features and the local aggregated features are added and fused using the residual connection module of the encoder's local feature learner to obtain the output features; Among them, the point cloud of the first coding layer is the original input point cloud, and the point clouds of the other coding layers are obtained by downsampling the geometric space of the point cloud of the upper coding layer and the feature space of the output features of the upper coding layer using farthest point sampling (FPS). After obtaining the output features of all coding layers, the decoder with embedded multi-scale feature fusion module (MFF) upsamples the output features of coding layers at all resolutions, and simultaneously fuses the upsampled features from coding layers at other resolutions layer by layer to obtain a feature representation with the same resolution as the original input point cloud. The feature representation is converted into a semantic segmentation result using a segmentation prediction head and then output.

[0008] Furthermore, each coding layer of the encoder includes only the K-nearest neighbor search operator KNN and the local feature learner w-LFL.

[0009] Furthermore, the local feature learner w-LFL is composed of a local feature encoding block, an attention pooling module, and a residual connection module connected together; The local feature coding block is used to encode the neighborhood semantic feature representation and the geometric information ignoring the absolute coordinates of the neighborhood of the point cloud respectively to obtain bilateral coding features, and to fuse the bilateral coding features to obtain fused coding features. The attention pooling module is used to discriminately aggregate the fused bilateral coding features to obtain the local aggregated features of the point cloud of this coding layer. The residual connection module is used to add and fuse the bilateral input features of the local feature learner w-LFL and the local aggregated features.

[0010] Furthermore, by utilizing the local feature encoding blocks of the local feature learners in each encoding layer of the encoder, feature encoding is performed on the bilateral input features to obtain bilateral encoded features, including: Ignoring the absolute coordinates of the K-nearest neighbors, we use only relative coordinates and relative distances to represent local geometric relationships. Obtain geometric feature information encoding ; semantic features of K nearest neighbors To capture the neighborhood center point s of the point cloud i Surrounding local semantic relations Obtain local semantic feature information encoding ; By encoding geometric feature information and local semantic feature information, bilateral encoded features containing different implicit information are obtained. .

[0011] The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 4 is characterized in that the local geometric relations , is represented as: (1); In the formula, Concat represents a cascading operation. Represents the neighborhood center point s of the point cloud i Spatial coordinate information, Represents the neighborhood center point s of the point cloud i The spatial coordinates of the k-th neighboring point in the neighborhood, where i represents the sort number of each point in the point cloud, and k represents the sort number of the neighboring point in the neighborhood of the point cloud. Represents the 2-norm operator; The geometric feature information encoding , is represented as: (2); In the formula, MLP() represents the multilayer perceptron function. Representing local geometric relationships; The local semantic relations , is represented as: (3); In the formula, Concat represents a cascading operation, i represents the sorting number of each point in the point cloud, and k represents the sorting number of the neighboring points in the point cloud's neighborhood. Represents the neighborhood center point s of the point cloud i semantic feature information, Represents the neighborhood center point s of the point cloud i The semantic feature information of the k-th neighboring point in the neighborhood. Represents a scalar computation operator; The local semantic feature information encoding , is represented as: (4); In the formula, MLP() represents the multilayer perceptron function. It represents local semantic relationships.

[0012] Furthermore, the bilateral encoded features are fused through feature addition, as shown below: (5); In the formula, ⊕ represents the characteristic addition operation. This represents the fused bilateral encoded features. Represents geometrically encoded feature information. It represents local semantic encoding feature information.

[0013] Furthermore, the attention pooling module of the encoder's local feature learner is used to process the fused bilateral encoded features to obtain the local aggregated features of the point cloud in this encoding layer, including: From the fused bilateral encoded features using shared MLP and softmax functions Learning attention weight matrix ; Through the attention weight matrix The fused bilateral coding features By performing weighted summation, the local aggregated features of the point cloud in this coding layer are obtained.

[0014] Furthermore, the attention weight matrix , is represented as: (6); In the formula, softmax() represents the normalized exponential function, and MLP() represents the multilayer perceptron function. This represents the fused bilateral encoded features; Furthermore, the local aggregation feature is represented as: (7); In the formula, Sum() represents the summation function. This represents the attention weight matrix, and ° denotes the element-wise product of the matrices. This represents the fused bilateral coding features.

[0015] Furthermore, after obtaining the output features of all coding layers, the decoder with embedded multi-scale feature fusion (MFF) module upsamples the output features of the coding layers at all resolutions. Simultaneously, it fuses upsampled features from coding layers at other resolutions layer by layer to obtain a feature representation with the same resolution as the original input point cloud, including: Suppose the encoder has J layers, and for the j-th coding layer E j Output characteristics Upsampling is performed, and upsampled features from other resolution coding layers are fused to obtain the output features. In decoding layer D k Fusion features The obtained fusion features The process is represented as: (8); In the formula, j and k represent the numbers of the encoding layer and the decoding layer, respectively, and E j Dk These represent the j-th encoding layer and the k-th decoding layer, respectively, and ⊕ represents the feature addition operation. Indicates coding layer E j The output features, f D express Continuous upsampling to the decoding layer D k The input features at time, where k = j, j-1, ..., 1. Indicates the upsampling operator, Indicated in decoding layer D k Upper coding layer E j-1 ~E k The features obtained by upsampling and fusing the output features.

[0016] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This invention constructs an efficient encoder capable of learning and aggregating information-rich local contextual features from point clouds. Each coding layer of the encoder consists of a K-nearest neighbor search operator and a local feature learner that does not rely on strict geometric relationships. A multi-scale feature fusion module (MFF) is designed in the decoder, which can effectively fuse multi-scale coding layer features at all resolutions across all decoder layers, providing an efficient representation of variable-scale targets. Finally, semantic segmentation results are obtained through a segmentation prediction head. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the network architecture of a 3D point cloud semantic segmentation method for large-scale, multi-target environments provided in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the structure of the local feature learner provided in an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of the structure of the multi-scale feature fusion module (MFF) provided in an embodiment of the present invention;

[0020] Figure 4 This is an example diagram comparing the semantic segmentation results of the method provided by this invention with those of currently popular semantic segmentation methods on the indoor scene point cloud dataset S3DIS, as provided in this embodiment of the invention.

[0021] Figure 5 This is an example of a comparison between the semantic segmentation results of the method provided by this invention and currently popular semantic segmentation methods on the Toronto3D point cloud dataset of urban road scenes, provided by an embodiment of this invention.

[0022] Figure 6 This is an example image of semantic segmentation results on the Semantic KITTI point cloud dataset for autonomous driving scenarios, provided by an embodiment of the present invention.

[0023] Figure 7 This is a schematic diagram showing the semantic segmentation results of this method and other advanced semantic segmentation methods on the Area5 test set of S3DIS, based on the indoor scene point cloud dataset S3DIS provided in this embodiment of the invention.

[0024] Figure 8 This is a schematic diagram of the semantic segmentation results generated by the method provided in this invention and some currently popular methods, based on the Toronto3D point cloud dataset of urban road scenes.

[0025] Figure 9 This is a schematic diagram showing the semantic segmentation results of this method and other advanced semantic segmentation methods on the S3DIS test set Area5, based on the Semantic KITTI point cloud dataset for autonomous driving scenarios provided in this embodiment of the invention.

[0026] Figure 10 This is a schematic diagram showing the complexity comparison between the method provided by this invention and some recently popular semantic segmentation methods, based on the indoor scene point cloud dataset S3DIS. Detailed Implementation

[0027] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0028] Example 1

[0029] like Figure 1 As shown, this embodiment introduces a 3D point cloud semantic segmentation method for large-scale, multi-target environments, including:

[0030] By using the K-nearest neighbor search operator KNN for each coding layer of the encoder, the feature space and geometric space of the point cloud in this coding layer are searched to obtain the neighborhood semantic feature representation of the point cloud and the geometric information ignoring the absolute coordinates of the neighborhood, which are used as the bilateral input features of each coding layer.

[0031] By utilizing the local feature encoding blocks of the local feature learner of each encoding layer of the encoder, the bilateral input features are feature-encoded to obtain bilateral encoded features.

[0032] The bilateral encoded features are fused through feature addition operations;

[0033] By utilizing the attention pooling module of the encoder's local feature learner, the fused bilateral encoded features are processed to obtain the local aggregated features of the point cloud in this encoding layer.

[0034] The semantic features of the bilateral input features and the local aggregated features are added and fused using the residual connection module of the encoder's local feature learner to obtain the output features;

[0035] Among them, the point cloud of the first coding layer is the original input point cloud, and the point clouds of the other coding layers are obtained by downsampling the geometric space of the point cloud of the upper coding layer and the feature space of the output features of the upper coding layer using farthest point sampling (FPS).

[0036] After obtaining the output features of all coding layers, the decoder with embedded multi-scale feature fusion module (MFF) upsamples the output features of coding layers at all resolutions, and simultaneously fuses the upsampled features from coding layers at other resolutions layer by layer to obtain a feature representation with the same resolution as the original input point cloud.

[0037] The feature representation is converted into a semantic segmentation result using a segmentation prediction head and then output.

[0038] A schematic diagram of the w-LFL local feature learner is shown below. Figure 2 As shown, by using the K-nearest neighbor search operator KNN and the local feature learner w-LFL in the encoder, the local geometric relationship and semantic feature information of each point in the point cloud can be accurately captured, ensuring the ability to capture details in high-density point clouds, thereby improving the accuracy of semantic segmentation.

[0039] A schematic diagram of the multi-scale feature fusion module (MFF) structure is shown below. Figure 3 As shown, the multi-scale feature fusion module (MFF) integrates features from different resolution coding layers, enabling it to process targets of different scales simultaneously. It can produce accurate semantic segmentation results for targets of different sizes in the scene, thus improving the semantic segmentation capability for targets of different sizes.

[0040] Through efficient layer-by-layer fusion operations in the decoder stage, semantic segmentation results with the same resolution as the input point cloud can be generated quickly, providing efficient and accurate representations for large-scale scenes.

[0041] The segmentation prediction head transforms the features fused by the decoder into the final semantic segmentation result, predicting a semantic label for each point.

[0042] The present invention provides a 3D point cloud semantic segmentation device for large-scale, multi-target environments, such as autonomous driving, robot navigation, virtual reality and other fields, which provides these applications with accurate 3D environment perception capabilities.

[0043] The local feature learner w-LFL is composed of a local feature encoding block, an attention pooling module, and a residual connection module.

[0044] The local feature encoding block adds and fuses the bilateral input features of the local feature learner w-LFL with the local aggregated features, which can represent local features more richly.

[0045] The attention pooling module discriminately aggregates the fused bilateral coding features to obtain local aggregated features of the point cloud of this coding layer, which can focus on the most discriminative local features.

[0046] The residual connection module adds and fuses the bilateral input features of the local feature learner w-LFL with the local aggregated features, preserving more original information and facilitating gradient propagation during network training.

[0047] Example 2

[0048] This embodiment introduces a 3D point cloud semantic segmentation method for large-scale, multi-object environments. The specific steps include:

[0049] (1) Using the K-nearest neighbor search operator (KNN) of each coding layer of the encoder, the feature space and geometric space of the point cloud of this coding layer are searched to obtain the neighborhood semantic feature representation of the point cloud and the geometric information ignoring the absolute coordinates of the neighborhood, which are used as the bilateral input features of the coding layer:

[0050] For the first coding layer of the encoder, the original input point cloud is input into the first coding layer of the encoder as the point cloud of this coding layer. The K-nearest neighbor search operator KNN of the first coding layer of the encoder is used to search the feature space and geometric space of the point cloud of this coding layer to obtain the neighborhood semantic feature representation of the point cloud and the geometric information ignoring the absolute coordinates of the neighborhood, which are used as the bilateral input features of the first coding layer.

[0051] For the remaining coding layers of the encoder, the point cloud obtained by downsampling the geometric space of the point cloud of the upper coding layer and the feature space of the output features of the upper coding layer using farthest point sampling (FPS) is used as the input point cloud of the remaining coding layers of the encoder. The K-nearest neighbor search operator (KNN) of the remaining coding layers of the encoder is used to search the feature space and geometric space of the point cloud of this coding layer to obtain the neighborhood semantic feature representation of the point cloud and the geometric information ignoring the absolute coordinates of the neighborhood, which is used as the bilateral input features of the remaining coding layers.

[0052] The original input point cloud S ∈ℝ N×(3+d) Includes 3D spatial coordinates 𝒫∈ℝ representing geometric information N×3 And the basic attributes representing feature information, such as RGB information or reflectance intensity. Where, S = {s1, s2, ..., s...}N}, i = 1,⋯, N; N represents the number of point clouds, d represents the dimension of the feature information of the original input point cloud, s1 represents the first point of the original input point cloud, s2 represents the second point of the original input point cloud, s N This represents the Nth point in the original input point cloud, ℝ N×(3+d) Describe the feature space of N×(3+d), ℝ N×3 Represents the geometric space of N×d.

[0053] Initially, the original input point cloud S is processed through a fully connected layer FC to extract pointwise features ℱ∈ℝ in the high-dimensional feature space. N×D , among which, ℝ N×D The space dimension of the high-dimensional feature is represented by N, where N represents the number of point clouds and D represents the dimension of the high-dimensional feature space.

[0054] The K-nearest neighbor (KNN) search operator for each coding layer of the encoder is used to search the feature space and geometric space of the point cloud S of that coding layer to obtain the neighborhood information of the point cloud of that coding layer. As the bilateral input feature of this coding layer, the neighborhood information includes the neighborhood semantic feature representation of the original input point cloud and geometric information ignoring the absolute coordinates of the neighborhood. Represents the geometric feature information of the K nearest neighbors. This represents the local feature information of the K nearest neighbors, where K represents the K nearest neighbors, k represents the number of neighboring points in the neighborhood, and i represents the sorting number of the points in the point cloud.

[0055] (2) Use the local feature coding blocks of the local feature learner of each coding layer of the encoder to perform feature coding on the bilateral input features to obtain bilateral coded features;

[0056] When encoding local geometric information, the absolute coordinates of the K nearest neighbors are ignored, and only relative coordinates and relative distances are used to represent local geometric relationships. Obtain geometric feature information encoding ;

[0057] The local geometric relationship , is represented as:

[0058] (1);

[0059] In the formula, Concat represents a cascading operation. Represents the neighborhood center point s of the point cloud i Spatial coordinate information, Represents the neighborhood center point s of the point cloud i The spatial coordinates of the k-th neighboring point in the neighborhood, where i represents the sort number of each point in the point cloud, and k represents the sort number of the neighboring point in the neighborhood of the point cloud. Represents the 2-norm operator;

[0060] The geometric feature information encoding , is represented as:

[0061] (2);

[0062] In the formula, MLP() represents the multilayer perceptron function. Represents local geometric relationships.

[0063] When encoding local feature information, the semantic features of the K nearest neighbors are used. To capture the neighborhood center point s of the point cloud i Surrounding local semantic relations Obtain local feature information encoding ;

[0064] The local semantic relations , is represented as:

[0065] (3);

[0066] In the formula, Concat represents a cascading operation, i represents the sorting number of each point in the point cloud, and k represents the sorting number of the neighboring points in the point cloud's neighborhood. Represents the neighborhood center point s of the point cloud i semantic feature information, s i The semantic feature information of the k-th neighboring point in the neighborhood. This represents a scalar computation operator, similar to Euclidean distance;

[0067] The local semantic feature information encoding , is represented as:

[0068] (4);

[0069] In the formula, MLP() represents the multilayer perceptron function. It represents local semantic relationships.

[0070] By encoding geometric feature information and local semantic feature information, bilateral encoded features containing different implicit information are obtained. .

[0071] (3) The bilateral encoded features are fused through feature addition, as shown below:

[0072] (5);

[0073] In the formula, ⊕ represents the characteristic addition operation. This represents the fused bilateral encoded features. Represents geometrically encoded feature information. It represents local semantic encoding feature information.

[0074] (4) Utilizing the attention pooling module of the encoder's local feature learner, the fused bilateral encoded features are automatically selected and aggregated to obtain highly representative information, while discarding irrelevant features, thus acquiring locally aggregated features, including:

[0075] From the fused bilateral encoded features using shared MLP and softmax functions Learning attention weight matrix The attention weight matrix , is represented as:

[0076] (6);

[0077] In the formula, softmax() represents the normalized exponential function, and MLP() represents the multilayer perceptron function. This represents the fused bilateral encoded features;

[0078] Through the attention weight matrix The fused bilateral coding features By performing weighted summation, the local aggregated features of the point cloud at this coding layer are obtained, represented as:

[0079] (7);

[0080] In the formula, Sum() represents the summation function. This represents the attention weight matrix, and ° denotes the element-wise product of the matrices. This represents the fused bilateral coding features.

[0081] (5) Using the residual connection module of the encoder local feature learner, the semantic features of the bilateral input features and the local aggregated features are added and fused to obtain the output features;

[0082] (6) After obtaining the output features of all coding layers, the multi-scale feature fusion module (MFF) is used to upsample the output features of all coding layers at all resolutions. At the same time, upsampled features from decoding layers at other resolutions are fused layer by layer to obtain a feature representation with the same resolution as the original input point cloud.

[0083] Suppose the encoder has J layers, and for the j-th coding layer E j Output characteristics Upsampling is performed, and upsampled features from other resolution coding layers are fused to obtain the output features. In decoding layer D kFusion features The obtained fusion features The process is represented as:

[0084] (8);

[0085] In the formula, j and k represent the numbers of the encoding layer and the decoding layer, respectively, and E j D k These represent the j-th encoding layer and the k-th decoding layer, respectively, and ⊕ represents the feature addition operation. Indicates coding layer E j The output features, f D express Continuous upsampling to the decoding layer D k The input features at time, where k = j, j-1, ..., 1. Indicates the upsampling operator, Indicated in decoding layer D k Upper coding layer E j-1 ~E k The features obtained by upsampling and fusing the output features.

[0086] (7) Use the segmentation prediction head to convert the feature representations with the same resolution as the original input point cloud into semantic segmentation results and output them.

[0087] (8) The semantic segmentation results obtained through the above method are used to verify the excellent performance of the 3D point cloud semantic segmentation method for large scenes and multi-target environments provided by this invention in large scenes and multi-target environments:

[0088] Three large-scale point cloud datasets were selected for experiments, including the indoor point cloud dataset S3DIS, the urban road scene point cloud dataset Toronto3D, and the autonomous driving scene point cloud dataset Semantic KITTI.

[0089] Experiment 1: Based on the indoor point cloud dataset S3DIS, the semantic segmentation results of a 3D point cloud semantic segmentation method for large scenes and multi-target environments are compared with the semantic segmentation results of some currently popular methods. This verifies that the 3D point cloud semantic segmentation method for large scenes and multi-target environments provided by this invention still has high performance when dealing with complex point cloud scene segmentation tasks with dense and close-range target objects.

[0090] Experiment 2: Based on the Toronto3D point cloud dataset for urban road scenes, the semantic segmentation results of a 3D point cloud semantic segmentation method for large scenes and multi-object environments are compared with the semantic segmentation results of some currently popular methods. This verifies that the 3D point cloud semantic segmentation method for large scenes and multi-object environments provided by this invention also has significant effectiveness in urban road scenes. In the experiment, the spatial coordinate information and color information of the point cloud are selected as the original input data.

[0091] Experiment 3: Based on the SemanticKITTI point cloud dataset for autonomous driving scenarios, the semantic segmentation results of a 3D point cloud semantic segmentation method for large scenes and multi-objective environments are compared with the semantic segmentation results of some currently popular methods to verify the effectiveness of the 3D point cloud semantic segmentation method for large scenes and multi-objective environments provided by this invention in dealing with complex traffic scenarios of multiple scales and multiple categories.

[0092] The above experimental results are as follows Figure 7-9 As shown.

[0093] Figure 7 This paper quantitatively presents the performance comparison results of the semantic segmentation results generated by the method provided in this invention on the Area5 test set of the S3DIS indoor scene point cloud dataset. The method provided in this invention achieves 66.5% and 88.9% in mIoU and OA metrics, respectively. Compared with the method provided by the BEMF-Net network structure, the method provided in this invention achieves the same mIoU result. Although the method provided in this invention reduces the OA metric by 0.4% compared with the BEMF-Net network structure, the comparison results of model complexity of different methods in Table 4 show that the method provided in this invention surpasses the existing network structure in both model parameter count and inference speed.

[0094] In summary, the method provided in this invention outperforms all current point-based semantic segmentation methods on the indoor scene point cloud dataset S3DIS. Furthermore, Figure 7 This also fully demonstrates that a 3D point cloud semantic segmentation method for large-scale, multi-target environments achieves more balanced IoU results across different scale categories, further proving that the method provided by this invention exhibits stronger adaptability to segmentation of target objects at different scales when dealing with complex point cloud scenes with dense and close-range target objects.

[0095] at the same time Figure 4 The visualization results of the semantic segmentation results generated by the method provided by this invention and the method provided by the RandLA–Net network structure on the indoor scene point cloud dataset S3DIS are also qualitatively demonstrated. Figure 4This demonstrates that the semantic segmentation results produced by the method provided by this invention in these segmentation regions are superior to those produced by the method provided by the RandLA–Net network structure. In particular, the method provided by this invention shows significant improvements in detail when segmenting objects at different scales, such as chairs, tables, and bookshelves in the first row of segmented images; picture frames and windows in the second row; and doors and clutter in the third row. Furthermore, the method provided by this invention still exhibits excellent segmentation performance in complex scenes with multiple closely overlapping objects. For example, in the first scene, although chairs and desks are closely adjacent, a 3D point cloud semantic segmentation method for large scenes and multi-object environments can still accurately distinguish their boundaries; and in the second scene, the fine segmentation performance of the picture frame is also improved. These visualized semantic segmentation results fully demonstrate the superior performance of the method provided by this invention in terms of local detail and multi-scale object segmentation.

[0096] Figure 8 This paper quantitatively presents the semantic segmentation results generated by the method provided in this invention and those generated by some currently popular methods, comparing their performance on the Toronto3D urban road scene point cloud dataset. The method provided in this invention achieves 81.7% and 97.0% accuracy in mIoU and OA metrics, respectively. Compared to... Figure 8 The BAF-LAC network structure, which yields the best mIoU result, is improved by only 0.3% in mIoU by the method provided in this invention, while achieving a 1.8% improvement in OA. Furthermore, the semantic segmentation results produced by the method provided in this invention achieve the highest segmentation accuracy in the Building and Car categories. Overall, the semantic segmentation results produced by the method provided in this invention demonstrate the best performance on the Toronto3D urban road scene point cloud dataset.

[0097] Figure 5 This paper visually demonstrates a comparison of the segmentation performance of a 3D point cloud semantic segmentation method for large-scene, multi-object environments proposed in this invention with that of currently popular methods on the Toronto3D test set. This further verifies the effectiveness and superiority of the 3D point cloud semantic segmentation method for large-scene, multi-object environments proposed in this invention. To more clearly illustrate the segmentation effect, in Figure 4 This demonstrates that the 3D point cloud semantic segmentation method of the present invention, designed for large-scale, multi-object environments, accurately captures local details of objects at different scales, achieving more refined segmentation. For example, details such as zebra crossings and fences connected to buildings, as marked in the image, have been segmented precisely and accurately.

[0098] Figure 9This paper presents the semantic segmentation results of a 3D point cloud semantic segmentation method for large-scale, multi-object environments, comparing them with those of some recent popular methods on the SemanticKITTI point cloud dataset for autonomous driving scenarios. In online submission tests of the semantic segmentation results generated by the method provided by this invention, the method achieved a 59.2% mIoU metric. Compared with other recent popular methods, the method provided by this invention achieves the best IoU performance across numerous categories, demonstrating outstanding performance in both fine segmentation of small-scale objects such as pedestrians, bicycles, and motorcycles, and overall recognition of large-scale objects such as roads, sidewalks, and trucks.

[0099] Figure 6 The semantic segmentation results of the method provided in this invention on the SemanticKITTI point cloud dataset for autonomous driving scenarios are visualized. The visualized segmentation results demonstrate that the method provides more accurate segmentation of objects at different scales in road traffic scenes. For example, in the third image, a 3D point cloud semantic segmentation method for large-scale, multi-object environments exhibits excellent segmentation capabilities, from small vegetation and tree trunks to high-altitude traffic lights and guardrails. In the fourth image, the method also achieves fine and accurate segmentation for small-scale objects such as pedestrians, bicycles, and motorcycles, as well as large-scale objects such as buildings. These results fully demonstrate the superiority and effectiveness of a 3D point cloud semantic segmentation method for large-scale, multi-object environments in handling complex road traffic scenarios.

[0100] Figure 10 The inference speed and parameter count of the method provided in this invention were quantitatively compared with other recently popular network structures on the S3DIS dataset, providing an evaluation of the model complexity of the method provided in this invention. Figure 10 As can be seen, the method provided by this invention achieves optimal performance in both model parameter count and inference speed. Compared with the recently popular BEMF-Net network structure, the method provided by this invention reduces the number of parameters to 2 / 3 and increases the inference speed to 3 / 4 times while maintaining the same mIoU performance, demonstrating higher efficiency and practicality. Compared with the classic RandLA-Net network structure, the method provided by this invention achieves a significant improvement of 4.1% in mIoU, while maintaining a similar level in model parameter count and inference speed. In summary, the method provided by this invention demonstrates excellent performance in handling large scenes and multi-object environments, and the model efficiency is also outstanding, providing an efficient and reliable solution for practical applications.

[0101] Example 3

[0102] A computer-readable storage medium storing computer instructions thereon, characterized in that the computer instructions, when executed by a processor, implement the steps of the method described in Embodiment 1 or 2 above.

[0103] Example 4

[0104] The present invention also provides a computer program product, including computer instructions, characterized in that the computer instructions, when executed by a processor, implement the steps of the methods described in Embodiment 1 or 2 above.

[0105] In summary, the present invention provides a 3D point cloud semantic segmentation method for large-scale, multi-target environments, capable of being used for point cloud semantic segmentation tasks in large-scale variable-scale target scenes. The present invention constructs a local feature learner w-LF that does not rely on strict geometric relationships, capturing and aggregating discriminative local point cloud features. Furthermore, the present invention innovatively designs a multi-scale feature fusion module MFF, integrating multi-scale coding layer features from all resolutions, enabling efficient representation of variable-scale targets.

[0106] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A 3D point cloud semantic segmentation method for large-scale, multi-target environments, characterized in that, include: By using the K-nearest neighbor search operator KNN for each coding layer of the encoder, the feature space and geometric space of the point cloud in this coding layer are searched to obtain the neighborhood semantic feature representation of the point cloud and the geometric information ignoring the absolute coordinates of the neighborhood, which are used as the bilateral input features of each coding layer. By utilizing the local feature encoding blocks of the local feature learner of each encoding layer of the encoder, the bilateral input features are feature-encoded to obtain bilateral encoded features. The bilateral encoded features are fused through feature addition operations; By utilizing the attention pooling module of the encoder's local feature learner, the fused bilateral encoded features are processed to obtain the local aggregated features of the point cloud in this encoding layer. The semantic features of the bilateral input features and the local aggregated features are added and fused using the residual connection module of the encoder's local feature learner to obtain the output features; Among them, the point cloud of the first coding layer is the original input point cloud, and the point clouds of the other coding layers are obtained by downsampling the geometric space of the point cloud of the upper coding layer and the feature space of the output features of the upper coding layer using farthest point sampling (FPS). After obtaining the output features of all coding layers, the decoder with embedded multi-scale feature fusion module (MFF) upsamples the output features of coding layers at all resolutions, and simultaneously fuses the upsampled features from coding layers at other resolutions layer by layer to obtain a feature representation with the same resolution as the original input point cloud. The feature representation is converted into a semantic segmentation result using a segmentation prediction head and then output.

2. The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 1, characterized in that, Each coding layer of the encoder includes only the K-nearest neighbor search operator KNN and the local feature learner w-LFL.

3. The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 1, characterized in that, The local feature learner w-LFL is composed of a local feature encoding block, an attention pooling module, and a residual connection module. The local feature coding block is used to encode the neighborhood semantic feature representation and the geometric information ignoring the absolute coordinates of the neighborhood of the point cloud respectively to obtain bilateral coding features, and to fuse the bilateral coding features to obtain fused coding features. The attention pooling module is used to discriminately aggregate the fused bilateral coding features to obtain the local aggregated features of the point cloud of this coding layer. The residual connection module is used to add and fuse the bilateral input features of the local feature learner w-LFL and the local aggregated features.

4. The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 1, characterized in that, By utilizing the local feature encoding blocks of the local feature learners in each encoding layer of the encoder, feature encoding is performed on the bilateral input features to obtain bilateral encoded features, including: Ignoring the absolute coordinates of the K-nearest neighbors, we use only relative coordinates and relative distances to represent local geometric relationships. Obtain geometric feature information encoding ; semantic features of K nearest neighbors To capture the center point s of the point cloud neighborhood i Surrounding local semantic relations Obtain local semantic feature information encoding ; By encoding geometric feature information and local semantic feature information, bilateral encoded features containing different implicit information are obtained. .

5. The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 4, characterized in that, The local geometric relationship , represented as: （1）； In the formula, Concat represents a cascading operation. Represents the neighborhood center point s of the point cloud i Spatial coordinate information, Represents the neighborhood center point s of the point cloud i The spatial coordinates of the k-th neighboring point in the neighborhood, where i represents the sort number of each point in the point cloud, and k represents the sort number of the neighboring point in the neighborhood of the point cloud. Represents the 2-norm operator; The geometric feature information encoding , represented as: （2）； In the formula, MLP() represents the multilayer perceptron function. Representing local geometric relationships; The local semantic relations , represented as: （3）； In the formula, Concat represents a cascading operation, i represents the sorting number of each point in the point cloud, and k represents the sorting number of the neighboring points in the point cloud's neighborhood. Represents the neighborhood center point s of the point cloud i semantic feature information, Represents the neighborhood center point s of the point cloud i The semantic feature information of the k-th neighboring point in the neighborhood. Represents a scalar computation operator; The local semantic feature information encoding , represented as: （4）； In the formula, MLP() represents the multilayer perceptron function. It represents local semantic relationships.

6. The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 4, characterized in that, The bilateral encoded features are fused using feature addition, as shown below: （5）； In the formula, ⊕ represents the characteristic addition operation. This represents the fused bilateral encoded features. Represents geometrically encoded feature information. It represents local semantic encoding feature information.

7. The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 4, characterized in that, By utilizing the attention pooling module of the encoder's local feature learner, the fused bilateral encoded features are processed to obtain the local aggregated features of the point cloud in this encoding layer, including: From the fused bilateral encoded features using shared MLP and softmax functions Learning attention weight matrix ; Through the attention weight matrix The fused bilateral coding features By performing weighted summation, the local aggregated features of the point cloud in this coding layer are obtained.

8. The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 7, characterized in that, The attention weight matrix , represented as: （6）； In the formula, softmax() represents the normalized exponential function, and MLP() represents the multilayer perceptron function. This represents the fused bilateral coding features.

9. The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 7, characterized in that, The local aggregation feature is represented as follows: （7）； In the formula, Sum() represents the summation function. This represents the attention weight matrix, and ° denotes the element-wise product of the matrices. This represents the fused bilateral coding features.

10. The 3D point cloud semantic segmentation method for large-scale, multi-target environments according to claim 1, characterized in that, After obtaining the output features of all coding layers, the decoder with embedded multi-scale feature fusion (MFF) module upsamples the output features of the coding layers at all resolutions, and fuses the upsampled features from coding layers at all other resolutions layer by layer to obtain a feature representation with the same resolution as the original input point cloud, including: Suppose the encoder has J layers, and for the j-th coding layer E j Output characteristics Upsampling is performed, and upsampled features from other resolution coding layers are fused to obtain the output features. In decoding layer D k Fusion features The obtained fusion features The process is represented as: （8）； In the formula, j and k represent the numbers of the encoding layer and the decoding layer, respectively, and E j D k These represent the j-th encoding layer and the k-th decoding layer, respectively, and ⊕ represents the feature addition operation. Indicates coding layer E j The output features, f D express Continuous upsampling to the decoding layer D k The input features at time, where k = j, j-1, ..., 1. Indicates the upsampling operator. Indicated in decoding layer D k Upper coding layer E j-1 ~E k The features obtained by upsampling and fusing the output features.

Images