Semantic segmentation method and device of laser point cloud, electronic equipment and storage medium

Abstract

The application provides a semantic segmentation method and device of a laser point cloud, electronic equipment and a storage medium, comprising: inputting three-dimensional laser point cloud data into a convolutional neural network model, performing orthogonal transformation processing on the original spatial coordinates of the three-dimensional laser point cloud data, and determining the aligned spatial coordinates; extracting the local features of each point cloud in the three-dimensional laser point cloud data through a dynamic graph convolution network layer by using the current spatial coordinates and the current features of the three point clouds; performing point cloud displacement calculation processing based on the interaction force based on the plurality of local features and the current spatial coordinates, updating the spatial coordinates of the point cloud to obtain the three-dimensional spatial coordinates of the displaced point cloud; continuing to repeat the local feature extraction and the spatial coordinate updating of the point cloud, splicing the local features and the global features extracted based on each layer, and outputting the segmentation result of each point cloud through a multilayer perceptron, thereby improving the accuracy of the semantic segmentation of the point cloud data.

Description

Technical Field

[0001] This application relates to the field of laser point cloud data processing technology, and in particular to semantic segmentation methods, apparatus, electronic devices and storage media for laser point clouds. Background Technology

[0002] With the rapid development of 3D sensing technology, devices such as LiDAR and RGB-D cameras have been widely used in industrial inspection, autonomous driving, and intelligent navigation because they can directly acquire 3D spatial information of object surfaces. However, point cloud data has characteristics such as disorder, sparsity, and non-uniform sampling, which pose significant challenges to its efficient processing and semantic understanding. Current mainstream point cloud semantic segmentation methods can be broadly classified into three categories: multi-view methods, voxel-based methods, and point-based methods. Multi-view methods project 3D point clouds onto multiple 2D perspectives (such as top view, front view, etc.) and utilize mature convolutional neural networks (CNNs) for image processing. However, the projection process leads to the loss of 3D geometric information, and the choice of perspective significantly affects the results, limiting its performance ceiling. Voxel-based methods quantize point clouds into regular 3D meshes and then use 3D convolution operations to extract features. Although the regular structure facilitates computation, it suffers from high memory consumption, limited resolution, and blurred boundaries, especially with a sharp increase in computational cost under high-density point clouds. Therefore, there is an urgent need for a point cloud semantic segmentation method that can achieve higher accuracy and stronger robustness. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a semantic segmentation method, apparatus, electronic device and storage medium for laser point clouds. By taking into account the characteristics of point cloud data, the physical concept of interaction force is introduced, which enables the dynamic adjustment of the three-dimensional spatial structure of the point cloud. In each neighborhood selection, not exactly the same neighborhood points are obtained, and the dimension and information content of local feature extraction are increased, thereby improving the accuracy of semantic segmentation of point cloud data.

[0004] This application provides a semantic segmentation method for laser point clouds, the semantic segmentation method comprising:

[0005] The three-dimensional laser point cloud data is input into a convolutional neural network model, and the original spatial coordinates of the three-dimensional laser point cloud data are orthogonally transformed to determine the aligned spatial coordinates.

[0006] Using the current spatial coordinates and current features of the three-dimensional laser point cloud data, the local features of each point cloud in the three-dimensional laser point cloud data are extracted through the dynamic graph convolutional network layer in the convolutional neural network model;

[0007] The point cloud displacement is calculated based on the interaction force of multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space, and the current spatial coordinates of the point cloud are updated to obtain the three-dimensional spatial coordinates of the point cloud after displacement.

[0008] The next layer of local feature extraction and point cloud spatial coordinate update is performed based on the displacement point cloud 3D spatial coordinates and the local features. After the local features extracted from each layer are concatenated with the global features, the segmentation result of each point cloud is output by the multilayer perceptron.

[0009] In one possible implementation, the step of inputting the three-dimensional laser point cloud data into a convolutional neural network model and performing orthogonal transformation processing on the original spatial coordinates of the three-dimensional laser point cloud data to determine the aligned spatial coordinates includes:

[0010] The k-nearest neighbor algorithm is used to determine the multiple nearest neighbor points of each point cloud in its original space;

[0011] Extract the neighborhood features of each neighborhood point, and perform convolution, pooling, and nonlinear activation operations on the neighborhood features to generate a transformation matrix;

[0012] The transformation matrix is ​​constrained by the orthogonality loss function to obtain the orthogonal matrix. The orthogonal matrix is ​​then multiplied by the original spatial coordinates of each point cloud to determine the aligned spatial coordinates.

[0013] In one possible implementation, the step of extracting local features of each point cloud in the 3D laser point cloud data using the current spatial coordinates and current features of the 3D laser point cloud data through the dynamic graph convolutional network layer in the convolutional neural network model includes:

[0014] Based on the current spatial coordinates of the point cloud, a k-nearest neighbor graph is reconstructed to determine the set of neighboring points for each point cloud.

[0015] Based on the current features of the point cloud, the neighborhood features of each neighboring point in the neighborhood point set are extracted, and convolution and max pooling operations are performed on the neighborhood features of each neighboring point to generate the local features of the point cloud.

[0016] In one possible implementation, the step of performing point cloud displacement calculation based on interaction forces on multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space includes:

[0017] Based on the current spatial coordinates of each point cloud, calculate the Euclidean distance between any two point clouds to obtain a spatial distance relationship matrix;

[0018] Calculate the vector dot product of local features of any two point clouds to obtain the feature similarity matrix;

[0019] The spatial distance relation matrix and the feature similarity relation matrix are concatenated along the channel dimension to obtain the fusion relation matrix;

[0020] The fusion relationship matrix is ​​subjected to two-dimensional convolution and one-dimensional convolution in sequence to determine the displacement of each point cloud in three-dimensional space.

[0021] In one possible implementation, the global feature is determined in the following manner:

[0022] The local features output from the last layer are adjusted by one-dimensional convolution, and max pooling is used to determine the global features of all point clouds.

[0023] In one possible implementation, the segmentation result of each point cloud, after concatenating the local and global features extracted from each layer and outputting them via a multilayer perceptron, includes:

[0024] The global feature is copied multiple times in the last dimension and concatenated with the local features of each intermediate layer according to the channel dimension to form a comprehensive feature containing multi-scale contextual information.

[0025] The integrated features are input into the multilayer perceptron for multiple convolutions, random deactivation, and normalization operations, and the segmentation result of each point cloud is output.

[0026] This application embodiment also provides a semantic segmentation device for laser point clouds, the semantic segmentation device comprising:

[0027] The orthogonal transformation module is used to input the three-dimensional laser point cloud data into the convolutional neural network model, perform orthogonal transformation processing on the original spatial coordinates of the three-dimensional laser point cloud data, and determine the aligned spatial coordinates.

[0028] The dynamic feature extraction module is used to extract local features of each point cloud in the three-dimensional laser point cloud data by using the current spatial coordinates and current features of the three-dimensional laser point cloud data through the dynamic graph convolutional network layer in the convolutional neural network model.

[0029] The coordinate update module is used to perform point cloud displacement calculation processing based on interaction forces on multiple local features and the current spatial coordinates, determine the displacement of each point cloud in three-dimensional space, and update the current spatial coordinates of the point cloud to obtain the three-dimensional spatial coordinates of the point cloud after displacement.

[0030] The determination module is used to continue to extract local features and update the spatial coordinates of the point cloud in the next layer based on the 3D spatial coordinates of the point cloud after displacement and the local features. After the local features extracted from each layer are concatenated with the global features, the segmentation result of each point cloud is output by the multilayer perceptron.

[0031] In one possible implementation, the orthogonal transformation module is used to input the 3D laser point cloud data into the convolutional neural network model, perform orthogonal transformation processing on the original spatial coordinates of the 3D laser point cloud data, and determine the aligned spatial coordinates:

[0032] The k-nearest neighbor algorithm is used to determine the multiple nearest neighbor points of each point cloud in its original space;

[0033] Extract the neighborhood features of each neighborhood point, and perform convolution, pooling, and nonlinear activation operations on the neighborhood features to generate a transformation matrix;

[0034] The transformation matrix is ​​constrained by the orthogonality loss function to obtain the orthogonal matrix. The orthogonal matrix is ​​then multiplied by the original spatial coordinates of each point cloud to determine the aligned spatial coordinates.

[0035] This application embodiment also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the semantic segmentation method for laser point clouds described above are performed.

[0036] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the semantic segmentation method for laser point clouds as described above.

[0037] The semantic segmentation method, apparatus, electronic device, and storage medium for laser point clouds provided in this application include: inputting three-dimensional laser point cloud data into a convolutional neural network model; performing orthogonal transformation on the original spatial coordinates of the three-dimensional laser point cloud data to determine aligned spatial coordinates; using the current spatial coordinates and current features of the three-dimensional laser point cloud data, extracting local features of each point cloud in the three-dimensional laser point cloud data through a dynamic graph convolutional network layer in the convolutional neural network model; performing point cloud displacement calculation based on interaction forces on multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space, and updating the current spatial coordinates of the point cloud to obtain the displaced three-dimensional spatial coordinates of the point cloud; continuing to extract local features and update the spatial coordinates of the point cloud in the next layer based on the displaced three-dimensional spatial coordinates of the point cloud and the local features; and outputting the segmentation result of each point cloud after concatenating the local features extracted from each layer with the global features through a multilayer perceptron. By considering the characteristics of point cloud data and introducing the physical concept of interaction force, the three-dimensional spatial structure of the point cloud is dynamically adjusted. In each neighborhood selection, not all neighborhood points are completely the same, and the dimension and information content of local feature extraction are increased, thereby improving the accuracy of semantic segmentation of point cloud data.

[0038] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating a semantic segmentation method for laser point clouds provided in an embodiment of this application;

[0041] Figure 2 A schematic diagram illustrating a semantic segmentation method for laser point clouds provided in an embodiment of this application;

[0042] Figure 3 A schematic diagram of the structure of a semantic segmentation device for laser point clouds provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0045] First, the applicable application scenarios of this application will be introduced. This application can be applied to the field of laser point cloud data processing technology.

[0046] Research has revealed that current mainstream point cloud semantic segmentation methods can be broadly categorized into three types: multi-view based methods, voxel-based methods, and point-based methods. Multi-view based methods project 3D point clouds onto multiple 2D viewpoints (such as top view, front view, etc.) and utilize mature convolutional neural networks (CNNs) for image processing. However, the projection process leads to the loss of 3D geometric information, and the choice of viewpoint significantly impacts the results, limiting their performance ceiling. Voxel-based methods quantize point clouds into regular 3D meshes and then use 3D convolution operations to extract features. While the regular structure facilitates computation, it suffers from high memory consumption, limited resolution, and blurred boundaries, especially with a sharp increase in computational cost for high-density point clouds. Therefore, there is an urgent need for a novel network architecture capable of dynamically adjusting the spatial structure of the point cloud during feature learning and enhancing local feature extraction capabilities to achieve higher accuracy and stronger robustness in point cloud semantic segmentation.

[0047] Based on this, embodiments of this application provide a semantic segmentation method, apparatus, electronic device, and storage medium for laser point clouds. By considering the characteristics of point cloud data and introducing the physical concept of interaction force, the three-dimensional spatial structure of the point cloud is dynamically adjusted, resulting in different neighboring points in each neighborhood selection. This also increases the dimension and information content of local feature extraction, thereby improving the accuracy of semantic segmentation of point cloud data.

[0048] Please see Figure 1 , Figure 1 This is a flowchart illustrating a semantic segmentation method for laser point clouds provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the semantic segmentation method includes:

[0049] S101: Input the three-dimensional laser point cloud data into the convolutional neural network model, perform orthogonal transformation on the original spatial coordinates of the three-dimensional laser point cloud data, and determine the aligned spatial coordinates.

[0050] In this step, the 3D laser point cloud data is input into a convolutional neural network model. A learnable transformation network module performs orthogonal transformation on the original spatial coordinates of the 3D laser point cloud data to determine the aligned spatial coordinates.

[0051] It should be noted that point cloud data collected by sensors such as LiDAR is inherently disordered and arbitrarily oriented. The same object will produce point sets with completely different spatial coordinates under different scanning angles. For example, the overall coordinate distribution of the point cloud of a table differs significantly when viewed from the front, side, or tilt. By introducing a learnable orthogonal transformation module (i.e., learning an orthogonal matrix), this application can automatically map the input point cloud to a normalized spatial coordinate system at the network front end, effectively eliminating unnecessary global rotational degrees of freedom, allowing the model to focus on the object's geometry rather than its acquisition posture, and significantly improving rotation invariance.

[0052] In one possible implementation, the step of inputting the three-dimensional laser point cloud data into a convolutional neural network model and performing orthogonal transformation processing on the original spatial coordinates of the three-dimensional laser point cloud data to determine the aligned spatial coordinates includes:

[0053] (1): The k-nearest neighbor algorithm is used to determine the multiple nearest neighbor points of each point cloud in its original space.

[0054] Here, the k-Nearest Neighbor (KNN) algorithm is used to find the spatial nearest neighbor for each point. k Let there be a neighboring point, and record this... k The set consisting of _ neighboring points is G .

[0055] (2): Extract the neighborhood features of each neighborhood point, and perform convolution, pooling and nonlinear activation operations on the neighborhood features to generate a transformation matrix.

[0056] Here, neighborhood features of each neighboring point are extracted, and convolution, pooling, and non-linear activation operations are performed on these neighborhood features to generate a 3D model. The transformation matrix of 3.

[0057] Among them, a point cloud in the point cloud data ( The characteristics of (the three-dimensional spatial coordinates of the point) are: Let the characteristics of its neighboring points be denoted as For each neighboring point, extract neighborhood features: , For the first i The set of neighborhood points corresponding to a point cloud.

[0058] (3): Obtain the orthogonal matrix by constraining the transformation matrix through the orthogonality loss function, and multiply the orthogonal matrix with the original spatial coordinates of each point cloud to determine the aligned spatial coordinates.

[0059] Here, the orthogonal matrix is ​​obtained by constraining the transformation matrix through the orthogonality loss function. The orthogonal matrix is ​​then multiplied by the original spatial coordinates of each point cloud to determine the aligned spatial coordinates.

[0060] S102: Using the current spatial coordinates and current features of the three-dimensional laser point cloud data, extract the local features of each point cloud in the three-dimensional laser point cloud data through the dynamic graph convolutional network layer in the convolutional neural network model.

[0061] In this step, the dynamic graph convolutional network layer in the convolutional neural network model is used to process the current spatial coordinates and current features of the 3D laser point cloud data, and to extract the local features of each point cloud in the 3D laser point cloud data.

[0062] It should be noted that the current spatial coordinates refer to the three-dimensional spatial position of the point cloud at the moment of execution of a certain dynamic graph convolution operation. These coordinates may have undergone the aforementioned orthogonal transformations or multiple displacement adjustments based on interaction forces, reflecting the actual geometric layout of the point cloud at the current network depth.

[0063] It should be noted that the current feature refers to the feature vector carried by the point cloud when it is input to the current network layer, which is usually a 1 / 2-dimensional real number vector. Initially, this feature is equal to its spatial coordinates (or the expanded low-dimensional embedding); as the network deepens, it gradually integrates local, global and even contextual semantic information, becoming an abstract semantic code that represents the category, geometric role and contextual relationship of the point.

[0064] Here, after initial spatial alignment, the process moves to a multi-level local feature extraction stage. This stage employs a Dynamic Graph Convolutional Layer, which combines the current spatial coordinates of the point cloud with current features to effectively aggregate geometric and semantic information within the local neighborhood of each point.

[0065] In one possible implementation, the step of extracting local features of each point cloud in the 3D laser point cloud data using the current spatial coordinates and current features of the 3D laser point cloud data through the dynamic graph convolutional network layer in the convolutional neural network model includes:

[0066] A: Reconstruct the k-nearest neighbor graph based on the current spatial coordinates of the point cloud to determine the set of neighboring points for each point cloud.

[0067] Here, at the beginning of each layer of processing, based on the spatial coordinate set of all current points, the K-Nearest Neighbors (KNN) algorithm is used to dynamically search for the k nearest neighbors in Euclidean space for each point cloud, determining the set of neighbors for each point cloud and forming a local neighborhood graph. The vertex set in the local neighborhood graph represents all point clouds, and the edge set is the adjacency relationship established by KNN.

[0068] B: Based on the current features of the point cloud, extract the neighborhood features of each neighboring point in the neighborhood point set, and perform convolution and max pooling operations on the neighborhood features of each neighboring point to generate the local features of the point cloud.

[0069] Here, the neighborhood features of the obtained point cloud are aggregated through convolution and max pooling, while maintaining the order invariance of the point cloud data, to obtain the local features of each point cloud.

[0070] It's important to note that this process selectively preserves the most discriminative feature responses within local regions, effectively capturing key geometric details (such as corners and edges) and exhibiting invariance to the input point cloud sequence. Notably, because the neighborhood structure is recalculated based on the current spatial coordinates before each graph convolution, this adjacency graph is "dynamic," meaning it continuously adjusts as the spatial distribution of the point cloud evolves. This mechanism allows points at the same physical location to have different neighbor combinations at different network depths, enabling the continuous extraction of new contextual information from multiple perspectives and significantly enhancing the richness and diversity of feature representation.

[0071] S103: Perform point cloud displacement calculation processing based on interaction forces on multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space, and update the current spatial coordinates of the point cloud to obtain the three-dimensional spatial coordinates of the point cloud after displacement.

[0072] In this step, the local features and current spatial coordinates of the obtained point cloud data are processed by point cloud displacement based on interaction forces to obtain the three-dimensional spatial coordinates of the point cloud after displacement. This allows for the proactive "fine-tuning" of the relative positional relationships of points in the point cloud based on the currently learned semantic and geometric features, thereby obtaining a more favorable local structure for classification in subsequent neighborhood searches (such as KNN).

[0073] In one possible implementation, the step of performing point cloud displacement calculation based on interaction forces on multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space includes:

[0074] a: Based on the current spatial coordinates of each point cloud, calculate the Euclidean distance between any two point clouds to obtain the spatial distance relationship matrix.

[0075] Here, the shape of the input point cloud's spatial coordinate information is as follows: Calculate the Euclidean distance between each point to obtain the spatial distance relationship matrix. ,in B This represents the size of a batch during the training of a deep network. N This indicates the number of points in the point cloud data.

[0076] b: Calculate the vector dot product of the local features of any two point clouds to obtain the feature similarity matrix.

[0077] Here, the features of the input point cloud The shape is ( D The feature dimension of the point cloud depth features (determined by the settings of the convolutional network parameters) is multiplied by the transpose of the data and itself. This yields the vector dot product of features between each point, and the feature similarity matrix is ​​also obtained. This matrix reflects the directional consistency between any two points in a high-dimensional feature space, and can be regarded as a semantic-level measure of "attraction".

[0078] c: Concatenate the spatial distance relation matrix and the feature similarity relation matrix along the channel dimension to obtain the fusion relation matrix.

[0079] Here, the spatial distance relation matrix and the feature similarity relation matrix are concatenated along the channel dimension, and the resulting fused relation matrix is: .

[0080] d: Perform two-dimensional convolution and one-dimensional convolution on the fusion relationship matrix in sequence to determine the displacement of each point cloud in three-dimensional space.

[0081] Here, the fusion relation matrix is ​​subjected to a two-dimensional convolution to obtain a result that fuses the feature relationships and distance relationships between points. Then, after another one-dimensional convolution, the displacement of each point is obtained. Furthermore, no batch normalization or ReLU activation function is used after this one-dimensional convolution, which better reflects the physical meaning of displacement, as the displacement can have both positive and negative directions. Finally, the displacement is added to the spatial coordinates of the initial input point cloud data to obtain the spatial coordinates of the point cloud data after displacement.

[0082] Specifically, by jointly learning the distance between points and feature similarity through convolutional networks, directional displacement vectors are simulated to dynamically adjust the spatial position of the point cloud. This mechanism enables the neighborhood structure to continuously evolve during multi-layer feature extraction, with each layer capturing local topological relationships from different perspectives. This overcomes the limitations of static graph structures and greatly enriches feature diversity.

[0083] Here, convolutional networks are used to simulate the interaction forces between point cloud data. and the displacements they produce, of which These are local features of the point cloud in the network. Let the point cloud be in three-dimensional spatial coordinates. To calculate a constant for this force, this constant is learned through continuous training of the convolutional network. F These are interaction forces.

[0084] In this application, a spatial structure dynamic evolution mechanism guided by deep features is implemented: semantically similar but spatially dispersed points tend to be brought closer together, while spatially overlapping points with large feature differences may be pushed apart, thereby enhancing the clustering of similar points and suppressing noise interference. This mechanism is executed multiple times (e.g., 5 times) throughout the network, causing the point cloud to gradually tend towards an optimal geometric configuration that is conducive to semantic segmentation, significantly improving the comprehensiveness and accuracy of local feature extraction.

[0085] S104: Continue to extract local features and update the spatial coordinates of the point cloud in the next layer based on the three-dimensional spatial coordinates of the point cloud after displacement and the local features. After the local features extracted from each layer are concatenated with the global features, the segmentation result of each point cloud is output by the multilayer perceptron.

[0086] In this step, the local feature extraction and spatial coordinate update of the point cloud are performed on the next layer based on the three-dimensional spatial coordinates and local features of the point cloud after displacement. The above steps are repeated until the number of iterations is reached. After the local features and global features extracted from each layer are concatenated, the segmentation result of each point cloud is output by the multilayer perceptron.

[0087] In one possible implementation, the global features are determined by performing a one-dimensional convolution on the local features output from the last layer to adjust the dimension, and then using max pooling to determine the global features of all point clouds.

[0088] Here, a one-dimensional convolutional layer is used to change the dimension of the input features, and then max pooling is performed to obtain the global features.

[0089] In one possible implementation, the segmentation result of each point cloud, after concatenating the local and global features extracted from each layer and outputting them via a multilayer perceptron, includes:

[0090] The global feature is copied multiple times in the last dimension and concatenated with the local features of each intermediate layer according to the channel dimension to form a comprehensive feature containing multi-scale contextual information; the comprehensive feature is input into the multilayer perceptron for multiple convolutions, random deactivation processing and normalization operations, and the segmentation result of each point cloud is output.

[0091] Here, the global features are copied multiple times along the last dimension and concatenated with the local features of each intermediate layer along the channel dimension to form a comprehensive feature containing multi-scale contextual information. The comprehensive feature is then input into the multilayer perceptron for multiple convolutions, random deactivation processing, and normalization operations, outputting the segmentation result for each point cloud.

[0092] The segmentation result of the point cloud is the semantic category label corresponding to each point cloud.

[0093] In one possible implementation, the convolutional neural network model is trained through the following steps:

[0094] Obtain 3D point cloud training samples with real semantic labels; input the training samples into the neural network model to be trained, and the model sequentially performs: learnable orthogonal transformation of point cloud coordinates, multiple alternating executions of dynamic graph convolutional local feature extraction and spatial displacement adjustment based on interaction forces, multi-scale feature fusion, and classification output; calculate the segmentation loss based on the difference between the predicted label output by the model and the real semantic label, and combine the orthogonality constraint loss of the orthogonal transformation module to form the total loss function; optimize the model parameters using the backpropagation algorithm until the convergence condition is met, and obtain the trained convolutional neural network model for point cloud semantic segmentation.

[0095] Here, the training steps of the convolutional neural network model in this application are as follows: Step 1, prepare the training sample dataset: obtain labeled point cloud data containing three-dimensional spatial coordinates and real semantic labels as training samples. Step 2, construct the network structure and initialize parameters: it mainly includes a learnable orthogonal transformation module (an improved version of the T-Net structure), a dynamic graph convolutional network layer (including KNN and local feature aggregation), a spatial displacement module based on interaction forces (fusion of distance and feature similarity to calculate displacement), a multi-scale feature concatenation and global pooling module, and a segmentation head MLP module (composed of multiple one-dimensional convolutions, batch normalization, random deactivation, and ReLU). Initialize the weight parameters of all convolutional layers and fully connected layers in the network using the He initialization method; set the orthogonal constraint loss coefficient to supervise the orthogonality of the T-Net output matrix. Step 3: Forward Propagation and Segmentation Result Generation: Input a training sample into the network and perform the following operations sequentially: First, spatially calibrate the original coordinates using the orthogonal transformation module; then enter the loop structure: repeat the "dynamic graph convolution feature extraction → spatial displacement adjustment" step 5 times (the last time only extracts features, without displacement), gradually enhancing the local feature representation ability; in each iteration, using the depth features and spatial coordinates of the current point, calculate the three-dimensional displacement of each point through the interaction force-based displacement module, update its spatial position, and use it for the next round of neighborhood search; finally, concatenate the local features output by each layer with the global max pooling features; feed it into the multilayer perceptron (MLP) module, output the original score of each point belonging to the category, and convert it into a probability distribution through the Softmax function. Step 4: Calculate the Loss Function and Backpropagate: Use the weighted cross-entropy loss function to measure the difference between the predicted result and the true label. Step 5: Optimizer Configuration and Training Control: Use the Adam optimizer to update parameters, with an initial learning rate of 0.001 that decays exponentially with the training process; batch size of 16; and a total of 100 training rounds. Every 10 epochs, the mIoU (mean Intersection over Union) metric is evaluated on the validation set, and the checkpoints of the models with the best performance are saved.

[0096] For further details, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating a semantic segmentation method for laser point clouds provided in an embodiment of this application. Figure 2As shown, input point cloud data undergoes orthogonal transformation to obtain aligned spatial coordinates. A dynamic graph convolutional network layer is used to extract local features of each point in the 3D point cloud data. Based on these local features and the current spatial coordinates, point cloud displacement calculations based on interaction forces are performed to obtain the displaced 3D spatial coordinates. This process of feature extraction and interaction force-based point cloud displacement is repeated multiple times. Finally, the local features of the point cloud are subjected to max pooling to obtain global features. This process is repeated and concatenated with the local features. The final segmentation result of the point cloud data is obtained through a multilayer perceptron.

[0097] This application effectively solves the problems of fixed neighborhood, insufficient feature extraction, and sensitivity to spatial transformation in traditional point cloud segmentation methods, and significantly improves the accuracy, robustness and generalization ability of semantic segmentation, which has important theoretical significance and broad application prospects.

[0098] This application provides a semantic segmentation method for laser point clouds. The semantic segmentation method includes: inputting three-dimensional laser point cloud data into a convolutional neural network model; performing orthogonal transformation on the original spatial coordinates of the three-dimensional laser point cloud data to determine the aligned spatial coordinates; using the current spatial coordinates and current features of the three-dimensional laser point cloud data, extracting local features of each point cloud in the three-dimensional laser point cloud data through a dynamic graph convolutional network layer in the convolutional neural network model; performing point cloud displacement calculation based on interaction forces on multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space, and updating the current spatial coordinates of the point cloud to obtain the displaced three-dimensional spatial coordinates of the point cloud; continuing to extract local features and update the spatial coordinates of the point cloud in the next layer based on the displaced three-dimensional spatial coordinates of the point cloud and the local features; and outputting the segmentation result of each point cloud after concatenating the local features extracted from each layer with the global features through a multilayer perceptron. By considering the characteristics of point cloud data and introducing the physical concept of interaction force, the three-dimensional spatial structure of the point cloud is dynamically adjusted. In each neighborhood selection, not all neighborhood points are completely the same, and the dimension and information content of local feature extraction are increased, thereby improving the accuracy of semantic segmentation of point cloud data.

[0099] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a semantic segmentation device for laser point clouds provided in an embodiment of this application. Figure 3 As shown, the semantic segmentation device 300 for laser point clouds includes:

[0100] The orthogonal transformation module 310 is used to input the three-dimensional laser point cloud data into the convolutional neural network model, perform orthogonal transformation processing on the original spatial coordinates of the three-dimensional laser point cloud data, and determine the aligned spatial coordinates.

[0101] The dynamic feature extraction module 320 is used to extract local features of each point cloud in the three-dimensional laser point cloud data by using the current spatial coordinates and current features of the three-dimensional laser point cloud data through the dynamic graph convolutional network layer in the convolutional neural network model.

[0102] The coordinate update module 330 is used to perform point cloud displacement calculation processing based on interaction forces on multiple local features and the current spatial coordinates, determine the displacement of each point cloud in three-dimensional space, and update the current spatial coordinates of the point cloud to obtain the three-dimensional spatial coordinates of the point cloud after displacement.

[0103] The determination module 340 is used to continue to extract local features and update the spatial coordinates of the point cloud in the next layer based on the three-dimensional spatial coordinates of the point cloud after displacement and the local features. After the local features extracted from each layer are concatenated with the global features, the segmentation result of each point cloud is output by the multilayer perceptron.

[0104] Furthermore, the orthogonal transformation module 310 is used to input the three-dimensional laser point cloud data into the convolutional neural network model, perform orthogonal transformation processing on the original spatial coordinates of the three-dimensional laser point cloud data, and determine the aligned spatial coordinates:

[0105] The k-nearest neighbor algorithm is used to determine the multiple nearest neighbor points of each point cloud in its original space;

[0106] Extract the neighborhood features of each neighborhood point, and perform convolution, pooling, and nonlinear activation operations on the neighborhood features to generate a transformation matrix;

[0107] The transformation matrix is ​​constrained by the orthogonality loss function to obtain the orthogonal matrix. The orthogonal matrix is ​​then multiplied by the original spatial coordinates of each point cloud to determine the aligned spatial coordinates.

[0108] Furthermore, the dynamic feature extraction module 320 is used to extract local features of each point cloud in the 3D laser point cloud data by utilizing the current spatial coordinates and current features of the 3D laser point cloud data through the dynamic graph convolutional network layer in the convolutional neural network model:

[0109] Based on the current spatial coordinates of the point cloud, a k-nearest neighbor graph is reconstructed to determine the set of neighboring points for each point cloud.

[0110] Based on the current features of the point cloud, the neighborhood features of each neighboring point in the neighborhood point set are extracted, and convolution and max pooling operations are performed on the neighborhood features of each neighboring point to generate the local features of the point cloud.

[0111] Furthermore, the coordinate update module 330 is used to perform point cloud displacement calculation processing based on interaction forces on multiple local features and the current spatial coordinates, and to determine the displacement of each point cloud in three-dimensional space, including:

[0112] Based on the distance to each of the points, a spatial distance relationship matrix is ​​obtained;

[0113] Calculate the vector dot product of local features of any two point clouds to obtain the feature similarity matrix;

[0114] The spatial distance relation matrix and the feature similarity relation matrix are concatenated along the channel dimension to obtain the fusion relation matrix;

[0115] The fusion relationship matrix is ​​subjected to two-dimensional convolution and one-dimensional convolution in sequence to determine the displacement of each point cloud in three-dimensional space.

[0116] Furthermore, the coordinate update module 330 is used to perform point cloud displacement calculation processing based on interaction forces on multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space:

[0117] Based on the current spatial coordinates of each point cloud, calculate the Euclidean distance between any two point clouds to obtain a spatial distance relationship matrix;

[0118] Calculate the vector dot product of local features of any two point clouds to obtain the feature similarity matrix;

[0119] The spatial distance relation matrix and the feature similarity relation matrix are concatenated along the channel dimension to obtain the fusion relation matrix;

[0120] The fusion relationship matrix is ​​subjected to two-dimensional convolution and one-dimensional convolution in sequence to determine the displacement of each point cloud in three-dimensional space.

[0121] Furthermore, the determining module 340 determines the global features in the following manner:

[0122] The local features output from the last layer are adjusted by one-dimensional convolution, and max pooling is used to determine the global features of all point clouds.

[0123] Furthermore, the determining module 340 is used to concatenate the local and global features extracted from each layer, and then output the segmentation result of each point cloud through a multilayer perceptron, including:

[0124] The global feature is copied multiple times in the last dimension and concatenated with the local features of each intermediate layer according to the channel dimension to form a comprehensive feature containing multi-scale contextual information.

[0125] The integrated features are input into the multilayer perceptron for multiple convolutions, random deactivation, and normalization operations, and the segmentation result of each point cloud is output.

[0126] This application provides a semantic segmentation device for laser point clouds. The semantic segmentation device includes: an orthogonal transformation module, used to input three-dimensional laser point cloud data into a convolutional neural network model, perform orthogonal transformation processing on the original spatial coordinates of the three-dimensional laser point cloud data, and determine the aligned spatial coordinates; a dynamic feature extraction module, used to extract local features of each point cloud in the three-dimensional laser point cloud data through the dynamic graph convolutional network layer in the convolutional neural network model using the current spatial coordinates and current features of the three-dimensional laser point cloud data; a coordinate update module, used to perform point cloud displacement calculation processing based on interaction forces on multiple local features and the current spatial coordinates, determine the displacement of each point cloud in three-dimensional space, and update the current spatial coordinates of the point cloud to obtain the displaced three-dimensional spatial coordinates of the point cloud; and a determination module, used to continue to perform local feature extraction and point cloud spatial coordinate update for the next layer based on the displaced three-dimensional spatial coordinates of the point cloud and the local features, and after concatenating the local features extracted from each layer with the global features, output the segmentation result of each point cloud through a multilayer perceptron. By considering the characteristics of point cloud data and introducing the physical concept of interaction force, the three-dimensional spatial structure of the point cloud is dynamically adjusted. In each neighborhood selection, not all neighborhood points are completely the same, and the dimension and information content of local feature extraction are increased, thereby improving the accuracy of semantic segmentation of point cloud data.

[0127] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0128] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 as well as Figure 2 The steps of the semantic segmentation method for laser point clouds in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0129] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 as well as Figure 2 The steps of the semantic segmentation method for laser point clouds in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0130] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0134] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A semantic segmentation method for laser point clouds, characterized in that, The semantic segmentation method includes: The three-dimensional laser point cloud data is input into a convolutional neural network model, and the original spatial coordinates of the three-dimensional laser point cloud data are orthogonally transformed to determine the aligned spatial coordinates. Using the current spatial coordinates and current features of the three-dimensional laser point cloud data, the local features of each point cloud in the three-dimensional laser point cloud data are extracted through the dynamic graph convolutional network layer in the convolutional neural network model; The point cloud displacement is calculated based on the interaction force of multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space, and the current spatial coordinates of the point cloud are updated to obtain the three-dimensional spatial coordinates of the point cloud after displacement. The next layer of local feature extraction and point cloud spatial coordinate update is performed based on the displacement point cloud 3D spatial coordinates and the local features. After the local features extracted from each layer are concatenated with the global features, the segmentation result of each point cloud is output by the multilayer perceptron. The step of performing point cloud displacement calculation based on interaction forces on multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space includes: Based on the current spatial coordinates of each point cloud, calculate the Euclidean distance between any two point clouds to obtain a spatial distance relationship matrix; Calculate the vector dot product of local features of any two point clouds to obtain the feature similarity matrix; The spatial distance relation matrix and the feature similarity relation matrix are concatenated along the channel dimension to obtain the fusion relation matrix; The fusion relationship matrix is ​​subjected to two-dimensional convolution and one-dimensional convolution in sequence to determine the displacement of each point cloud in three-dimensional space.

2. The semantic segmentation method according to claim 1, characterized in that, The step of inputting 3D laser point cloud data into a convolutional neural network model and performing orthogonal transformation on the original spatial coordinates of the 3D laser point cloud data to determine the aligned spatial coordinates includes: The k-nearest neighbor algorithm is used to determine the multiple nearest neighbor points of each point cloud in its original space; Extract the neighborhood features of each neighborhood point, and perform convolution, pooling, and nonlinear activation operations on the neighborhood features to generate a transformation matrix; The transformation matrix is ​​constrained by the orthogonality loss function to obtain the orthogonal matrix. The orthogonal matrix is ​​then multiplied by the original spatial coordinates of each point cloud to determine the aligned spatial coordinates.

3. The semantic segmentation method according to claim 1, characterized in that, The step of extracting local features of each point cloud in the 3D laser point cloud data by using the current spatial coordinates and current features of the 3D laser point cloud data through the dynamic graph convolutional network layer in the convolutional neural network model includes: Based on the current spatial coordinates of the point cloud, a k-nearest neighbor graph is reconstructed to determine the set of neighboring points for each point cloud. Based on the current features of the point cloud, the neighborhood features of each neighboring point in the neighborhood point set are extracted, and convolution and max pooling operations are performed on the neighborhood features of each neighboring point to generate the local features of the point cloud.

4. The semantic segmentation method according to claim 1, characterized in that, The global features are determined in the following way: The local features output from the last layer are adjusted by one-dimensional convolution, and max pooling is used to determine the global features of all point clouds.

5. The semantic segmentation method according to claim 1, characterized in that, The segmentation results of each point cloud, after being concatenated with the local and global features extracted from each layer and processed by a multilayer perceptron, include: The global feature is copied multiple times in the last dimension and concatenated with the local features of each intermediate layer according to the channel dimension to form a comprehensive feature containing multi-scale contextual information. The integrated features are input into the multilayer perceptron for multiple convolutions, random deactivation, and normalization operations, and the segmentation result of each point cloud is output.

6. A semantic segmentation device for laser point clouds, characterized in that, The semantic segmentation device includes: The orthogonal transformation module is used to input the three-dimensional laser point cloud data into the convolutional neural network model, perform orthogonal transformation processing on the original spatial coordinates of the three-dimensional laser point cloud data, and determine the aligned spatial coordinates. The dynamic feature extraction module is used to extract local features of each point cloud in the three-dimensional laser point cloud data by using the current spatial coordinates and current features of the three-dimensional laser point cloud data through the dynamic graph convolutional network layer in the convolutional neural network model. The coordinate update module is used to perform point cloud displacement calculation processing based on interaction forces on multiple local features and the current spatial coordinates, determine the displacement of each point cloud in three-dimensional space, and update the current spatial coordinates of the point cloud to obtain the three-dimensional spatial coordinates of the point cloud after displacement. The determination module is used to continue to extract local features and update the spatial coordinates of the point cloud in the next layer based on the three-dimensional spatial coordinates of the point cloud after displacement and the local features. After the local features extracted from each layer are concatenated with the global features, the segmentation result of each point cloud is output by the multilayer perceptron. The coordinate update module is used to perform point cloud displacement calculation based on interaction forces on multiple local features and the current spatial coordinates to determine the displacement of each point cloud in three-dimensional space. Based on the current spatial coordinates of each point cloud, calculate the Euclidean distance between any two point clouds to obtain a spatial distance relationship matrix; Calculate the vector dot product of local features of any two point clouds to obtain the feature similarity matrix; The spatial distance relation matrix and the feature similarity relation matrix are concatenated along the channel dimension to obtain the fusion relation matrix; The fusion relationship matrix is ​​subjected to two-dimensional convolution and one-dimensional convolution in sequence to determine the displacement of each point cloud in three-dimensional space.

7. The semantic segmentation apparatus according to claim 6, characterized in that, The orthogonal transformation module is used to input the 3D laser point cloud data into the convolutional neural network model, perform orthogonal transformation processing on the original spatial coordinates of the 3D laser point cloud data, and determine the aligned spatial coordinates: The k-nearest neighbor algorithm is used to determine the multiple nearest neighbor points of each point cloud in its original space; Extract the neighborhood features of each neighborhood point, and perform convolution, pooling, and nonlinear activation operations on the neighborhood features to generate a transformation matrix; The transformation matrix is ​​constrained by the orthogonality loss function to obtain the orthogonal matrix. The orthogonal matrix is ​​then multiplied by the original spatial coordinates of each point cloud to determine the aligned spatial coordinates.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the semantic segmentation method for laser point clouds as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the semantic segmentation method for laser point clouds as described in any one of claims 1 to 5.

Images