Road scene point cloud recognition method and system suitable for inspection robots

Abstract

The present invention provides a road scene point cloud recognition method and system suitable for an inspection robot, which relates to the field of point cloud recognition technology, including segmenting a pre-processed three-dimensional point cloud using a region growing method and a Euclidean clustering algorithm, respectively extracting features from a road surface, a road marking and a road boundary point cloud in a preliminary segmentation result, respectively mapping the optimized road surface, road marking and road boundary to different levels of an octree, and constructing a three-dimensional point cloud map based on an octree; constructing a point cloud classifier model; inputting the three-dimensional point cloud map of the octree into the point cloud classifier model for prediction, obtaining a category label and a confidence level of each point cloud, optimizing the preliminary segmentation result according to a confidence level threshold, planning a navigation path of the inspection robot according to the optimization result and generating navigation path data, and transmitting the navigation path data to a motion control module of the robot so that the robot can perform inspections.

Description

Technical Field

[0001] The present invention relates to point cloud recognition technology, and in particular to a road scene point cloud recognition method and system suitable for inspection robots. Background Art

[0002] With the continuous development of robotics technology, mobile robots have been widely used in various fields. Among them, autonomous navigation and scene understanding in outdoor environments are one of the key technical challenges facing mobile robots. Especially in application scenarios such as road inspection, robots need to navigate autonomously in complex road environments to complete tasks such as road condition detection, road damage identification, and traffic sign and marking detection. Therefore, how to achieve accurate perception and understanding of road scenes is an important basis for inspection robots to work autonomously.

[0003] Traditional road scene perception methods mainly rely on visual image information, and analyze and identify road images through image processing and computer vision algorithms. For example, image processing techniques such as edge detection and region segmentation are used to extract road boundaries and lane lines, and methods such as template matching and feature point matching are used to detect and recognize traffic signs. However, image-based road perception methods are easily affected by factors such as lighting changes and weather conditions, resulting in insufficient recognition accuracy and stability. In addition, image information lacks the expression of scene depth and three-dimensional structure, making it difficult to accurately obtain the geometric shape and spatial relationship of the road. Summary of the invention

[0004] The embodiments of the present invention provide a road scene point cloud recognition method and system suitable for an inspection robot, which can at least solve some problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provides a road scene point cloud recognition method suitable for inspection robots, including:

[0007] Collecting a three-dimensional point cloud of a road scene, the three-dimensional point cloud including point cloud information of a road surface, road markings, road boundaries, and roadside obstacles; preprocessing the three-dimensional point cloud, downsampling the point cloud using an adaptive voxel filtering algorithm, and removing outliers using a statistical filtering algorithm to obtain a preprocessed three-dimensional point cloud;

[0008] The preprocessed three-dimensional point cloud is segmented using the regional growing method and the Euclidean clustering algorithm to obtain the preliminary segmentation results of the road surface, road markings, road boundaries and roadside obstacles; the road surface point cloud, road marking point cloud and road boundary point cloud in the preliminary segmentation results are respectively feature extracted, and the preset algorithm is used to optimize them to obtain the optimized road surface point cloud, road marking point cloud and road boundary point cloud; the optimized road surface point cloud, road marking point cloud and road boundary point cloud are respectively mapped to different levels of the octree to construct a three-dimensional point cloud map based on the octree;

[0009] Construct a point cloud classifier model; input the octree's three-dimensional point cloud map into the point cloud classifier model for prediction, obtain the category label and confidence of each point cloud, optimize the preliminary segmentation results according to the confidence threshold, plan the inspection robot's navigation path according to the optimization results and generate navigation path data, and transmit the navigation path data to the robot's motion control module so that the robot can perform inspections.

[0010] According to a second aspect of the embodiments of the present invention,

[0011] Provides a road scene point cloud recognition system suitable for inspection robots, including:

[0012] The first unit is used to collect a three-dimensional point cloud of a road scene, wherein the three-dimensional point cloud includes point cloud information of a road surface, road markings, road boundaries, and roadside obstacles; pre-process the three-dimensional point cloud, down-sample the point cloud using an adaptive voxel filtering algorithm, and remove outliers using a statistical filtering algorithm to obtain a pre-processed three-dimensional point cloud;

[0013] The second unit is used to segment the preprocessed three-dimensional point cloud using the regional growing method and the Euclidean clustering algorithm to obtain the preliminary segmentation results of the road surface, road markings, road boundaries and roadside obstacles; the road surface point cloud, road marking point cloud and road boundary point cloud in the preliminary segmentation results are respectively feature extracted, and the preset algorithm is used to optimize them to obtain the optimized road surface point cloud, road marking point cloud and road boundary point cloud; the optimized road surface point cloud, road marking point cloud and road boundary point cloud are respectively mapped to different levels of the octree to construct a three-dimensional point cloud map based on the octree;

[0014] The third unit is used to build a point cloud classifier model; the three-dimensional point cloud map of the octree is input into the point cloud classifier model for prediction, the category label and confidence of each point cloud are obtained, the preliminary segmentation result is optimized according to the confidence threshold, the navigation path of the inspection robot is planned according to the optimization result and the navigation path data is generated, and the navigation path data is transmitted to the motion control module of the robot so that the robot can perform inspection.

[0015] According to a third aspect of the embodiments of the present invention,

[0016] An electronic device is provided, comprising:

[0017] processor;

[0018] a memory for storing processor-executable instructions;

[0019] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0020] According to a fourth aspect of the embodiments of the present invention,

[0021] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0022] Aiming at the problem of point cloud classification in three-dimensional reconstruction of road scenes, this application proposes a point cloud classifier model based on octree and attention mechanism. First, the large-scale point cloud is spatially divided and hierarchically organized using octree, and the geometric and semantic features of the point cloud are extracted at the leaf node level of the octree. Then, the extracted leaf node feature vector is input into a three-dimensional convolutional neural network integrated with the attention mechanism, and the category label of the point cloud is obtained through end-to-end feature learning and classification. Finally, guided by cross entropy loss and focal loss, the point cloud classifier model is trained and optimized using optimizer and learning rate scheduling strategy.

[0023] This application proposes a method for extracting the road centerline and planning the robot navigation path based on the point cloud segmentation results. The method first optimizes the point cloud segmentation results based on the confidence threshold and conditional random field to obtain the refined semantic segmentation of the road surface, markings, boundaries and obstacles. Then, the centerline skeleton is extracted from the road surface point cloud, and the centerline is optimized by integrating semantic elements such as road markings and boundaries to obtain an accurate and smooth road centerline. Finally, based on the road centerline and combined with the roadside obstacle information, the robot's navigation path is planned, and path data is generated to guide the robot to drive autonomously. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a method for identifying a road scene point cloud applicable to an inspection robot according to an embodiment of the present invention;

[0025] Figure 2 Schematic diagram of the structure of a road scene point cloud recognition system applicable to an inspection robot according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0028] Figure 1 FIG. 1 is a flow chart of a method for identifying a road scene point cloud applicable to an inspection robot according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0029] S101. Collecting a three-dimensional point cloud of a road scene, the three-dimensional point cloud including point cloud information of a road surface, road markings, road boundaries, and roadside obstacles; preprocessing the three-dimensional point cloud, downsampling the point cloud using an adaptive voxel filtering algorithm, and removing outliers using a statistical filtering algorithm to obtain a preprocessed three-dimensional point cloud;

[0030] S102. Segment the preprocessed three-dimensional point cloud using the region growing method and the Euclidean clustering algorithm to obtain preliminary segmentation results of the road surface, road markings, road boundaries and roadside obstacles; extract features from the road surface point cloud, road marking point cloud and road boundary point cloud in the preliminary segmentation results respectively, and optimize them using a preset algorithm to obtain optimized road surface point cloud, road marking point cloud and road boundary point cloud; map the optimized road surface point cloud, road marking point cloud and road boundary point cloud to different levels of the octree respectively, and construct a three-dimensional point cloud map based on the octree;

[0031] S103. Construct a point cloud classifier model; input the three-dimensional point cloud map of the octree into the point cloud classifier model for prediction, obtain the category label and confidence of each point cloud, optimize the preliminary segmentation result according to the confidence threshold, plan the navigation path of the inspection robot according to the optimization result and generate navigation path data, and transmit the navigation path data to the robot's motion control module so that the robot can perform inspection.

[0032] In an optional embodiment,

[0033] The three-dimensional point cloud is preprocessed, the point cloud is downsampled by an adaptive voxel filtering algorithm, and outliers are removed by a statistical filtering algorithm to obtain a preprocessed three-dimensional point cloud, including:

[0034] Downsampling the three-dimensional point cloud using an adaptive voxel filtering algorithm, determining the voxel size according to the local density of the point cloud, recursively dividing the point cloud space into eight sub-voxels until the number of point clouds in the sub-voxel is less than a preset threshold or reaches a maximum division depth, and calculating the center of gravity of the point cloud in each non-empty sub-voxel to generate a downsampled point cloud;

[0035] For the downsampled point cloud, outliers are removed using a statistical filtering algorithm based on the local neighborhood covariance matrix. The local neighborhood covariance matrix of each point is calculated. According to the eigenvalues ​​and eigenvectors of the covariance matrix, a local coordinate system is constructed. The neighborhood points are projected into the local coordinate system, and the Mahalanobis distance of each neighborhood point to the origin of the local coordinate system is calculated. According to the Mahalanobis distance distribution, the outlier threshold is self-determined, and the neighborhood points with a Mahalanobis distance greater than the threshold are marked as potential outliers. Based on the spatial distribution and connectivity characteristics of the potential outliers, the outlier threshold is adjusted to obtain the optimized potential outlier marking result.

[0036] For each point in the optimized potential outlier marking result, the proportion of optimized potential outliers in its local neighborhood is counted. If the proportion exceeds the preset threshold, the point is marked as an outlier. The potential outliers and outlier marks are updated through iterative optimization until the marking results converge. All points marked as outliers are removed to obtain the point cloud after removing the outliers. The point cloud around the outliers is repaired and smoothed using local surface reconstruction technology to generate a preprocessed three-dimensional point cloud.

[0037] Exemplarily, the original three-dimensional point cloud is downsampled using an adaptive voxel filtering algorithm, and the specific steps are as follows:

[0038] According to the local point density characteristics of the input point cloud, the size of the voxel division is adaptively determined. Smaller voxels are used in areas with higher point cloud density, and larger voxels are used in areas with lower point cloud density to achieve adaptive sparseness of the point cloud. The three-dimensional point cloud space is recursively divided into eight sub-voxels using the octree space division method. In each division, it is judged whether the number of point clouds in the current sub-voxel is less than the set threshold, or whether the current division depth has reached the preset maximum depth, as the recursive termination condition. For each non-empty sub-voxel obtained by the division, the centroid position of all point clouds in the voxel is calculated as the representative point of the voxel. Empty voxels that do not contain point clouds are discarded to obtain the downsampled point cloud. The three-dimensional point cloud downsampled by the adaptive voxel filtering algorithm is output as the input for subsequent processing.

[0039] The statistical filtering algorithm based on the local neighborhood covariance matrix is ​​used to remove outliers in the point cloud. The specific steps are as follows:

[0040] For each point in the downsampled point cloud, extract the point cloud subset in its local neighborhood and calculate the 3×3 covariance matrix of this point cloud subset. Perform eigenvalue decomposition on the covariance matrix to obtain three eigenvalues ​​and corresponding eigenvectors. Sort by eigenvalue size and construct a local coordinate system. The eigenvector corresponding to the smallest eigenvalue is used as the normal vector direction, and the other two directions are orthogonal to it.

[0041] Project each point in the local neighborhood to the local coordinate system and calculate the Mahalanobis distance from the projection point to the origin of the local coordinate system. The Mahalanobis distance takes into account the anisotropy of the point cloud distribution and can better measure the local discreteness of the point cloud.

[0042] According to the distribution of Mahalanobis distances of all neighborhood points, the threshold for judging outliers is determined adaptively. Neighborhood points whose Mahalanobis distance is greater than the threshold are marked as potential outliers. The potential outliers obtained in the previous step are further optimized by combining the spatial distribution characteristics and connectivity characteristics of potential outliers. Potential outliers with dense spatial distribution and connectivity are retained, while potential outliers with sparse distribution and isolation are removed.

[0043] Iterate the optimization of potential outliers. For each potential outlier, count the percentage of potential outliers in its local neighborhood. If the percentage exceeds the preset threshold, the point is finally marked as an outlier. The potential outlier marks are continuously updated in an iterative manner until the outlier marking results converge globally.

[0044] All points marked as outliers are removed from the point cloud to obtain a clean point cloud after outliers are removed. In order to avoid discontinuous holes in the point cloud after outliers are removed, the point cloud local surface reconstruction technology is used to repair and smooth the area around the outliers so that the continuity and integrity of the point cloud can be maintained. The point cloud after outliers are removed and repaired by statistical filtering method is output as the result of 3D point cloud preprocessing.

[0045] The above scheme first uses the adaptive voxel filtering algorithm to achieve downsampling of the point cloud according to the distribution characteristics of the point cloud, and uses the statistical filtering of the local neighborhood covariance matrix to achieve the identification and removal of outliers in the point cloud. The entire preprocessing process fully considers the local distribution characteristics of the point cloud, and finally obtains a sparse, clean, and complete three-dimensional point cloud, which lays a good data foundation for subsequent point cloud registration, three-dimensional reconstruction and other links.

[0046] In an optional embodiment,

[0047] The road surface point cloud, road marking point cloud and road boundary point cloud in the preliminary segmentation results are respectively feature extracted and optimized using a preset algorithm to obtain the optimized road surface point cloud, road marking point cloud and road boundary point cloud, including:

[0048] For the road surface point cloud obtained by preliminary segmentation, the principal component analysis algorithm is used to calculate the normal vector and curvature characteristics of each road surface point cloud, where the curvature is calculated by the ratio of the minimum eigenvalue to the sum of the eigenvalues ​​of the point cloud neighborhood covariance matrix; the road plane is fitted using a random sampling consistency algorithm, and the angle between the normal vector of each point cloud and the normal vector of the fitting plane is calculated. The point clouds with angles less than a preset threshold are retained, and the point clouds with angles greater than or equal to the preset threshold are eliminated to obtain the optimized road surface point cloud;

[0049] For the road marking point cloud obtained by preliminary segmentation, the reflection intensity value of each road marking point cloud is extracted, and the reflection intensity segmentation threshold is determined by using the Otsu threshold method, and the target optimal threshold is determined by maximizing the inter-class variance; the point cloud is binarized and segmented using the target optimal threshold to obtain a binarization result; the binarization result is optimized using morphological opening and closing operations to obtain a morphological optimization result; the connected domain in the morphological optimization result is extracted, the number of pixels in each connected domain is calculated, and the connected domain with a pixel number less than the preset threshold is eliminated to obtain the optimized road marking point cloud;

[0050] For the road boundary point cloud obtained by preliminary segmentation, the finite difference method is used to calculate the three-dimensional gradient features of each road boundary point cloud, and the central difference format is used to calculate the gradient components of the road boundary point cloud in each direction; the road boundary is extracted using the region growing algorithm, and the point cloud with a gradient value exceeding a preset threshold is used as the seed point, and the point cloud with a gradient value exceeding the preset threshold and not visited in the neighborhood point cloud of the seed point is merged into the same region; for the regional point cloud obtained by growth, the connected domain is extracted, the number of point clouds and the area of ​​each connected domain are calculated, and the connected domains with a point cloud number less than the preset threshold or an area less than the preset threshold are eliminated, and the point clouds in the remaining connected domains are locally plane fitted to obtain the optimized road boundary point cloud.

[0051] Exemplarily, the road surface point cloud obtained by preliminary segmentation is optimized, and the specific steps are as follows:

[0052] Using the principal component analysis algorithm, the local neighborhood of each road surface point cloud is decomposed to obtain the eigenvalue and the corresponding eigenvector. Among them, the eigenvector corresponding to the minimum eigenvalue is the normal vector of the point cloud, which represents the direction information of the local point cloud.

[0053] The local curvature characteristics of each point cloud are calculated based on the eigenvalues. The curvature is obtained by the ratio of the minimum eigenvalue of the point cloud neighborhood covariance matrix to the sum of the eigenvalues. The larger the curvature value, the greater the curvature of the local surface of the point cloud.

[0054] In the road surface point cloud, three non-collinear points are randomly selected to form a sampling set. The random sampling consistency algorithm is used to iteratively fit the optimal plane model of the sampling set as an approximation of the road plane.

[0055] The angle between the normal vector of each road surface point cloud and the normal vector of the fitting plane is calculated. According to the preset angle threshold, the inner points with angles less than the threshold are retained, and the outliers with angles greater than or equal to the threshold are eliminated to obtain the optimized road surface point cloud.

[0056] The road marking point cloud obtained by preliminary segmentation is optimized. The specific steps are as follows:

[0057] The reflection intensity attribute value of each road marking point cloud is extracted and used to perform binary segmentation of the point cloud. Since most road markings are made of high reflectivity materials, the reflection intensity can be used to better distinguish between marking and non-marking point clouds. The Otsu threshold method is used to adaptively determine the segmentation threshold of reflection intensity. By maximizing the inter-class variance of the marking and non-marking classes, the optimal segmentation threshold is obtained, and the point clouds above the threshold are marked as markings, and those below the threshold are marked as background.

[0058] Based on the binary segmentation results, the road marking point cloud is optimized using morphological opening and closing operations. The opening operation removes discrete noise points through corrosion and expansion, and the closing operation fills the internal holes of the marking through expansion and corrosion to make the marking edge smoother. The connected domain is extracted from the results after morphological processing, that is, the area composed of the marking point clouds that are adjacent to each other in space. The small connected domains with point clouds less than the preset threshold are eliminated, and the larger complete marking connected domains are retained to obtain the optimized road marking point cloud.

[0059] The road boundary point cloud obtained by preliminary segmentation is optimized. The specific steps are as follows:

[0060] The finite difference method is used to calculate the three-dimensional gradient features of each road boundary point cloud. The central difference format is used to calculate the gradient components of the point cloud in the X, Y, and Z directions respectively. The modulus of the gradient reflects the drastic change of the local point cloud normal vector. The point cloud with a gradient size exceeding the preset threshold is selected as the seed point of the region growing algorithm and marked as the boundary. With the seed point as the center, the point cloud with a gradient size exceeding the threshold and not yet visited in the neighborhood is merged into the current region, and iterative growth is performed until there are no neighboring points that meet the conditions.

[0061] Connected domains are extracted from the boundary regions obtained by region growing, and the number of point clouds and area of ​​each connected domain are calculated. Connected domains with point cloud numbers or areas less than the preset threshold are eliminated, and larger complete road boundary connected domains are retained. Local plane fitting is performed on the point clouds in the selected road boundary connected domains, and the local tangent plane of the boundary point cloud is fitted by the least squares method, the boundary surface is smoothed, the influence of outliers is eliminated, and the optimized road boundary point cloud is obtained.

[0062] After the above-mentioned optimization method for road surface, markings, and boundary point clouds is processed, a refined road element point cloud with outliers removed, smooth edges, and complete semantics can be obtained, providing high-quality data support for subsequent road three-dimensional reconstruction and parametric modeling.

[0063] In an optional embodiment,

[0064] Build a point cloud classifier model, including:

[0065] For each leaf node of the octree, the geometric features and semantic features of the point cloud in the node are calculated. The geometric features include point cloud density, point cloud centroid, eigenvalue and normal vector of the covariance matrix, and the semantic features include average color and average reflection intensity. The main direction of the point cloud in the leaf node is calculated by principal component analysis, and the eigenvector with the smallest eigenvalue is used as the normal vector of the local point cloud. The calculated geometric features and semantic features are combined into the feature vector of the leaf node.

[0066] The feature vectors of all leaf nodes in the octree are input into a three-dimensional convolutional neural network based on an attention mechanism for feature extraction and classification. The network structure of the three-dimensional convolutional neural network includes a three-dimensional convolutional layer, an attention module, a pooling layer, and a fully connected layer; wherein the three-dimensional convolutional layers are connected by residual connections; the attention module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to determine the importance weights of different feature channels output by the three-dimensional convolutional layer through global average pooling and a fully connected layer, and weight each channel of the feature map. The spatial attention submodule is used to determine the importance weights of different spatial positions of the weighted feature map through a convolutional layer and a normalization function, and weight each spatial position of the feature map to obtain an attention-enhanced feature map.

[0067] A three-dimensional convolutional neural network is trained based on the attention-enhanced feature map, wherein a cross entropy loss function and a focal loss function are used as the loss function of the network, and an optimizer and a cosine annealing learning rate scheduling strategy are used to optimize the training of the three-dimensional convolutional neural network to obtain a point cloud classifier model.

[0068] Exemplarily, for the problem of point cloud classification in three-dimensional reconstruction of road scenes, this application proposes a point cloud classifier model based on octree and attention mechanism. First, the octree is used to spatially divide and hierarchically organize large-scale point clouds, and the geometric and semantic features of the point cloud are extracted at the leaf node level of the octree. Then, the extracted leaf node feature vector is input into a three-dimensional convolutional neural network integrated with an attention mechanism, and the category label of the point cloud is obtained through end-to-end feature learning and classification. Finally, guided by cross entropy loss and focal loss, the point cloud classifier model is trained and optimized using an optimizer and a learning rate scheduling strategy. The construction process of the point cloud classifier model is described in detail below.

[0069] Since the 3D point cloud collected from road scenes is huge in scale and unevenly distributed in space, it is difficult to balance efficiency and accuracy by directly processing it. Therefore, this paper uses the classic spatial data structure of octree to hierarchically organize the original point cloud and extract local features to prepare for the subsequent classification task.

[0070] Octree is an adaptive spatial partitioning data structure that organizes and indexes point clouds in layers by recursively dividing the three-dimensional space into eight sub-regions. The root node of the octree represents the global point cloud range, the leaf node represents the local point cloud fragment, and the non-leaf node connects the parent node and the child node. Through the octree, the point cloud is divided into nested cubic units at different spatial scales, which is convenient for divide-and-conquer processing. This paper adopts a top-down, layer-by-layer subdivision strategy to construct an octree of point cloud. The specific steps are as follows:

[0071] Take the bounding box of the original point cloud as the root node of the octree, and calculate the cube center and side length of the root node. Take the root node as the current processing node, and recursively perform the following operations until the number of point clouds of the current node is less than the preset threshold or the depth of the tree reaches the predetermined maximum value:

[0072] Count the number of point clouds in the current node. If the number is greater than the threshold and has not reached the maximum depth, subdivide the current node into eight child nodes. Taking the center of the current node cube as the boundary, divide the cube into two equal parts in the three coordinate axes to form eight child cubes, each of which corresponds to a child node of the octree. Divide the point cloud in the current node into the child nodes to which it belongs, and add the non-empty child nodes to the next layer of the queue to be processed. Set the eight child node pointers of the current node to point to the corresponding child nodes.

[0073] After all nodes in the queue are processed, the octree is constructed. The leaf node set contains the hierarchical organization results of the original point cloud.

[0074] After the octree is constructed, the geometric features and semantic features of the point cloud are extracted for each leaf node, and the feature vector of the node is synthesized for subsequent classification model training. For the local point cloud within the leaf node, the following geometric features are extracted:

[0075] Point cloud density: Calculate the average number of point clouds within a unit cube volume to reflect the density of the local point cloud distribution. Point cloud centroid: Calculate the arithmetic mean center position of the point cloud to reflect the spatial position of the local point cloud. Eigenvalue of the covariance matrix: Calculate the three eigenvalues ​​of the point cloud covariance matrix to reflect the degree of anisotropy of the point cloud distribution. Normal vector: Calculate the main direction of the point cloud through principal component analysis, and use the eigenvector with the smallest eigenvalue as the normal vector of the point cloud to reflect the direction information of the local point cloud. For the local point cloud in the leaf node, extract the following semantic features:

[0076] Average color: The arithmetic average of the three RGB channels of the point cloud is calculated to reflect the main color tone of the area. Average reflection intensity: The arithmetic average of the reflection intensity values ​​of the point cloud is calculated to reflect the reflection characteristics of the material in the area. Finally, the geometric features and semantic features extracted from the leaf node are concatenated into a feature vector of fixed dimension as the comprehensive feature representation of the node, which is used for the training and reasoning of the point cloud classifier.

[0077] This application inputs the feature vector of the octree leaf node into the attention-enhanced three-dimensional convolutional neural network to realize the point cloud classification function. The network mainly includes a three-dimensional convolution layer, an attention module, a pooling layer, and a fully connected layer. The three-dimensional convolution is responsible for extracting the spatial features of the point cloud, the attention module adaptively enhances the significant features, the pooling layer gradually reduces the size of the feature map, and the fully connected layer outputs the category to which the point cloud belongs.

[0078] The 3D convolution layer extracts the spatial structure and features of the local point cloud neighborhood by sliding the 3D convolution kernel on the point cloud feature map. Compared with the traditional 2D convolution, 3D convolution can model the spatial information of the point cloud well. Assume that the size of the 3D convolution kernel is dxdxd, the number of channels of the output feature map of the previous layer is m, and the number of channels of the output feature map of this layer is n, then the total number of convolution kernel parameters is dxdxdxmxn.

[0079] In order to further improve the ability of 3D convolution to extract features, this paper adopts a residual connection network structure. Two adjacent 3D convolution layers are connected by short circuits, and the input of the previous layer is directly added to the output of the next layer, so that the network can learn the transformation relationship from the input feature map to the residual feature map. On the one hand, the residual connection increases the expression ability of the network, and on the other hand, it also alleviates the problem of network degradation, which is conducive to training deeper networks.

[0080] Based on the feature map extracted by three-dimensional convolution, the attention module adaptively adjusts the feature map from the two dimensions of channel and space, enhances the significant features, suppresses the non-significant features, and makes the network pay more attention to the key factors of the classification task. The attention module includes two sub-modules: channel attention and spatial attention.

[0081] Channel attention submodule: Perform global average pooling on the feature map output by the three-dimensional convolutional layer to obtain a descriptor of size 1x1x1xC, where C is the number of channels of the input feature map; pass the descriptor through two fully connected layers and a Sigmoid activation function to learn the correlation between channels and output a weight vector of size C; multiply the weight vector by the original feature map channel by channel to obtain a feature map weighted by channel attention.

[0082] Spatial attention submodule: On the feature map after channel attention weighting, maximum pooling and average pooling are performed along three spatial directions respectively to obtain two spatial descriptors of size HxWxD; the two spatial descriptors are concatenated, and the significance of different spatial positions in the feature map is learned through a three-dimensional convolutional layer and a Sigmoid function, and a spatial weight matrix of size HxWxD is output; the spatial weight matrix is ​​element-wise multiplied with the feature map after channel attention weighting to obtain a feature map enhanced with spatial attention.

[0083] Through the synergy of channel attention and spatial attention, the network can adaptively focus on feature channels and salient areas that have a greater impact on classification tasks, suppress redundant features, and improve feature discrimination.

[0084] After building a three-dimensional convolutional neural network, it is necessary to train the network model from end to end and optimize the model parameters so that it can accurately classify point cloud samples.

[0085] Loss function design: This paper uses the cross entropy loss function and the focal loss function as the optimization objectives of the network. Among them, the cross entropy loss measures the difference between the predicted category distribution and the true distribution, and the focal loss dynamically adjusts the sample weights to make the difficult samples receive more attention. The weighted combination of the two loss functions makes the network pay more attention to the overall classification accuracy and the correctness of difficult examples during the training process.

[0086] Optimizer and learning rate strategy: The Adam optimizer is used to iteratively update the network weights and optimize the model with an adaptive learning rate. At the same time, the cosine annealing learning rate scheduling strategy is adopted. As the number of iterations increases, the learning rate shows a periodic decay trend, which makes the optimization process oscillate back and forth near the local optimal value, jumping out of the local optimal value and entering a better global optimal value.

[0087] Hyperparameter selection: Hyperparameters such as batch size, initial learning rate, and regularization coefficient are tuned through cross-validation to control the risk of overfitting and improve the generalization performance of the model.

[0088] After the above training and optimization process, a point cloud classifier model with excellent performance can be obtained. This model uses the feature vector of the octree leaf node as input, and uses the attention-enhanced three-dimensional convolutional neural network to accurately classify the point cloud, realizing automatic recognition and semantic understanding of different categories of road scene point clouds.

[0089] In an optional embodiment,

[0090] The preliminary segmentation results are optimized according to the confidence threshold, and the navigation path of the inspection robot is planned according to the optimization results and the navigation path data is generated, including:

[0091] The preliminary segmentation results are optimized according to the confidence threshold, the point cloud labels with confidence lower than the threshold are reallocated, and the point cloud labels with confidence higher than the threshold are kept unchanged to obtain the optimized segmentation results; the optimized segmentation results are processed by the conditional random field model to obtain the refined segmentation results including four categories: road surface, road marking, road boundary and roadside obstacles;

[0092] The road centerline is extracted from the road surface point cloud in the refined segmentation result, and the road surface point cloud is aggregated into continuous road clusters through Euler clustering, each road cluster represents a continuous road segment; for each road cluster, the main direction of the road cluster is extracted through the principal component analysis method, and the point cloud coordinates in each road cluster are projected onto the extracted main direction vector to obtain the one-dimensional distribution of the road cluster in the main direction; the one-dimensional distribution of the road cluster in the main direction is analyzed by sliding windows to obtain the preliminary extraction result of the road centerline;

[0093] Establish the geometric constraint relationship between the preliminary extraction result of the road centerline, the road markings and the road boundaries in the refined segmentation results; construct a least squares optimization problem, in which the point coordinates on the road centerline are optimized variables and the geometric constraint relationship is the error term; use the Levenberg-Marquardt algorithm to solve the least squares optimization problem, and fit the precise road centerline after fusion optimization; post-process the extracted precise road centerline, including breakpoint connection, smoothing and simplified downsampling, to obtain the post-processed road centerline; plan the navigation path of the inspection robot and generate navigation path data based on the post-processed road centerline and the roadside obstacles in the refined segmentation results.

[0094] For example, after obtaining the refined semantic segmentation results of the point cloud, it is necessary to further optimize the segmentation results, extract the road centerline, and plan the robot's navigation path based on the road and obstacle information. Based on the optimization of the segmentation results, this application proposes a road centerline extraction method that integrates the geometric constraints of road semantic elements, and generates a path suitable for robot navigation.

[0095] Since the classifier model used for point cloud segmentation outputs the category probability of each point cloud, it is necessary to optimize the preliminary segmentation results based on the probability value to improve the confidence and consistency of the segmentation labels.

[0096] For each category label of the point cloud in the preliminary segmentation result, calculate its corresponding confidence value, that is, the probability that the classifier predicts that the point cloud belongs to the current category. Compare the confidence with the preset threshold, reassign the category label to the point cloud with confidence below the threshold, and keep the original label unchanged for the point cloud with confidence above the threshold.

[0097] The strategy for reallocating labels is as follows: For point clouds with low confidence, count the category labels of point clouds in their neighborhood, and select the label with the highest frequency as the new label of the point cloud. The definition of the neighborhood can be a spherical area with a fixed radius centered on the point cloud, or a set of K nearest neighbor point clouds centered on the point cloud. Through the voting mechanism of point cloud categories in the neighborhood, the labels of point clouds with low confidence can be corrected to labels that are more consistent with the surrounding environment, thereby improving the smoothness of the segmentation results.

[0098] In order to further improve the spatial consistency of segmentation labels, after optimizing the confidence threshold, the conditional random field model is used to globally optimize the segmentation results. The conditional random field regards point cloud segmentation as a probabilistic graph model, and by defining the similarity between point clouds and the compatibility between labels, the category labels of all point clouds are jointly inferred to obtain the globally optimal segmentation result.

[0099] The energy function of the conditional random field contains two terms: data term and smoothing term. The data term measures the degree of consistency between the category label of each point cloud and its observed features, that is, the confidence of the point cloud classifier output; the smoothing term measures the consistency of adjacent point cloud labels, encouraging spatially close point clouds to be assigned the same category label. By minimizing the energy function of the conditional random field, the globally optimal point cloud label configuration can be obtained, making the segmentation result smoother in the local area and consistent with the overall semantics of the scene.

[0100] After conditional random field optimization, the point cloud is divided into four categories: road surface, road markings, road boundaries and roadside obstacles. Each type of point cloud has high intra-class consistency and good semantic integrity, laying the foundation for subsequent analysis and application.

[0101] After obtaining the refined semantic segmentation results, the road centerline is extracted from the road surface point cloud as a reference for robot navigation path planning. The main steps of road centerline extraction are as follows.

[0102] Euler clustering is performed on the point clouds belonging to the road surface category in the refined segmentation results, and the spatially adjacent road point clouds are aggregated into independent clusters. Euler clustering iteratively merges point clouds with a distance less than a given threshold into the same cluster until all road surface point clouds are divided into corresponding clusters.

[0103] Each point cloud cluster represents a continuous road in the road network, and there may be intersections or bifurcations between different point cloud clusters. Subsequent centerline extraction for each road cluster can obtain the skeleton structure of the complete road network.

[0104] The principal component analysis is performed on the point cloud in each road cluster to extract the main direction of the point cloud distribution. The principal component analysis performs eigenvalue decomposition on the point cloud covariance matrix and projects the point cloud onto the eigenvector with the largest variance. The eigenvector is the main direction of the point cloud, which reflects the extension direction of the road cluster.

[0105] By projecting the coordinates of all point clouds in the road cluster onto the main direction vector, we can get the one-dimensional distribution of the point cloud in the main direction. By analyzing the distribution law of the point cloud in the main direction, we can approximate the skeleton shape of the road cluster.

[0106] A sliding window analysis is performed on the road cluster point cloud along the main direction, and the center position of the point cloud within the window is extracted as a preliminary approximation of the road centerline.

[0107] Assume that the size of the sliding window is a fixed step length, starting from the starting point of the main direction, take the average of the X, Y, and Z coordinates of the point cloud in the window, obtain the coordinates of the center of the window, and record it as a control point on the center line of the road. With the step length as the interval, translate the sliding window along the main direction, repeat the above process until the entire main direction is covered, generate a series of central control points, and constitute a preliminary representation of the center line of the road.

[0108] The road centerline extracted by sliding window analysis is close to the real skeleton shape of the road cluster, but the position of the control points may be biased and needs further optimization.

[0109] In order to obtain a more accurate and smoother road centerline, the semantic elements such as road markings and road boundaries in the segmentation results are comprehensively utilized to optimize the initially extracted centerline.

[0110] First, the geometric constraint relationship between the road centerline control point and the road marking and road boundary point cloud is established. Assume that in an ideal situation, the road centerline should be equidistant from the parallel lane markings and located at the center of the left and right road boundaries. Based on this, the distance error term from the road centerline control point to the lane markings and road boundaries is constructed.

[0111] Then, the road centerline optimization problem is transformed into a least squares optimization problem. The coordinates of the centerline control points are taken as the variables to be optimized, and the distance errors from the control points to the lane markings and road boundaries are taken as the observation equation to construct the least squares error function. The least squares problem is solved using the Levenberg-Marquardt algorithm, and the corrected coordinates of the road centerline control points are obtained through iterative optimization to better meet the geometric constraints of the road semantic elements.

[0112] After global optimization by integrating road semantic elements, the position of the road centerline is more accurate, the shape is smoother, and it is more consistent with the real road skeleton.

[0113] The optimized road centerline is post-processed to meet the needs of practical applications. It mainly includes the following steps:

[0114] Breakpoint connection: Detect the distance between the centerline control points. For distance spans exceeding the threshold, the centerline is considered to have a breakpoint. Additional control points are interpolated at the breakpoints to make the centerline continuous. Smoothing: Smoothing and filtering the centerline control points to remove high-frequency jitter of the control point coordinates and make the centerline shape smoother. Gaussian smoothing, B-spline curve fitting and other methods can be used. Simplified downsampling: Downsample the centerline control points according to the actual accuracy required by the application to reduce the number of control points. By setting a distance threshold, redundant points whose distance between adjacent control points is less than the threshold are removed. After the above post-processing steps, the final road centerline representation is obtained as the input for the inspection robot's navigation path planning.

[0115] According to the extracted road centerline and the roadside obstacle information in the refined segmentation results, the robot's navigation path is planned and the path data is generated. The extracted road centerline is used as the skeleton to generate the topological structure of the path. By sampling the centerline control points, a series of navigation path nodes are generated. Adjacent nodes are connected to form path segments, and multiple path segments are connected end to end to form a complete path.

[0116] On the basis of path nodes and path segments, semantic attributes are further annotated. For example, path nodes at road intersections are marked as decision points, and path segments connecting roads in different directions are marked as turning segments. These semantic attributes provide clues for subsequent navigation decisions.

[0117] In order for the robot to accurately drive along the planned path, it is necessary to generate the geometric information of the path, mainly including node coordinates, node tangent direction, path width, etc. The node coordinates can be directly obtained from the road centerline control points. The node tangent direction is approximated by calculating the connecting direction of adjacent control points. The path width is determined according to the distance between the left and right road boundary point clouds in the segmentation result. The path geometry information is combined with the path topology structure to form a complete navigation path representation, which can be used to guide the robot's motion control.

[0118] On the basis of generating the navigation path, the path is optimized with reference to the roadside obstacles in the segmentation results to avoid the obstacle area. For each path node, the point cloud of the roadside obstacle is detected within a certain search range with it as the center. If there is an obstacle, a safe passage that can be bypassed is found on both sides of the path node, and the position of the path node is adjusted accordingly to offset it into the safe passage. After obstacle avoidance optimization, the navigation path can guide the robot away from roadside obstacles and improve driving safety. At the same time, the optimized path still maintains consistency with the road topology, ensuring that the robot moves in the correct direction of the road.

[0119] The planned navigation path is converted into a data format executable by the robot to guide the robot's autonomous movement. The content of the navigation path data includes: path node ID, node type, node coordinates, node tangent direction, path segment ID, path segment start and end node ID, path segment length, path segment width, steering information, etc. This information is organized and stored in a format that can be read by the robot control system, such as XML, JSON, etc. The robot control system controls the robot to drive autonomously along the planned path based on the navigation path data to complete the road inspection task. When encountering unknown obstacles, the robot can dynamically adjust the local path based on sensor feedback to ensure safe passage.

[0120] In summary, this application proposes a method for extracting the road centerline and planning the robot navigation path based on the point cloud segmentation results. The method first optimizes the point cloud segmentation results based on the confidence threshold and conditional random field to obtain the refined semantic segmentation of the road surface, markings, boundaries and obstacles. Then, the centerline skeleton is extracted from the road surface point cloud, and the centerline is optimized by integrating semantic elements such as road markings and boundaries to obtain an accurate and smooth road centerline. Finally, based on the road centerline and combined with the roadside obstacle information, the robot's navigation path is planned, and path data is generated to guide the robot to drive autonomously.

[0121] In an optional embodiment,

[0122] The optimized segmentation results are processed using the conditional random field model to obtain refined segmentation results of four categories including road surface, road markings, road boundaries and roadside obstacles, including:

[0123] According to the optimized segmentation results, a conditional random field model is constructed to transform the point cloud segmentation problem into an inference problem of the conditional random field: each point cloud in the optimized segmentation result is used as a node in the conditional random field, and the category label of the node is the random variable to be inferred; the energy function of the conditional random field is defined, including unary terms and binary terms, where the unary term represents the likelihood probability that the point cloud belongs to a certain category, and the binary term represents the prior probability that adjacent point clouds belong to the same category;

[0124] Extract geometric features and texture features of each point cloud, the geometric features include three-dimensional coordinates, normal vectors, curvature and eigenvalues, and the texture features include color, reflection intensity and texture gradient; use the extracted geometric features and texture features to model the feature distribution of each category through a Gaussian mixture model, and calculate the unary likelihood probability in the conditional random field energy function; calculate the spatial distance weight and appearance similarity weight between point clouds, the spatial distance weight is calculated based on the Euclidean distance and normal vector angle between point clouds, and the appearance similarity weight is calculated based on the color distance and reflection intensity difference of the point clouds; perform weighted summation of the spatial distance weight and the appearance similarity weight to obtain a comprehensive similarity measure between point clouds, and transform the similarity through a Gaussian kernel function as a binary prior probability in the conditional random field energy function;

[0125] The constructed conditional random field model is used to infer and optimize the optimized segmentation results: the belief propagation algorithm is used to perform iterative inference on the conditional random field model. In each iteration, the confidence distribution of the node belonging to each category is updated according to the unary likelihood probability of each node and the binary prior probability from the adjacent nodes; after the iteration, the category with the highest confidence is selected as the optimal label for each node to obtain the refined segmentation result of the point cloud.

[0126] For example, in order to further improve the accuracy and consistency of point cloud segmentation, on the basis of optimizing the segmentation results based on the confidence threshold, a conditional random field model is introduced to globally optimize the segmentation results, and a refined semantic segmentation result including four categories: road surface, road markings, road boundaries and roadside obstacles is obtained.

[0127] The problem of semantic segmentation of point clouds is transformed into an inference problem of conditional random fields. Conditional random fields are a probabilistic graphical model that models the joint probability distribution of variables by defining the relationship between random variables. In the point cloud segmentation task, each point cloud is regarded as a node in the conditional random field, and the category label of the node is the random variable to be inferred. By reasonably designing the energy function of the conditional random field, the observed features and contextual relationships of the point cloud can be considered, and the category labels of all point clouds can be globally inferred.

[0128] The energy function of the conditional random field consists of two parts: a unary term and a pairwise term. The unary term represents the likelihood that each point cloud node belongs to a certain category, that is, the possibility of judging whether the point cloud belongs to a certain category based on its own characteristics. The pairwise term represents the prior probability that adjacent point cloud nodes belong to the same category, that is, spatially close point clouds should tend to have the same category label. By balancing the unary term and the pairwise term, the conditional random field can obtain the global optimal point cloud category label configuration.

[0129] In order to construct the energy function of the conditional random field, it is necessary to extract the feature vector describing the geometric shape and appearance characteristics of each point cloud. The geometric features and texture features are connected into a high-dimensional feature vector as the observation feature of the point cloud node, which is used to calculate the unary term of the conditional random field.

[0130] The energy function of the conditional random field defines the joint probability distribution of the point cloud category labels. The unary term of the energy function represents the likelihood probability that the point cloud node belongs to each category, which can be estimated by modeling the feature distribution of each category. The commonly used modeling method is the Gaussian mixture model, which uses a weighted combination of multiple Gaussian distributions to fit the feature distribution of each category. For a given point cloud node, the likelihood probability of its feature vector under the Gaussian mixture model of each category is calculated as the value of the unary term.

[0131] The binary term of the energy function represents the prior probability that adjacent point cloud nodes belong to the same category, which encourages point clouds that are close in space and have similar features to be assigned the same category label. First, it is necessary to define the similarity measure between point cloud nodes. On the one hand, the similarity measure considers the distance of the point cloud in three-dimensional space, and calculates the spatial distance weight through the Euclidean distance and the angle between the normal vectors; on the other hand, it considers the difference in the appearance features of the point cloud, and calculates the appearance similarity weight through the color distance and the difference in reflection intensity. The spatial distance weight and the appearance similarity weight are weighted and summed to obtain a comprehensive similarity measure.

[0132] In order to convert the similarity measure into a probability value, the similarity is transformed by the Gaussian kernel function. The Gaussian kernel function maps the similarity measure to between 0 and 1. The higher the similarity, the greater the probability that the point cloud pair is assigned the same label. The bandwidth parameter of the Gaussian kernel function controls the scale of the transformation from similarity to probability, and needs to be appropriately selected according to the distribution characteristics of the point cloud.

[0133] Through the definition of unary and binary terms, the energy function of the conditional random field establishes the probabilistic dependency between the point cloud node category labels and their observed features and contextual relationships, laying the foundation for global inference of point cloud semantic labels.

[0134] After building the conditional random field model, it is necessary to optimize the category labels of the point cloud nodes through the inference algorithm to obtain the global optimal semantic segmentation result. Commonly used conditional random field inference algorithms include belief propagation and mean field approximation.

[0135] Taking the belief propagation algorithm as an example, the category label of each point cloud node is optimized through an iterative message passing process. In each iteration, each node calculates the likelihood probability of the univariate term based on its own observed features, and receives the binary prior probability information from the adjacent nodes. The node fuses the information of the univariate term and the binary term and updates its confidence distribution for each category. The confidence distribution represents the probability that the node belongs to each category, which is initially determined by the category probability predicted by the classifier.

[0136] Message passing is carried out in the node network of the conditional random field, and each node sends messages carrying its own category tendency to its neighboring nodes. After aggregating messages from neighboring nodes, the node updates its own category label based on the confidence distribution and neighborhood information. After multiple iterations, the confidence distribution of the node gradually converges and reaches the global optimal state.

[0137] After the iteration, the category with the highest confidence is selected as the final semantic label for each point cloud node. The semantic segmentation result of the point cloud optimized by conditional random field inference fully considers the observation characteristics and spatial contextual relationship of the point cloud. The category label is smoother and more consistent in the local area and conforms to the overall semantic structure of the scene.

[0138] By introducing the conditional random field model, the accuracy and robustness of point cloud semantic segmentation have been significantly improved. The segmentation results based on conditional random field optimization include four categories: road surface, road markings, road boundaries, and roadside obstacles. The point cloud of each category has high intra-class consistency and semantic integrity, which provides a good foundation for subsequent road analysis and robot navigation.

[0139] In an optional embodiment,

[0140] According to the post-processed road centerline and roadside obstacles in the refined segmentation results, the navigation path of the inspection robot is planned and the navigation path data is generated, including:

[0141] According to the post-processed road centerline and the roadside obstacles in the refined segmentation result, the road centerline point cloud and the roadside obstacle point cloud are extracted; the extracted road centerline point cloud is smoothed by using a B-spline curve to obtain a smoothed road centerline; the extracted roadside obstacle point cloud is clustered by using a Euclidean distance clustering algorithm to obtain independent obstacle objects, and a bounding box model is performed on each obstacle object, and a rectangular bounding box of the obstacle is fitted by a principal component analysis method to obtain the position, size and direction information of the obstacle;

[0142] Based on the smoothed road centerline and combined with the bounding box information of the obstacle, the navigation path of the inspection robot is planned, and the smoothed road centerline is discretized into a series of navigation waypoints, each of which contains position, tangent and curvature information. For each navigation waypoint, it is checked whether there are roadside obstacles around it. If there are obstacles, the navigation waypoint is offset in the normal direction according to the position and size of the obstacle, and the discrete navigation waypoints are connected into a continuous navigation path curve through smoothing spline interpolation;

[0143] Perform feasibility analysis and optimization on the planned navigation path curve, check whether the curvature radius and slope of the navigation path curve meet the kinematic constraints of the inspection robot, make local adjustments and optimizations to the sections that do not meet the constraints, and use a dynamic programming algorithm to globally optimize the navigation path curve. The optimization goals include minimizing the length of the navigation path and maximizing the smoothness to obtain an optimized navigation path; convert the optimized navigation path into control instructions for the inspection robot, and generate navigation path data including position, speed, acceleration and steering angle.

[0144] For example, based on the results of road centerline extraction and refined semantic segmentation, this application proposes a navigation path planning method for a road inspection robot. By comprehensively utilizing the information of the road centerline and roadside obstacles, a safe, smooth, and efficient robot motion trajectory is planned, and corresponding navigation path data is generated, providing a basis for the robot's autonomous navigation.

[0145] The basis of navigation path planning is to obtain accurate information about the road centerline and roadside obstacles. First, the road centerline point cloud is obtained from the results of road centerline extraction, and the point cloud marked as roadside obstacles is extracted from the results of refined semantic segmentation.

[0146] For the extracted road centerline point cloud, due to the influence of point cloud discretization and noise, the centerline often has jitter and discontinuity problems. In order to obtain a smooth and continuous road centerline, the B-spline curve is used to fit and smooth the centerline point cloud. The B-spline curve is a parametric curve that represents the curve shape through a combination of control points and basis functions. It has the advantages of local support and shape retention. By selecting appropriate control points and spline orders, the B-spline curve is fitted using the least squares method to obtain the smoothed road centerline. The smoothed centerline maintains the overall trend of the road direction and removes local noise interference.

[0147] For the extracted roadside obstacle point cloud, it is necessary to further analyze the location, size and direction information of the obstacle to provide a reference for navigation path planning. The Euclidean distance clustering algorithm is used to cluster the roadside obstacle point cloud, and the point clouds with similar spatial positions are divided into independent obstacle objects. When clustering, the granularity of clustering is controlled by setting the distance threshold, which ensures that the point clouds of the same obstacle are classified into one category and avoids the mistaken merging of different obstacles.

[0148] For each obstacle object obtained by clustering, its position, size and direction are estimated by bounding box modeling. The principal component analysis (PCA) method is used to fit the rectangular bounding box of the obstacle point cloud. By performing eigenvalue decomposition on the covariance matrix of the point cloud, three main directions and corresponding eigenvalues ​​are obtained. The size of the eigenvalue reflects the degree of dispersion of the point cloud in each main direction and can be used to estimate the size of the bounding box. The maximum and minimum coordinate values ​​of the point cloud in the three main directions are used as the vertex coordinates of the bounding box to obtain the rectangular bounding box model of the obstacle. The center position, length, width, height and direction information of the bounding box can be directly obtained from the model for subsequent navigation path planning.

[0149] Based on the smoothed road centerline and obstacle bounding box information, the inspection robot's navigation path planning is carried out. The navigation path needs to move along the road centerline while avoiding obstacles on the roadside, while satisfying the robot's kinematic constraints to ensure the feasibility and safety of the path.

[0150] First, the smoothed road centerline is discretized into a series of navigation waypoints. Through equidistant sampling, a certain number of waypoints are uniformly selected on the centerline as the basis of the navigation path. Each navigation waypoint contains information such as position, tangent, and curvature. The position information represents the coordinates of the waypoint in three-dimensional space, the tangent information represents the tangent direction of the road centerline at the waypoint, and the curvature information represents the curvature of the road at the waypoint, which can be estimated by calculating the first-order and second-order derivatives of the centerline.

[0151] For each navigation waypoint, check whether there are roadside obstacles around it. Set a search area of ​​a certain range around the navigation waypoint as the center, and determine whether there is an obstacle bounding box intersecting with it in the search area. If there is an intersecting obstacle, the navigation waypoint needs to be partially adjusted to avoid the obstacle. The adjustment strategy is to offset along the normal direction of the road centerline at the navigation waypoint. The size of the offset is determined according to the size and position relationship of the obstacle bounding box, and it is necessary to ensure that there is enough safety distance between the adjusted navigation waypoint and the obstacle.

[0152] The local adjusted navigation waypoints are used as control points, and the discrete waypoints are connected into a continuous and smooth navigation path curve using the smoothing spline interpolation method. Smoothing spline interpolation generates smooth and continuous curve segments between control points by minimizing the bending energy of the curve, thus obtaining a complete navigation path.

[0153] In order to ensure the feasibility and optimality of the navigation path, the planned path needs to be analyzed and optimized. First, check whether the curvature radius and slope of the navigation path meet the robot's kinematic constraints. For sections where the curvature radius is less than the robot's minimum turning radius, or where the slope exceeds the robot's climbing ability, local path adjustment and optimization are required. The adjusted path can meet the constraints by reducing the local waypoint density, increasing the offset, and other measures.

[0154] On the basis of satisfying local constraints, the navigation path is globally optimized using the dynamic programming algorithm. The optimization goal is to make the navigation path take into account the length and smoothness of the path while meeting the safety and feasibility. By setting a suitable cost function, considering factors such as path length, curvature change, obstacle distance, etc., dynamic programming is used to search for the optimal connection method between waypoints to obtain the global optimal navigation path.

[0155] The navigation path obtained after planning and optimization needs to be converted into control instructions and data formats that can be executed by the robot. The navigation path data includes a series of timestamps and corresponding robot status information. The status information includes the robot's position coordinates, speed, acceleration, and steering angle at each moment.

[0156] By discretizing the navigation path curve, we can get the position of the waypoints at a series of timestamps. According to the position difference and time interval between adjacent waypoints, we can calculate the velocity vector at each waypoint. By differential velocity vectors, we can get the acceleration information. The steering angle can be obtained by calculating the change in the tangent direction between adjacent waypoints.

[0157] The information such as timestamp, position, speed, acceleration and steering angle is packaged into a standardized data format to generate a complete navigation path data file. The data file can be stored in text format or binary format, which is convenient for the robot control system to read and parse. At the same time, the navigation path data can also be visualized to generate intuitive path diagrams and animations for users to view and analyze.

[0158] Through the above navigation path planning and data generation method, the transformation from the road centerline and roadside obstacle information to the robot's executable navigation path is realized. The planned navigation path comprehensively considers the road environment, obstacle constraints and robot motion characteristics, and has the characteristics of safety, smoothness and feasibility. The navigation path data provides a basis for the robot's autonomous navigation, enabling the robot to accurately and efficiently complete the road inspection task along the planned path.

[0159] Figure 2 FIG. 1 is a schematic diagram of a road scene point cloud recognition system applicable to an inspection robot according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0160] The first unit is used to collect a three-dimensional point cloud of a road scene, wherein the three-dimensional point cloud includes point cloud information of a road surface, road markings, road boundaries, and roadside obstacles; pre-process the three-dimensional point cloud, down-sample the point cloud using an adaptive voxel filtering algorithm, and remove outliers using a statistical filtering algorithm to obtain a pre-processed three-dimensional point cloud;

[0161] The second unit is used to segment the preprocessed three-dimensional point cloud using the regional growing method and the Euclidean clustering algorithm to obtain the preliminary segmentation results of the road surface, road markings, road boundaries and roadside obstacles; the road surface point cloud, road marking point cloud and road boundary point cloud in the preliminary segmentation results are respectively feature extracted, and the preset algorithm is used to optimize them to obtain the optimized road surface point cloud, road marking point cloud and road boundary point cloud; the optimized road surface point cloud, road marking point cloud and road boundary point cloud are respectively mapped to different levels of the octree to construct a three-dimensional point cloud map based on the octree;

[0162] The third unit is used to build a point cloud classifier model; the three-dimensional point cloud map of the octree is input into the point cloud classifier model for prediction, the category label and confidence of each point cloud are obtained, the preliminary segmentation result is optimized according to the confidence threshold, the navigation path of the inspection robot is planned according to the optimization result and the navigation path data is generated, and the navigation path data is transmitted to the motion control module of the robot so that the robot can perform inspection.

[0163] According to a third aspect of the embodiments of the present invention,

[0164] An electronic device is provided, comprising:

[0165] processor;

[0166] a memory for storing processor-executable instructions;

[0167] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0168] According to a fourth aspect of the embodiments of the present invention,

[0169] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0170] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A road scene point cloud recognition method suitable for inspection robots, characterized in that: include: Collecting a three-dimensional point cloud of a road scene, wherein the three-dimensional point cloud includes point cloud information of a road surface, road markings, road boundaries, and roadside obstacles; Preprocessing the three-dimensional point cloud, downsampling the point cloud by using an adaptive voxel filtering algorithm, and removing outliers by using a statistical filtering algorithm to obtain a preprocessed three-dimensional point cloud; The preprocessed three-dimensional point cloud is segmented using the regional growing method and the Euclidean clustering algorithm to obtain the preliminary segmentation results of the road surface, road markings, road boundaries and roadside obstacles; the road surface point cloud, road marking point cloud and road boundary point cloud in the preliminary segmentation results are respectively feature extracted, and the preset algorithm is used to optimize them to obtain the optimized road surface point cloud, road marking point cloud and road boundary point cloud; the optimized road surface point cloud, road marking point cloud and road boundary point cloud are respectively mapped to different levels of the octree to construct a three-dimensional point cloud map based on the octree; Construct a point cloud classifier model; input the octree's three-dimensional point cloud map into the point cloud classifier model for prediction, obtain the category label and confidence of each point cloud, optimize the preliminary segmentation results according to the confidence threshold, plan the inspection robot's navigation path according to the optimization results and generate navigation path data, and transmit the navigation path data to the robot's motion control module so that the robot can perform inspections; The preliminary segmentation results are optimized according to the confidence threshold, and the navigation path of the inspection robot is planned according to the optimization results and the navigation path data is generated, including: The preliminary segmentation results are optimized according to the confidence threshold, the point cloud labels with confidence lower than the threshold are reallocated, and the point cloud labels with confidence higher than the threshold are kept unchanged to obtain the optimized segmentation results; the optimized segmentation results are processed by the conditional random field model to obtain the refined segmentation results including four categories: road surface, road marking, road boundary and roadside obstacles; The road centerline is extracted from the road surface point cloud in the refined segmentation result, and the road surface point cloud is aggregated into continuous road clusters through Euler clustering, each road cluster represents a continuous road segment; for each road cluster, the main direction of the road cluster is extracted through the principal component analysis method, and the point cloud coordinates in each road cluster are projected onto the extracted main direction vector to obtain the one-dimensional distribution of the road cluster in the main direction; the one-dimensional distribution of the road cluster in the main direction is analyzed by sliding windows to obtain the preliminary extraction result of the road centerline; Establish the geometric constraint relationship between the preliminary extraction result of the road centerline, the road markings and the road boundaries in the refined segmentation results; construct a least squares optimization problem, in which the point coordinates on the road centerline are optimized variables and the geometric constraint relationship is the error term; use the Levenberg-Marquardt algorithm to solve the least squares optimization problem, and fit the precise road centerline after fusion optimization; post-process the extracted precise road centerline, including breakpoint connection, smoothing and simplified downsampling, to obtain the post-processed road centerline; plan the navigation path of the inspection robot and generate navigation path data based on the post-processed road centerline and the roadside obstacles in the refined segmentation results.

2. The method according to claim 1, characterized in that The three-dimensional point cloud is preprocessed, the point cloud is downsampled by an adaptive voxel filtering algorithm, and outliers are removed by a statistical filtering algorithm to obtain a preprocessed three-dimensional point cloud, including: Downsampling the three-dimensional point cloud using an adaptive voxel filtering algorithm, determining the voxel size according to the local density of the point cloud, recursively dividing the point cloud space into eight sub-voxels until the number of point clouds in the sub-voxel is less than a preset threshold or reaches a maximum division depth, and calculating the center of gravity of the point cloud in each non-empty sub-voxel to generate a downsampled point cloud; For the downsampled point cloud, outliers are removed using a statistical filtering algorithm based on the local neighborhood covariance matrix. The local neighborhood covariance matrix of each point is calculated. According to the eigenvalues ​​and eigenvectors of the covariance matrix, a local coordinate system is constructed. The neighborhood points are projected into the local coordinate system, and the Mahalanobis distance of each neighborhood point to the origin of the local coordinate system is calculated. According to the Mahalanobis distance distribution, the outlier threshold is determined, and the neighborhood points with a Mahalanobis distance greater than the threshold are marked as potential outliers. Based on the spatial distribution and connectivity characteristics of the potential outliers, the outlier threshold is adjusted to obtain the optimized potential outlier marking result. For each point in the optimized potential outlier marking result, the proportion of optimized potential outliers in its local neighborhood is counted. If the proportion exceeds the preset threshold, the point is marked as an outlier. The potential outliers and outlier marks are updated through iterative optimization until the marking results converge. All points marked as outliers are removed to obtain the point cloud after removing the outliers. The point cloud around the outliers is repaired and smoothed using local surface reconstruction technology to generate a preprocessed three-dimensional point cloud.

3. The method according to claim 1, characterized in that The road surface point cloud, road marking point cloud and road boundary point cloud in the preliminary segmentation results are respectively feature extracted and optimized using a preset algorithm to obtain the optimized road surface point cloud, road marking point cloud and road boundary point cloud, including: For the road surface point cloud obtained by preliminary segmentation, the principal component analysis algorithm is used to calculate the normal vector and curvature characteristics of each road surface point cloud, where the curvature is calculated by the ratio of the minimum eigenvalue to the sum of the eigenvalues ​​of the point cloud neighborhood covariance matrix; the road plane is fitted using a random sampling consistency algorithm, and the angle between the normal vector of each point cloud and the normal vector of the fitting plane is calculated. The point clouds with angles less than a preset threshold are retained, and the point clouds with angles greater than or equal to the preset threshold are eliminated to obtain the optimized road surface point cloud; For the road marking point cloud obtained by preliminary segmentation, the reflection intensity value of each road marking point cloud is extracted, and the reflection intensity segmentation threshold is determined by using the Otsu threshold method, and the target optimal threshold is determined by maximizing the inter-class variance; the point cloud is binarized and segmented using the target optimal threshold to obtain a binarization result; the binarization result is optimized using morphological opening and closing operations to obtain a morphological optimization result; the connected domain in the morphological optimization result is extracted, the number of pixels in each connected domain is calculated, and the connected domain with a pixel number less than the preset threshold is eliminated to obtain the optimized road marking point cloud; For the road boundary point cloud obtained by preliminary segmentation, the finite difference method is used to calculate the three-dimensional gradient features of each road boundary point cloud, and the central difference format is used to calculate the gradient components of the road boundary point cloud in each direction; the road boundary is extracted using the region growing algorithm, and the point cloud with a gradient value exceeding a preset threshold is used as the seed point, and the point cloud with a gradient value exceeding the preset threshold and not visited in the neighborhood point cloud of the seed point is merged into the same region; for the regional point cloud obtained by growth, the connected domain is extracted, the number of point clouds and the area of ​​each connected domain are calculated, and the connected domains with a point cloud number less than the preset threshold or an area less than the preset threshold are eliminated, and the point clouds in the remaining connected domains are locally plane fitted to obtain the optimized road boundary point cloud.

4. The method according to claim 1, characterized in that: Build a point cloud classifier model, including: For each leaf node of the octree, the geometric features and semantic features of the point cloud in the node are calculated. The geometric features include point cloud density, point cloud centroid, eigenvalue and normal vector of the covariance matrix, and the semantic features include average color and average reflection intensity. The main direction of the point cloud in the leaf node is calculated by principal component analysis, and the eigenvector with the smallest eigenvalue is used as the normal vector of the local point cloud. The calculated geometric features and semantic features are combined into the feature vector of the leaf node. The feature vectors of all leaf nodes in the octree are input into a three-dimensional convolutional neural network based on an attention mechanism for feature extraction and classification. The network structure of the three-dimensional convolutional neural network includes a three-dimensional convolutional layer, an attention module, a pooling layer, and a fully connected layer; wherein the three-dimensional convolutional layers are connected by residual connections; the attention module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to determine the importance weights of different feature channels output by the three-dimensional convolutional layer through global average pooling and a fully connected layer, and weight each channel of the feature map. The spatial attention submodule is used to determine the importance weights of different spatial positions of the weighted feature map through a convolutional layer and a normalization function, and weight each spatial position of the feature map to obtain an attention-enhanced feature map. A three-dimensional convolutional neural network is trained based on the attention-enhanced feature map, wherein a cross entropy loss function and a focal loss function are used as the loss function of the network, and an optimizer and a cosine annealing learning rate scheduling strategy are used to optimize the training of the three-dimensional convolutional neural network to obtain a point cloud classifier model.

5. The method according to claim 1, characterized in that The optimized segmentation results are processed using the conditional random field model to obtain refined segmentation results of four categories including road surface, road markings, road boundaries and roadside obstacles, including: According to the optimized segmentation results, a conditional random field model is constructed to transform the point cloud segmentation problem into an inference problem of the conditional random field: each point cloud in the optimized segmentation result is used as a node in the conditional random field, and the category label of the node is the random variable to be inferred; the energy function of the conditional random field is defined, including unary terms and binary terms, where the unary term represents the likelihood probability that the point cloud belongs to a certain category, and the binary term represents the prior probability that adjacent point clouds belong to the same category; Extract geometric features and texture features of each point cloud, the geometric features include three-dimensional coordinates, normal vectors, curvature and eigenvalues, and the texture features include color, reflection intensity and texture gradient; use the extracted geometric features and texture features to model the feature distribution of each category through a Gaussian mixture model, and calculate the unary likelihood probability in the conditional random field energy function; calculate the spatial distance weight and appearance similarity weight between point clouds, the spatial distance weight is calculated based on the Euclidean distance and normal vector angle between point clouds, and the appearance similarity weight is calculated based on the color distance and reflection intensity difference of the point clouds; perform weighted summation of the spatial distance weight and the appearance similarity weight to obtain a comprehensive similarity measure between point clouds, and transform the similarity through a Gaussian kernel function as a binary prior probability in the conditional random field energy function; The constructed conditional random field model is used to infer and optimize the optimized segmentation results: the belief propagation algorithm is used to perform iterative inference on the conditional random field model. In each iteration, the confidence distribution of the node belonging to each category is updated according to the unary likelihood probability of each node and the binary prior probability from the adjacent nodes; after the iteration, the category with the highest confidence is selected as the optimal label for each node to obtain the refined segmentation result of the point cloud.

6. The method according to claim 1, characterized in that According to the post-processed road centerline and roadside obstacles in the refined segmentation results, the navigation path of the inspection robot is planned and the navigation path data is generated, including: According to the post-processed road centerline and the roadside obstacles in the refined segmentation result, the road centerline point cloud and the roadside obstacle point cloud are extracted; the extracted road centerline point cloud is smoothed by using a B-spline curve to obtain a smoothed road centerline; the extracted roadside obstacle point cloud is clustered by using a Euclidean distance clustering algorithm to obtain independent obstacle objects, and a bounding box model is performed on each obstacle object, and a rectangular bounding box of the obstacle is fitted by a principal component analysis method to obtain the position, size and direction information of the obstacle; Based on the smoothed road centerline and combined with the bounding box information of the obstacle, the navigation path of the inspection robot is planned, and the smoothed road centerline is discretized into a series of navigation waypoints, each of which contains position, tangent and curvature information. For each navigation waypoint, it is checked whether there are roadside obstacles around it. If there are obstacles, the navigation waypoint is offset in the normal direction according to the position and size of the obstacle, and the discrete navigation waypoints are connected into a continuous navigation path curve through smoothing spline interpolation; Perform feasibility analysis and optimization on the planned navigation path curve, check whether the curvature radius and slope of the navigation path curve meet the kinematic constraints of the inspection robot, make local adjustments and optimizations to the sections that do not meet the constraints, and use a dynamic programming algorithm to globally optimize the navigation path curve. The optimization goals include minimizing the length of the navigation path and maximizing the smoothness to obtain an optimized navigation path; convert the optimized navigation path into control instructions for the inspection robot, and generate navigation path data including position, speed, acceleration and steering angle.

7. A road scene point cloud recognition system suitable for an inspection robot, used to implement the method described in any one of claims 1 to 6, characterized in that: include: The first unit is used to collect a three-dimensional point cloud of a road scene, wherein the three-dimensional point cloud includes point cloud information of a road surface, road markings, road boundaries, and roadside obstacles; Preprocessing the three-dimensional point cloud, downsampling the point cloud by using an adaptive voxel filtering algorithm, and removing outliers by using a statistical filtering algorithm to obtain a preprocessed three-dimensional point cloud; The second unit is used to segment the preprocessed three-dimensional point cloud using the regional growing method and the Euclidean clustering algorithm to obtain the preliminary segmentation results of the road surface, road markings, road boundaries and roadside obstacles; the road surface point cloud, road marking point cloud and road boundary point cloud in the preliminary segmentation results are respectively feature extracted, and the preset algorithm is used to optimize them to obtain the optimized road surface point cloud, road marking point cloud and road boundary point cloud; the optimized road surface point cloud, road marking point cloud and road boundary point cloud are respectively mapped to different levels of the octree to construct a three-dimensional point cloud map based on the octree; The third unit is used to build a point cloud classifier model; the three-dimensional point cloud map of the octree is input into the point cloud classifier model for prediction, the category label and confidence of each point cloud are obtained, the preliminary segmentation result is optimized according to the confidence threshold, the navigation path of the inspection robot is planned according to the optimization result and the navigation path data is generated, and the navigation path data is transmitted to the motion control module of the robot so that the robot can perform inspection.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images