Method, device and vehicle for real-time semantic segmentation of three-dimensional point cloud with multi-view fusion

Abstract

The present invention relates to the field of autonomous driving technology, and in particular to a method, device and vehicle for real-time semantic segmentation of three-dimensional point clouds with multi-view fusion, including: registering and splicing the original point cloud data to obtain full-view angle point cloud data centered on the vehicle, converting the basic features of each point in the point cloud into a series of feature vectors, refining the point cloud layer by layer, capturing local details and integrating global context information, performing multi-scale fusion of features according to the point cloud attention mechanism, and outputting the semantic label prediction value of each point through the confidence filtering mechanism. Thus, the problem that the point cloud semantic segmentation method is difficult to adapt to complex and changeable road scenes, the feature expression is incomplete, and there are limitations such as quantization loss, semantic inconsistency, and low computational efficiency is solved, and the semantic segmentation accuracy of point cloud data in autonomous driving scenarios is significantly improved, and the local detail processing and global context information integration in the point cloud semantic segmentation in autonomous driving scenarios are realized.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device and vehicle for real-time semantic segmentation of three-dimensional point clouds with multi-view fusion. Background Art

[0002] As a cutting-edge innovation in the field of transportation, autonomous driving technology is gradually driving the development of intelligent vehicles. It uses sensor arrays, artificial intelligence and advanced algorithms to give vehicles the ability to autonomously perceive the environment, make decisions and drive safely. Therefore, when processing point cloud information, higher requirements are placed on semantic segmentation performance to ensure that the vehicle can accurately identify surrounding objects and road conditions.

[0003] In related technologies, point cloud semantic segmentation methods use supervised learning algorithms. The segmentation accuracy of such algorithms is often limited by artificially set empirical thresholds and is difficult to adapt to complex and changing road scenes. There are also related technologies that perform semantic segmentation based on deep learning, such as projection-based semantic segmentation methods and point-based semantic segmentation methods. Although these methods have achieved good results in some public point cloud datasets, their segmentation performance in road scenes still faces the problem of incomplete feature expression. Summary of the invention

[0004] The present invention provides a method, device and vehicle for real-time semantic segmentation of three-dimensional point clouds with multi-perspective fusion, so as to solve the problems in related technologies that the point cloud semantic segmentation method is difficult to adapt to complex and changeable road scenes, the feature expression is incomplete, and there are limitations such as quantization loss, semantic inconsistency, and low computational efficiency. The method significantly improves the semantic segmentation accuracy of point cloud data in autonomous driving scenarios, realizes the processing of local details and the integration of global context information in point cloud semantic segmentation in autonomous driving scenarios, and helps to realize a safer and more efficient autonomous driving system.

[0005] The first aspect of the present invention provides a method for real-time semantic segmentation of three-dimensional point clouds with multi-perspective fusion, comprising the following steps: based on a preset point cloud semantic segmentation network, obtaining a point cloud data set, and obtaining a feature vector of each point cloud data according to the basic features of each point cloud data; determining a neighborhood point set of each point cloud data and a feature matrix of the neighborhood point set, calculating a feature maximum value of each feature matrix, and obtaining a neighborhood feature vector of each point cloud data according to the feature maximum value of each feature matrix; calculating a feature weight of each point cloud data based on the neighborhood feature vector, and obtaining a final feature vector of each point cloud data according to the feature weight and a preset multi-layer perceptron; calculating a comprehensive confidence score for each point cloud data according to the prediction confidence, category confidence and the final feature vector of each point cloud data at different semantic levels, and obtaining a semantic label prediction result of each point cloud data according to the comprehensive confidence score, and combining the semantic label prediction results to obtain a semantic label data set of the point cloud data set.

[0006] Optionally, in some embodiments, the multi-view fused 3D point cloud real-time semantic segmentation method further includes:

[0007] The frequencies of different categories of point cloud data in each range area are counted, and the pseudo labels of each range area are determined according to the frequencies; and the preset point cloud semantic segmentation network is optimized according to the pseudo labels and a preset overall loss function.

[0008] Optionally, in some embodiments, obtaining the feature vector of each point cloud data according to the basic features of each point cloud data includes: converting the basic features of each point cloud data into 2D (two-dimensional) range features, and reconverting the 2D range features of each point cloud data into 3D (three-dimensional) point features to obtain the feature vector of each point cloud data.

[0009] Optionally, in some embodiments, the feature weight of each point cloud data is calculated based on the neighborhood feature vector, including: mapping the 2D range features and the 3D point features of each point cloud data to a corresponding neighborhood point set, and based on the corresponding neighborhood point set, indexing and aggregating the neighboring points of each point cloud data; and calculating the feature weight of each point cloud data through a preset fully connected layer according to the features of the neighboring points.

[0010] Optionally, in some embodiments, obtaining the final prediction result of each point cloud data according to the comprehensive confidence score includes: comparing the comprehensive confidence scores of different semantic levels, and taking the maximum value of the comprehensive confidence scores as the final prediction result.

[0011] Optionally, in some embodiments, determining the neighborhood point set of each point cloud data and the characteristic matrix of the neighborhood point set includes: judging whether the first point cloud data and the second point cloud data are adjacent in the point space, and whether the first point cloud data and the second point cloud data are adjacent in the range space; if the first point cloud data and the second point cloud data are adjacent in the point space, and the first point cloud data and the second point cloud data are adjacent in the range space, then judging that the first point cloud data and the second point cloud data belong to the same unit space; determining the point cloud data belonging to the same unit space as each point cloud data, and obtaining the neighborhood point set of each point cloud data according to the point cloud data of the same unit space.

[0012] Optionally, in some embodiments, before obtaining the point cloud data set, it also includes: obtaining multiple field of view angle point cloud data of the current vehicle; based on the vehicle coordinate system, aligning and splicing the multiple field of view angle point cloud data through preset displacement vectors and rotation matrices to obtain the point cloud data set.

[0013] The second aspect of the present invention provides a multi-perspective fused three-dimensional point cloud real-time semantic segmentation device, including: an acquisition module, which is used to acquire a point cloud data set based on a preset point cloud semantic segmentation network, and obtain a feature vector of each point cloud data according to the basic features of each point cloud data; a feature aggregation module, which is used to determine a neighborhood point set of each point cloud data and a feature matrix of the neighborhood point set, calculate the feature maximum of each feature matrix, and obtain a neighborhood feature vector of each point cloud data according to the feature maximum of each feature matrix; a feature enhancement module, which is used to calculate a feature weight of each point cloud data based on the neighborhood feature vector, and obtain a final feature vector of each point cloud data according to the feature weight and a preset multi-layer perceptron; a feature selection module, which is used to calculate a comprehensive confidence score of each point cloud data according to the prediction confidence, category confidence and the final feature vector of each point cloud data at different semantic levels, and obtain a semantic label prediction result of each point cloud data according to the comprehensive confidence score, and combine the semantic label prediction results to obtain a semantic label data set of the point cloud data set.

[0014] Optionally, in some embodiments, the multi-perspective fused three-dimensional point cloud real-time semantic segmentation device also includes: a statistical module, used to count the frequency of different categories of point cloud data in each range area, and determine the pseudo-label of each range area based on the frequency; an optimization module, used to optimize the preset point cloud semantic segmentation network based on the pseudo-label and a preset overall loss function.

[0015] Optionally, in some embodiments, the acquisition module is specifically used to: convert the basic features of each point cloud data into two-dimensional 2D range features, and reconvert the 2D range features of each point cloud data into three-dimensional 3D point features to obtain a feature vector of each point cloud data.

[0016] Optionally, in some embodiments, the feature enhancement module is specifically used to: map the 2D range features and the 3D point features of each point cloud data to a corresponding neighborhood point set, and based on the corresponding neighborhood point set, index and aggregate the neighboring points of each point cloud data; and calculate the feature weight of each point cloud data through a preset fully connected layer according to the features of the neighboring points.

[0017] Optionally, in some embodiments, the feature selection module is specifically used to: compare the comprehensive confidence scores of the different semantic levels, and take the maximum value of the comprehensive confidence scores as the final prediction result.

[0018] Optionally, in some embodiments, the feature aggregation module is specifically used to: determine whether the first point cloud data and the second point cloud data are adjacent in the point space, and whether the first point cloud data and the second point cloud data are adjacent in the range space; when the first point cloud data and the second point cloud data are adjacent in the point space, determine that the first point cloud data and the second point cloud data belong to the same unit space; determine the point cloud data that belongs to the same unit space as each point cloud data, and obtain the neighborhood point set of each point cloud data based on the point cloud data of the same unit space.

[0019] Optionally, in some embodiments, before acquiring the point cloud data set, the acquisition module is also used to: acquire multiple field of view angle point cloud data of the current vehicle; based on the vehicle coordinate system, align and splice the multiple field of view angle point cloud data through preset displacement vectors and rotation matrices to obtain the point cloud data set.

[0020] A third aspect of the present invention provides a vehicle, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for real-time semantic segmentation of three-dimensional point clouds with multi-perspective fusion as described in the above embodiments.

[0021] A fourth aspect of the present invention provides a computer program product having a computer program stored thereon, which is executed to implement the method for real-time semantic segmentation of three-dimensional point clouds with multi-perspective fusion as described in the above embodiments.

[0022] Therefore, the present invention receives the laser radar point cloud data, aligns and splices the original point cloud data, splices the point cloud in the local coordinate system into a complete 360-degree full-view angle point cloud data centered on the vehicle, captures the basic features of each point in the point cloud, and converts it into a series of feature vectors, further refines the point cloud layer by layer, captures local details and integrates global context information, introduces the point cloud attention mechanism, realizes the multi-scale fusion of features, and combines the confidence filtering mechanism to accurately output the semantic label prediction value of each point, thereby realizing accurate and efficient point cloud semantic segmentation. Therefore, the problem that the point cloud semantic segmentation method in the related technology is difficult to adapt to complex and changeable road scenes, the feature expression is incomplete, and there are limitations, such as quantization loss, semantic inconsistency, and low computational efficiency, is solved, and the semantic segmentation accuracy of point cloud data in the autonomous driving scenario is significantly improved, and the processing of local details and the integration of global context information in the point cloud semantic segmentation in the autonomous driving scenario are realized, which helps to realize a safer and more efficient autonomous driving system.

[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a method for real-time semantic segmentation of a three-dimensional point cloud with multi-view fusion provided according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a real-time LiDAR point cloud semantic segmentation method for autonomous driving perception according to a specific embodiment of the present invention;

[0027] Figure 3 A schematic block diagram of a device for real-time semantic segmentation of a three-dimensional point cloud with multi-view fusion according to an embodiment of the present invention;

[0028] Figure 4 The present invention is a block diagram of a vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0030] The following describes a method, device, and vehicle for real-time semantic segmentation of three-dimensional point clouds with multi-view fusion according to an embodiment of the present invention with reference to the accompanying drawings. In view of the fact that the point cloud semantic segmentation method mentioned in the above background technology is difficult to adapt to complex and changeable road scenes, the feature expression is incomplete, and there are limitations, such as quantization loss, semantic inconsistency, and low computational efficiency, the present invention provides a multi-perspective fusion three-dimensional point cloud real-time semantic segmentation method, in which a point cloud data set is obtained based on a preset point cloud semantic segmentation network, and a feature vector of each point cloud data is obtained according to the basic features of each point cloud data; a neighborhood point set and a feature matrix of the neighborhood point set of each point cloud data are determined, the feature maximum value of each feature matrix is ​​calculated, and the neighborhood feature vector of each point cloud data is obtained according to the feature maximum value of each feature matrix; the feature weight of each point cloud data is calculated based on the neighborhood feature vector, and the final feature vector of each point cloud data is obtained according to the feature weight and a preset multi-layer perceptron; a comprehensive confidence score of each point cloud data is calculated according to the prediction confidence, category confidence and final feature vector of each point cloud data at different semantic levels, and a semantic label prediction result of each point cloud data is obtained according to the comprehensive confidence score, and the semantic label prediction results are combined to obtain a semantic label data set of the point cloud data set. As a result, the problems of point cloud semantic segmentation methods being difficult to adapt to complex and changeable road scenes, incomplete feature expression, and limitations such as quantization loss, semantic inconsistency, and low computational efficiency are solved. The semantic segmentation accuracy of point cloud data in autonomous driving scenarios is significantly improved, and the processing of local details in point cloud semantic segmentation in autonomous driving scenarios and the integration of global contextual information are realized, which helps to achieve a safer and more efficient autonomous driving system.

[0031] Specifically, Figure 1 A schematic flow chart of a method for real-time semantic segmentation of three-dimensional point clouds with multi-view fusion provided in an embodiment of the present invention.

[0032] like Figure 1 As shown, the multi-view fusion 3D point cloud real-time semantic segmentation method includes the following steps:

[0033] In step S101, based on a preset point cloud semantic segmentation network, a point cloud data set is acquired, and a feature vector of each point cloud data is obtained according to the basic features of each point cloud data.

[0034] Among them, the point cloud semantic segmentation network preset in the embodiment of the present invention is used for the segmentation of point clouds on autonomous driving vehicles, and predicts the semantic labels of the full-view angle point clouds centered on the vehicle in real time. The point cloud semantic segmentation network performs range-point feature extraction, range scanning spatial aggregation, key feature enhancement, and multi-level feature selection on the full-view point cloud data, and accurately outputs the semantic label prediction value of each point cloud data, thereby achieving accurate and efficient point cloud semantic segmentation. The following description of the embodiment will specifically explain the principle of the point cloud semantic segmentation network.

[0035] The point cloud dataset of the embodiment of the present invention is a complete 360-degree surround view full-field angle point cloud data centered on the vehicle. Therefore, the point cloud dataset provided by the embodiment of the present invention can reflect a comprehensive view of the vehicle's surrounding environment. The point cloud data has spatial continuity and comprehensive feature expression, which helps to improve the accuracy of semantic segmentation of point cloud data.

[0036] Optionally, in some embodiments, before obtaining the point cloud data set, it also includes: obtaining multiple field of view angle point cloud data of the current vehicle; based on the vehicle coordinate system, aligning and splicing the multiple field of view angle point cloud data through preset displacement vectors and rotation matrices to obtain a point cloud data set.

[0037] Among them, multiple field of view angle point cloud data of the embodiment of the present invention are obtained through laser radar. Specifically, the embodiment of the present invention sets a lightweight laser scanning system on the current vehicle, and measures through a vehicle-mounted installation method. During the measurement, multiple lightweight radars are arranged, and the multiple radars collect point clouds with different field of view angles to ensure that the complete point cloud data of the full field of view angle of the vehicle, that is, the original point cloud data, is obtained.

[0038] Furthermore, after acquiring the original point cloud data, the embodiment of the present invention aligns and stitches the original point cloud data, and stitches the point cloud in the local coordinate system into a complete 360-degree surround full-field-of-view point cloud data centered on the vehicle, that is, the point cloud data set of the embodiment of the present invention.

[0039] It is understandable that in order to unify the point cloud data of multiple radars on the vehicle into the vehicle coordinate system and align the point cloud data, the embodiment of the present invention first defines the displacement vector and rotation matrix of each radar relative to the vehicle coordinate system. Then, the points in each radar point cloud are converted to the vehicle coordinate system through the rotation matrix and displacement vector.

[0040] Specifically, for radar A point , the conversion formula is:

[0041] ; (1)

[0042] in, is the point cloud data in the vehicle coordinate system, is the rotation matrix, For each point in the radar point cloud, is the displacement vector.

[0043] Therefore, the embodiment of the present invention can apply the transformation of formula (1) to obtain the converted point cloud set. Then, the two converted point cloud data sets are registered, and the optimal rotation matrix and displacement vector are obtained by minimizing the error of the corresponding points in the two point cloud data sets. Repeat this process, that is, align the registered point cloud data set with the point cloud data set of the next radar, until all point cloud data are registered to the self-vehicle coordinate system, and finally obtain a unified complete point cloud data of the full field of view of the self-vehicle.

[0044] It should be noted that the embodiments of the present invention can be based on an optimization algorithm, such as an iterative nearest algorithm, to minimize the error between corresponding points in two point cloud data sets. This process will find an optimal rotation matrix and displacement vector, that is, the preset displacement vector and rotation matrix of the embodiment of the present invention, so that the two point cloud sets are aligned in the vehicle coordinate system.

[0045] Furthermore, the basic features of each point cloud data in the embodiment of the present invention include position coordinates, which are usually expressed in three-dimensional coordinates, and also include information such as confidence and timestamp. Furthermore, the embodiment of the present invention obtains a feature vector of each point cloud data based on the basic features of each captured point cloud data.

[0046] Optionally, in some embodiments, a feature vector of each point cloud data is obtained based on the basic features of each point cloud data, including: converting the basic features of each point cloud data into 2D range features, and reconverting the 2D range features of each point cloud data into 3D point features to obtain the feature vector of each point cloud data.

[0047] It is understandable that the point cloud semantic segmentation network of the embodiment of the present invention can perform range-point feature extraction operations on full-view point cloud data to achieve feature extraction and 2D projection of point cloud data. This process includes two key steps: Point-to-Range (projection of 3D points to 2D pixels, referred to as P2R) and Range-to-Point (reverse mapping of 2D range pixels to 3D points, referred to as R2P). P2R and R2P together form an efficient downsampling pipeline, which maintains efficient downsampling capabilities while expanding the receptive field during the downsampling process based on the range view, and fully utilizes the optimization capabilities of CNN (Convolutional Neural Network‌, ‌‌convolutional neural network) on GPU (Graphics Processing Unit‌, ‌‌graphics processor).

[0048] Specifically, the Point-to-Range operation aims to convert 3D point features into 2D range features. 3D points Pixels projected into a 2D range view, where For the The x-axis coordinate of the 3D point, is the corresponding y-axis coordinate, is the corresponding z-axis coordinate to obtain the corresponding 2D coordinate ,in is the u-axis coordinate in the corresponding range view, is the corresponding v-axis coordinate. Then, the maximum pooling is used to transform the features of the points Aggregation generates corresponding 2D range features The process can be described by equation (2):

[0049] ; (2)

[0050] in, is a 2D range feature, is a 3D point feature, h and w correspond to the x-axis and y-axis coordinates in the range view, k is the number of neighboring points, and c is the number of feature channels.

[0051] Furthermore, in order to realize the conversion of the range view, the embodiment of the present invention converts the 3D point from the 3D Cartesian coordinate system Convert to spherical coordinates The conversion formula is shown in formula (3):

[0052] ; (3)

[0053] in, and They represent distance, zenith distance and azimuth respectively.

[0054] Then, by discretizing and Generate a predefined width and height The discretization process is shown in formula (4):

[0055] ; (4)

[0056] in, They respectively represent the field of view angle above the horizontal plane and the reciprocal of the overall field of view angle of the laser radar.

[0057] It should be noted that the operation of Range-to-Point is opposite to that of Point-to-Range. To avoid redundancy, we will not elaborate on it here. The features are transferred from 2D range pixels to 3D points, and the mapping relationship formed in P2R is constructed.

[0058] Therefore, each 3D point in the embodiment of the present invention will have a feature vector associated with it, and this feature vector is extracted and mapped back from the 2D range view, that is, the embodiment of the present invention obtains a series of feature vectors associated with each point in the 3D point cloud.

[0059] Based on the above embodiment, after step S101, the embodiment of the present invention captures key information of point cloud data through range-point feature extraction, which helps to process complex scenes and is crucial for subsequent analysis and processing.

[0060] In step S102, the neighborhood point set and the feature matrix of each point cloud data are determined, the feature maximum value of each feature matrix is ​​calculated, and the neighborhood feature vector of each point cloud data is obtained according to the feature maximum value of each feature matrix.

[0061] The neighborhood point set of each point cloud data is obtained based on the three-dimensional space structure Range-Scan Space (3D space representation, RSS for short) of an embodiment of the present invention, and the RSS optimizes the local feature update and neighbor index of the point cloud data.

[0062] It can be understood that the embodiment of the present invention performs a range scan spatial aggregation operation on the full-view point cloud data through a point cloud semantic segmentation network, that is, further refines the point cloud data layer by layer, and utilizes the 3D spatial representation method of the embodiment of the present invention, namely RSS, which not only captures local details, but also integrates global context information, while effectively saving computing resources.

[0063] Optionally, in some embodiments, determining a neighborhood point set and a characteristic matrix of the neighborhood point set for each point cloud data includes: judging whether the first point cloud data and the second point cloud data are adjacent in the point space, and whether the first point cloud data and the second point cloud data are adjacent in the range space; if the first point cloud data and the second point cloud data are adjacent in the point space, and the first point cloud data and the second point cloud data are adjacent in the range space, then judging that the first point cloud data and the second point cloud data belong to the same unit space; determining the point cloud data that belongs to the same unit space as each point cloud data, and obtaining the neighborhood point set of each point cloud data based on the point cloud data of the same unit space.

[0064] The first point cloud data and the second point cloud data in the embodiment of the present invention are arbitrary points in the point cloud data.

[0065] First, the embodiment of the present invention defines an adjacency relationship. Specifically, in point space (PS), if two points are determined and are adjacent, then they are adjacent in point space, expressed as If two points and are adjacent in the range space (RS), and then use To represent. Furthermore, the embodiment of the present invention constructs a Range-Scan Space, that is, in this space, if two points are adjacent in both the point space and the range space, then the two points belong to the same RSS unit space. In other words, the RSS of the embodiment of the present invention is used to represent a set of points that have an adjacent relationship in both the point space and the range space.

[0066] Therefore, the embodiment of the present invention defines the point cloud data belonging to the same unit space as each point cloud data based on the above-mentioned index definition, thereby obtaining a neighborhood point set for each point cloud data.

[0067] Furthermore, for a set of points belonging to the same RSS unit space , the embodiment of the present invention constructs a feature matrix , that is, the feature matrix of the neighborhood point set of the embodiment of the present invention, and aggregates the neighborhood features by calculating the maximum value of each feature dimension to obtain the RSS neighborhood feature vector y.

[0068] Therefore, after step S102, the present invention considers the neighborhood point set of each point, can capture local geometric features, understand the local structure of the point cloud data, calculate the feature matrix of the neighborhood point set and extract the maximum eigenvalue, which is helpful to distinguish different point cloud data, and convert the feature matrix of the neighborhood point set into a feature vector, which can reduce the dimension of the data, make subsequent processing more efficient, and improve the accuracy of point cloud data classification and recognition.

[0069] In step S103, the feature weight of each point cloud data is calculated based on the neighborhood feature vector, and the final feature vector of each point cloud data is obtained according to the feature weight and a preset multi-layer perceptron.

[0070] It can be understood that the point cloud semantic segmentation network of the embodiment of the present invention performs key feature enhancement operations on the full-view point cloud data. The embodiment of the present invention introduces a point cloud attention mechanism, which not only enhances the recognition of key features, but also cascades the feature information of the encoding stage through jump connections, thereby realizing multi-scale fusion of features.

[0071] Optionally, in some embodiments, the feature weight of each point cloud data is calculated based on the neighborhood feature vector, including: mapping the 2D range features and 3D point features of each point cloud data to a corresponding neighborhood point set, and indexing and aggregating the neighboring points of each point cloud data based on the corresponding neighborhood point set; and calculating the feature weight of each point cloud data through a preset fully connected layer according to the features of the neighboring points.

[0072] Specifically, the embodiment of the present invention maps 2D range features and 3D point features to Range-Scan Space to uniformly process these two features, which is convenient for subsequent feature aggregation and enhancement. Furthermore, the features of neighboring points are quickly indexed and aggregated to form a richer feature representation. The embodiment of the present invention calculates feature weights for each 3D point through a fully connected layer, and applies these weights to generate the final feature vector through the following equations (5) and (6):

[0073] ; (5)

[0074] ; (6)

[0075] in, is the sigmoid activation function, which is used to introduce nonlinearity, and and is the weight parameter of the Multilayer Perceptron (MLP), is the point feature of the Lth layer, is the point feature of the L+1th layer, It is the L-th level feature of RSS.

[0076] Thus, after step S103, the present invention calculates feature weights through a fully connected layer, which can highlight important features and suppress unimportant features, thereby improving the expressiveness of features, and using the sigmoid activation function to introduce nonlinearity, so that the model can capture more complex feature relationships, improving the expressiveness and generalization performance of the model. Ultimately, the final feature vector generated provides a compact and information-rich representation for each 3D point, which is helpful for subsequent classification, detection, and segmentation tasks.

[0077] In step S104, a comprehensive confidence score for each point cloud data is calculated based on the prediction confidence, category confidence and final feature vector of each point cloud data at different semantic levels, and a semantic label prediction result for each point cloud data is obtained based on the comprehensive confidence score. The semantic label prediction results are combined to obtain a semantic label dataset of the point cloud dataset.

[0078] It can be understood that the point cloud semantic segmentation network of the embodiment of the present invention performs multi-level feature selection operations on the full-view point cloud data. In this step, the embodiment of the present invention subdivides the point cloud into sub-clouds with multiple semantic levels, and adaptively fuses features at different levels. Combined with the confidence filtering mechanism, it accurately outputs the semantic label prediction value of each point, thereby achieving accurate and efficient point cloud semantic segmentation.

[0079] First, the embodiment of the present invention predicts features at different semantic levels and the corresponding confidence score and category confidence scores Input a multi-level feature selection head, where i is the feature level, and then, for each 3D point , calculate the comprehensive confidence score, the specific calculation method is shown in formula (7):

[0080] ; (7)

[0081] in, is the prediction confidence of the current point for category i, is the prediction confidence of the current point, and the calculated Append to list.

[0082] Furthermore, in some embodiments, the final prediction result of each point cloud data is obtained according to the comprehensive confidence score, including: comparing the comprehensive confidence scores of different semantic levels, and taking the maximum value of the comprehensive confidence scores as the final prediction result.

[0083] Specifically, the embodiment of the present invention compares the comprehensive confidence scores of different levels and selects the level with the highest confidence. The corresponding features are used as the final prediction results. The selection formula is shown in formula (8):

[0084] ; (8)

[0085] Final prediction results By the corresponding level characteristics Decide.

[0086] Finally, the embodiment of the present invention converts the final prediction result of each point into The points are combined to form an overall point cloud classification result, and the classification result is output in the form of semantic labels, i.e., the semantic label data set of an embodiment of the present invention, thereby achieving accurate and efficient semantic segmentation of the point cloud.

[0087] Thus, after step S104, the comprehensive confidence score of the present invention takes into account the prediction confidence at multiple semantic levels, greatly improving the accuracy of the semantic label prediction of each point cloud data. By combining the confidence at different levels, the influence of the prediction error at a single level can be reduced, thereby enhancing the robustness of the point cloud semantic segmentation network to noise and outliers. In addition, the point cloud semantic segmentation network of the present invention realizes automated semantic label prediction, improving the efficiency and automation level of data processing.

[0088] Optionally, in some embodiments, the real-time semantic segmentation method of three-dimensional point cloud with multi-perspective fusion also includes: counting the frequency of different categories of point cloud data in each range area, and determining the pseudo-label of each range area according to the frequency; optimizing the preset point cloud semantic segmentation network according to the pseudo-label and the preset overall loss function.

[0089] It can be understood that in order to achieve more accurate real-time semantic segmentation of three-dimensional point clouds, the embodiment of the present invention trains the point cloud semantic segmentation network, and uses the trained point cloud semantic segmentation network of the present invention for autonomous driving vehicle-mounted point cloud segmentation, and predicts the semantic labels of the full-field-of-view angular point cloud centered on the vehicle in real time.

[0090] In the related art, the final semantic score is supervised by a point-level loss function. The present invention uses an additional auxiliary head and a range-level loss function to optimize the range feature. Specifically, the embodiment of the present invention needs to create a pseudo-range label. Since a range area usually contains multiple points that may belong to different categories, in order to solve this problem, the embodiment of the present invention draws on the sparse voxel technology and adopts a statistical method to generate a pseudo-label for each range by counting the frequency of each category in the range area, and assigns the category with the highest frequency as a pseudo-label, and then uses these pseudo-labels to supervise the range-level features. The overall loss function is expressed as shown in formula (9):

[0091] ; (9)

[0092] in, and denote point-level and range loss functions respectively, Adjusts the weight of the range loss.

[0093] It should be noted that the point-level loss The cross-pick loss is used for calculation. Meanwhile, the range loss It is a combination of cross-picking loss, Lovasz-Softmax loss and boundary loss.

[0094] Therefore, the embodiment of the present invention can deploy the trained point cloud semantic segmentation network on the autonomous driving vehicle, obtain the point cloud data scanned by the on-board laser radar in real time, use the point cloud semantic segmentation network to perform semantic segmentation on the point cloud data, predict the semantic label of each point, and form a real-time point cloud semantic segmentation result, thereby solving the real-time problem of neighborhood feature aggregation and the difficulty of multi-semantic level joint prediction of the cone-type laser radar point cloud semantic segmentation method in autonomous driving perception, improving the autonomous driving vehicle's ability to understand the surrounding environment and enhancing driving safety.

[0095] In order to enable those skilled in the art to further understand the method for real-time semantic segmentation of three-dimensional point clouds with multi-view fusion according to an embodiment of the present invention, the following examples are listed to schematically illustrate the implementation steps of the method.

[0096] Specifically, Figure 2 FIG. 1 is a schematic diagram of a real-time LiDAR point cloud semantic segmentation method for autonomous driving perception according to a specific embodiment of the present invention. Figure 2 As shown, the multi-view fusion 3D point cloud real-time semantic segmentation method includes the following steps:

[0097] Step S1, using laser radar to acquire multi-point, multi-view point cloud data.

[0098] Step S2, registering and stitching the original point cloud data, stitching the point cloud in the local coordinate system into a complete 360-degree surround full-field-of-view point cloud data centered on the vehicle.

[0099] Step S3: In the encoding layer, a range-point feature extraction module is constructed to capture the basic features of each point in the point cloud and convert them into a series of feature vectors; a range scan spatial aggregation module is constructed to further refine the point cloud layer by layer, using a novel 3D spatial representation (RSS) to not only capture local details but also integrate global context information, while effectively saving computing resources;

[0100] In the decoding layer, the point cloud attention mechanism is introduced, which not only enhances the recognition of key features, but also realizes multi-scale fusion of features by cascading with the feature information of the encoding stage through jump connections; a multi-level feature selection mechanism module is constructed to subdivide the point cloud into sub-clouds with multiple semantic levels, and adaptively fuse features at different levels. Combined with the confidence filtering mechanism, the semantic label prediction value of each point is accurately output, thereby realizing accurate and efficient point cloud semantic segmentation.

[0101] Step S4, training and reasoning the autonomous driving perception point cloud semantic segmentation algorithm (point cloud semantic segmentation network), using the trained point cloud semantic segmentation network of the present invention for autonomous driving vehicle-mounted point cloud segmentation, and real-time prediction of the full-field angle point cloud semantic label centered on the vehicle.

[0102] Therefore, based on the point cloud semantic segmentation network of the embodiment of the present invention, by introducing a multi-level feature selection mechanism, combined with the encoding-decoding architecture and the point cloud attention mechanism, the semantic segmentation accuracy of point cloud data in the autonomous driving scenario is significantly improved. The method based on feature extraction and multi-scale fusion effectively solves the problem of integrating local details with global context information in the semantic segmentation of point clouds in autonomous driving scenarios. The real-time semantic segmentation method of three-dimensional point clouds with multi-perspective fusion proposed in the embodiment of the present invention can be applied to autonomous driving perception tasks, which helps to achieve a safer and more efficient autonomous driving system.

[0103] According to the multi-view fused three-dimensional point cloud real-time semantic segmentation method proposed in an embodiment of the present invention, a point cloud data set is obtained based on a preset point cloud semantic segmentation network, and a feature vector of each point cloud data is obtained according to the basic features of each point cloud data, a neighborhood point set of each point cloud data and a feature matrix of the neighborhood point set are determined, the feature maximum value of each feature matrix is ​​calculated, and the neighborhood feature vector of each point cloud data is obtained according to the feature maximum value of each feature matrix, the feature weight of each point cloud data is calculated based on the neighborhood feature vector, and the final feature vector of each point cloud data is obtained according to the feature weight and a preset multi-layer perceptron, a comprehensive confidence score of each point cloud data is calculated according to the prediction confidence, category confidence and final feature vector of each point cloud data at different semantic levels, and a semantic label prediction result of each point cloud data is obtained according to the comprehensive confidence score, and the semantic label prediction results are combined to obtain a semantic label data set of the point cloud data set. As a result, the problems in related technologies such as the difficulty of point cloud semantic segmentation methods in adapting to complex and changeable road scenes, incomplete feature expression, and limitations such as quantization loss, semantic inconsistency, and low computational efficiency are solved. The semantic segmentation accuracy of point cloud data in autonomous driving scenarios is significantly improved, and the processing of local details and the integration of global contextual information in point cloud semantic segmentation in autonomous driving scenarios are realized, which helps to achieve a safer and more efficient autonomous driving system.

[0104] Next, a device for real-time semantic segmentation of three-dimensional point clouds using multi-view fusion proposed in an embodiment of the present invention will be described with reference to the accompanying drawings.

[0105] Figure 3 It is a block diagram of a device for real-time semantic segmentation of a three-dimensional point cloud with multi-view fusion according to an embodiment of the present invention.

[0106] like Figure 3 As shown, the multi-view fused three-dimensional point cloud real-time semantic segmentation device 10 includes: an acquisition module 100, a feature aggregation module 200, a feature enhancement module 300 and a feature selection module 400.

[0107] Specifically, the acquisition module 100 is used to acquire a point cloud data set based on a preset point cloud semantic segmentation network, and obtain a feature vector for each point cloud data according to the basic features of each point cloud data; the feature aggregation module 200 is used to determine the neighborhood point set and the feature matrix of the neighborhood point set for each point cloud data, calculate the feature maximum value of each feature matrix, and obtain the neighborhood feature vector of each point cloud data according to the feature maximum value of each feature matrix; the feature enhancement module 300 is used to calculate the feature weight of each point cloud data based on the neighborhood feature vector, and obtain the final feature vector of each point cloud data according to the feature weight and a preset multi-layer perceptron; the feature selection module 400 is used to calculate the comprehensive confidence score of each point cloud data according to the prediction confidence, category confidence and final feature vector of each point cloud data at different semantic levels, and obtain the semantic label prediction result of each point cloud data according to the comprehensive confidence score, and combine the semantic label prediction results to obtain the semantic label data set of the point cloud data set.

[0108] Optionally, in some embodiments, the multi-perspective fused three-dimensional point cloud real-time semantic segmentation device 10 also includes: a statistical module, used to count the frequency of different categories of point cloud data in each range area, and determine the pseudo-label of each range area according to the frequency; an optimization module, used to optimize the preset point cloud semantic segmentation network according to the pseudo-label and the preset overall loss function.

[0109] Optionally, in some embodiments, the acquisition module 100 is specifically used to: convert the basic features of each point cloud data into two-dimensional 2D range features, and reconvert the 2D range features of each point cloud data into three-dimensional 3D point features to obtain a feature vector of each point cloud data.

[0110] Optionally, in some embodiments, the feature enhancement module 300 is specifically used to: map the 2D range features and 3D point features of each point cloud data to a corresponding neighborhood point set, and based on the corresponding neighborhood point set, index and aggregate the neighboring points of each point cloud data; and calculate the feature weight of each point cloud data through a preset fully connected layer according to the features of the neighboring points.

[0111] Optionally, in some embodiments, the feature selection module 400 is specifically used to compare the comprehensive confidence scores of different semantic levels and take the maximum value of the comprehensive confidence scores as the final prediction result.

[0112] Optionally, in some embodiments, the feature aggregation module 200 is specifically used to: determine whether the first point cloud data and the second point cloud data are adjacent in the point space, and whether the first point cloud data and the second point cloud data are adjacent in the range space; when the first point cloud data and the second point cloud data are adjacent in the point space, determine that the first point cloud data and the second point cloud data belong to the same unit space; determine the point cloud data that belongs to the same unit space as each point cloud data, and obtain a neighborhood point set for each point cloud data based on the point cloud data in the same unit space.

[0113] Optionally, in some embodiments, before acquiring the point cloud data set, the acquisition module 100 is also used to: acquire multiple field of view angle point cloud data of the current vehicle; based on the vehicle coordinate system, align and splice the multiple field of view angle point cloud data through preset displacement vectors and rotation matrices to obtain a point cloud data set.

[0114] It should be noted that the above explanation of the embodiment of the method for real-time semantic segmentation of three-dimensional point clouds with multi-perspective fusion is also applicable to the device for real-time semantic segmentation of three-dimensional point clouds with multi-perspective fusion of this embodiment, and will not be repeated here.

[0115] According to the multi-perspective fused three-dimensional point cloud real-time semantic segmentation device proposed in the embodiment of the present invention, a point cloud data set is acquired based on a preset point cloud semantic segmentation network, and a feature vector of each point cloud data is obtained according to the basic features of each point cloud data, a neighborhood point set of each point cloud data and a feature matrix of the neighborhood point set are determined, the feature maximum value of each feature matrix is ​​calculated, and the neighborhood feature vector of each point cloud data is obtained according to the feature maximum value of each feature matrix, the feature weight of each point cloud data is calculated based on the neighborhood feature vector, and the final feature vector of each point cloud data is obtained according to the feature weight and a preset multi-layer perceptron, a comprehensive confidence score of each point cloud data is calculated according to the prediction confidence, category confidence and final feature vector of each point cloud data at different semantic levels, and a semantic label prediction result of each point cloud data is obtained according to the comprehensive confidence score, and the semantic label prediction results are combined to obtain a semantic label data set of the point cloud data set. As a result, the problems in related technologies such as the difficulty of point cloud semantic segmentation methods in adapting to complex and changeable road scenes, incomplete feature expression, and limitations such as quantization loss, semantic inconsistency, and low computational efficiency are solved. The semantic segmentation accuracy of point cloud data in autonomous driving scenarios is significantly improved, and the processing of local details and the integration of global contextual information in point cloud semantic segmentation in autonomous driving scenarios are realized, which helps to achieve a safer and more efficient autonomous driving system.

[0116] Figure 4 A schematic diagram of the structure of a vehicle provided in an embodiment of the present invention. The vehicle may include:

[0117] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0118] When the processor 402 executes the program, the multi-view fusion three-dimensional point cloud real-time semantic segmentation method provided in the above embodiment is implemented.

[0119] Furthermore, the vehicle also includes:

[0120] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0121] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0122] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0123] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0124] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0125] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0126] An embodiment of the present invention further provides a computer program product having a computer program stored thereon, which, when executed, implements the above-mentioned multi-view fusion three-dimensional point cloud real-time semantic segmentation method.

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0128] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0129] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0130] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0131] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0132] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A multi-view fusion 3D point cloud real-time semantic segmentation method, characterized in that: The following steps are involved: Based on a preset point cloud semantic segmentation network, a point cloud data set is obtained, and a feature vector of each point cloud data is obtained according to the basic features of each point cloud data; Determine a neighborhood point set of each point cloud data and a feature matrix of the neighborhood point set, calculate a feature maximum value of each feature matrix, and obtain a neighborhood feature vector of each point cloud data according to the feature maximum value of each feature matrix; Calculating a feature weight of each point cloud data based on the neighborhood feature vector, and obtaining a final feature vector of each point cloud data according to the feature weight and a preset multi-layer perceptron; A comprehensive confidence score for each point cloud data is calculated based on the prediction confidence, category confidence and the final feature vector of each point cloud data at different semantic levels, and a semantic label prediction result for each point cloud data is obtained based on the comprehensive confidence score. The semantic label prediction results are combined to obtain a semantic label dataset of the point cloud dataset.

2. The method for real-time semantic segmentation of 3D point cloud with multi-view fusion according to claim 1, characterized in that: Also includes: Counting the frequencies of different categories of point cloud data in each range area, and determining a pseudo label for each range area according to the frequencies; The preset point cloud semantic segmentation network is optimized according to the pseudo-label and a preset overall loss function.

3. The multi-view fusion 3D point cloud real-time semantic segmentation method according to claim 1, characterized in that: The step of obtaining the feature vector of each point cloud data according to the basic features of each point cloud data comprises: The basic features of each point cloud data are converted into two-dimensional 2D range features, and the 2D range features of each point cloud data are reconverted into three-dimensional 3D point features to obtain a feature vector of each point cloud data.

4. The method for real-time semantic segmentation of 3D point cloud with multi-view fusion according to claim 3, characterized in that: The calculating the feature weight of each point cloud data based on the neighborhood feature vector comprises: Mapping the 2D range features and the 3D point features of each point cloud data to a corresponding neighborhood point set, and indexing and aggregating neighboring points of each point cloud data based on the corresponding neighborhood point set; According to the features of the neighboring points, the feature weight of each point cloud data is calculated through a preset fully connected layer.

5. The multi-view fusion 3D point cloud real-time semantic segmentation method according to claim 1, characterized in that: The final prediction result of each point cloud data is obtained according to the comprehensive confidence score, including: The comprehensive confidence scores of the different semantic levels are compared, and the maximum value of the comprehensive confidence scores is taken as the final prediction result.

6. The multi-view fusion 3D point cloud real-time semantic segmentation method according to claim 1, characterized in that: The determining of the neighborhood point set of each point cloud data and the feature matrix of the neighborhood point set includes: Determining whether the first point cloud data and the second point cloud data are adjacent in the point space, and whether the first point cloud data and the second point cloud data are adjacent in the range space; If the first point cloud data and the second point cloud data are adjacent in the point space, and the first point cloud data and the second point cloud data are adjacent in the range space, then determining that the first point cloud data and the second point cloud data belong to the same unit space; Point cloud data belonging to the same unit space as each point cloud data is determined, and a neighborhood point set of each point cloud data is obtained according to the point cloud data in the same unit space.

7. The multi-view fusion 3D point cloud real-time semantic segmentation method according to claim 1, characterized in that: Before obtaining the point cloud dataset, it also includes: Obtain multiple field of view point cloud data of the current vehicle; Based on the vehicle coordinate system, the plurality of field of view angle point cloud data are aligned and spliced ​​through a preset displacement vector and rotation matrix to obtain the point cloud data set.

8. A multi-view fusion 3D point cloud real-time semantic segmentation device, characterized in that: include: An acquisition module is used to acquire a point cloud data set based on a preset point cloud semantic segmentation network, and obtain a feature vector of each point cloud data according to the basic features of each point cloud data; A feature aggregation module, used to determine a neighborhood point set of each point cloud data and a feature matrix of the neighborhood point set, calculate a feature maximum value of each feature matrix, and obtain a neighborhood feature vector of each point cloud data according to the feature maximum value of each feature matrix; A feature enhancement module, used to calculate the feature weight of each point cloud data based on the neighborhood feature vector, and obtain the final feature vector of each point cloud data according to the feature weight and a preset multi-layer perceptron; A feature selection module is used to calculate the comprehensive confidence score of each point cloud data according to the prediction confidence, category confidence and the final feature vector of each point cloud data at different semantic levels, and obtain the semantic label prediction result of each point cloud data according to the comprehensive confidence score, and combine the semantic label prediction results to obtain the semantic label data set of the point cloud data set.

9. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-view fused three-dimensional point cloud real-time semantic segmentation method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the multi-view fusion three-dimensional point cloud real-time semantic segmentation method as described in any one of claims 1 to 7.

Images