Semantic segmentation method and system based on 3D point cloud

Abstract

The application discloses a semantic segmentation method and system based on 3D point cloud, and relates to the field of point cloud semantic segmentation, wherein the method comprises the following steps: a first point cloud semantic segmentation model is obtained by training a first point cloud dataset; when performing semantic segmentation on to-be-segmented point cloud data, it is judged whether a feature adaptive gain control module is activated; if yes, the to-be-segmented point cloud is input into the feature adaptive gain control module, a feature gain coefficient is generated based on feature intensity information, the original feature is modulated, and an adapted point cloud feature is obtained; and the adapted point cloud feature is input into the first point cloud semantic segmentation model, and a semantic segmentation result is output. The application solves the technical problems that the existing semantic segmentation based on 3D point cloud is sensitive to the feature distribution difference of point clouds of heterogeneous sensors, and has weak cross-domain adaptability, and achieves the technical effects of improving the feature adaptation capability of the model for point clouds of heterogeneous sensors, and improving the accuracy and system reliability of environment perception in a cross-sensor scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of point cloud semantic segmentation, in particular to a semantic segmentation method and system based on 3D point cloud. BACKGROUND

[0002] As a core technology in the field of autonomous driving, intelligent robots, etc., the adaptation ability of 3D point cloud semantic segmentation in cross-sensor scenarios directly determines the reliability of system environment perception. The main method to solve this problem at present is to train a semantic segmentation model for a specific sensor data set, and then directly apply the trained model to the target scene. This method leads to a significant decline in model performance when applied to heterogeneous sensor data, as it does not take into account the huge differences in local geometric feature contrast of different sensor point clouds. Re-training the model faces the dilemma of high cost and long cycle.

[0003] At present, in the related art, the semantic segmentation based on 3D point cloud has the technical problems of model sensitivity to heterogeneous sensor point cloud feature distribution differences and weak cross-domain adaptability. SUMMARY

[0004] The present application provides a semantic segmentation method and system based on 3D point cloud. By training a first point cloud semantic segmentation model using a first point cloud data set collected by a first point cloud sensor, when the model is used for semantic segmentation of point cloud data collected by a second point cloud sensor, it is first determined whether a feature adaptive gain control module needs to be activated. If it needs to be activated, the point cloud to be segmented is input into the module. The module generates a feature gain coefficient based on the feature intensity information of the point cloud to be segmented, and uses it to modulate the original features of the point cloud to be segmented to obtain adapted point cloud features. Finally, the adapted point cloud features are input into the first point cloud semantic segmentation model, and the semantic segmentation result of the point cloud to be segmented is output. Technical means such as this solve the technical problems of existing semantic segmentation based on 3D point cloud, such as model sensitivity to heterogeneous sensor point cloud feature distribution differences and weak cross-domain adaptability, and achieve the technical effects of improving the feature adaptation ability of the model to heterogeneous sensor point clouds, and improving the accuracy and system reliability of environment perception in cross-sensor scenarios.

[0005] The application provides a semantic segmentation method based on a 3D point cloud, comprising: training a first point cloud semantic segmentation model by using a first point cloud data set, wherein the first point cloud data set is collected by a first point cloud sensor; when performing semantic segmentation on to-be-segmented point cloud data collected by a second point cloud sensor by using the first point cloud semantic segmentation model, determining whether to activate a feature adaptive gain control module, if not, directly inputting the to-be-segmented point cloud data into the first point cloud semantic segmentation model, and outputting a semantic segmentation result of the to-be-segmented point cloud data; if yes, inputting the to-be-segmented point cloud into the feature adaptive gain control module, wherein the feature adaptive gain control module generates a corresponding feature gain coefficient based on feature intensity information of the to-be-segmented point cloud data, modulates original features of the to-be-segmented point cloud data by using the feature gain coefficient, and obtains adapted point cloud features; and inputting the adapted point cloud features into the first point cloud semantic segmentation model, and outputting a semantic segmentation result of the to-be-segmented point cloud data.

[0006] In a possible implementation, when performing semantic segmentation on to-be-segmented point cloud data collected by a second point cloud sensor by using the first point cloud semantic segmentation model, it is determined whether to activate a feature adaptive gain control module, and the following processing is performed: testing the first point cloud semantic segmentation model with different point cloud feature intensities to generate a plurality of model test information corresponding to a plurality of point cloud feature intensity distributions; extracting a point cloud feature intensity distribution space that meets a preset segmentation accuracy based on the plurality of model test information; collecting real-time point cloud feature intensity distribution of the to-be-segmented point cloud data, and determining whether to meet the point cloud feature intensity distribution space, if not, activating the feature adaptive gain control module.

[0007] In a possible implementation, the following processing is performed: the quantitative indicators of the point cloud feature intensity include local point density, curvature, and channel statistics of an early feature map obtained after shallow feature extraction on the point cloud, and the channel statistics include mean or variance; and the shallow features are calculated by inputting the point cloud data into an input layer of the first point cloud semantic segmentation model.

[0008] In a possible implementation, the following processing is performed: the feature adaptive gain control module comprises a point cloud intensity detector, a reference comparator, and a gain smoothing unit, and a preset point cloud feature intensity distribution space is stored in the reference comparator.

[0009] In a possible implementation, the feature adaptive gain control module generates a corresponding feature gain coefficient based on feature intensity information of the to-be-segmented point cloud data, and performs the following processing: the point cloud feature intensity of each point in the to-be-segmented point cloud data is calculated by the point cloud intensity detector to generate the feature intensity information; the feature intensity information is compared with the point cloud feature intensity distribution space stored in the reference comparator to generate the feature gain coefficient, and the gain smoothing unit identifies feature activation noise based on the feature gain coefficient and matches a gain smoothing filtering parameter; the original feature of the to-be-segmented point cloud data is modulated according to the feature gain coefficient, and the modulated point cloud is filtered with the gain smoothing filtering parameter to generate the adapted point cloud feature.

[0010] In a possible implementation, when the first point cloud semantic segmentation model is used to perform semantic segmentation on to-be-segmented point cloud data collected by a second point cloud sensor, the following processing is further performed: a plurality of point cloud sensors are preset based on historical point cloud segmentation requirements; point cloud difference analysis is performed on the plurality of point cloud sensors, a fixed gain coefficient is configured, and a sensor-fixed gain mapping is established; it is judged whether the sensor-fixed gain mapping contains a fixed gain coefficient corresponding to the second point cloud sensor, and if so, the original feature of the to-be-segmented point cloud data is directly modulated with the corresponding fixed gain coefficient.

[0011] In a possible implementation, the point cloud difference analysis is performed on the plurality of point cloud sensors, the fixed gain coefficient is configured, and the sensor-fixed gain mapping is established, and the following processing is performed: point cloud segmentation data is collected for the plurality of point cloud sensors, and difference comparison is performed with the first point cloud data set; gain modulation test is performed by calling the first point cloud semantic segmentation model according to the comparison result, to generate a plurality of fixed gain coefficients of the feature adaptive gain control module for the plurality of point cloud sensors; the plurality of point cloud sensors are identified and associated with the plurality of fixed gain coefficients, and the sensor-fixed gain mapping is established.

[0012] In a possible implementation, the following processing is performed: the feature adaptive gain control module is located at the front end of the first point cloud semantic segmentation model to perform modulation once to obtain an adapted point cloud feature.

[0013] In a possible implementation, the following processing is performed: according to the number of network layers of the first point cloud semantic segmentation model, one feature adaptive gain control module is configured at each network layer, and feature extraction of the network layer is performed at each network layer to perform point cloud data modulation once.

[0014] The application also provides a 3D point cloud-based semantic segmentation system, comprising: a first point cloud semantic segmentation model training module, configured to train a first point cloud semantic segmentation model by using a first point cloud dataset, wherein the first point cloud dataset is collected by a first point cloud sensor; a judgment module, configured to judge whether to activate a feature adaptive gain control module when performing semantic segmentation on to-be-segmented point cloud data collected by a second point cloud sensor by using the first point cloud semantic segmentation model, and if not, directly input the to-be-segmented point cloud data into the first point cloud semantic segmentation model and output a semantic segmentation result of the to-be-segmented point cloud data; a feature adaptive gain module, configured to input the to-be-segmented point cloud into the feature adaptive gain control module if the judgment result is yes, wherein the feature adaptive gain control module generates a corresponding feature gain coefficient based on feature intensity information of the to-be-segmented point cloud data, modulates original features of the to-be-segmented point cloud data by using the feature gain coefficient, and obtains adapted point cloud features; and a semantic segmentation module, configured to input the adapted point cloud features into the first point cloud semantic segmentation model and output a semantic segmentation result of the to-be-segmented point cloud data.

[0015] The 3D point cloud-based semantic segmentation method and system provided in the application first train a first point cloud semantic segmentation model by using a first point cloud dataset, wherein the first point cloud dataset is collected by a first point cloud sensor, then judge whether to activate a feature adaptive gain control module when performing semantic segmentation on to-be-segmented point cloud data collected by a second point cloud sensor by using the first point cloud semantic segmentation model, if not, directly input the to-be-segmented point cloud data into the first point cloud semantic segmentation model and output a semantic segmentation result of the to-be-segmented point cloud data, if yes, input the to-be-segmented point cloud into the feature adaptive gain control module, wherein the feature adaptive gain control module generates a corresponding feature gain coefficient based on feature intensity information of the to-be-segmented point cloud data, modulates original features of the to-be-segmented point cloud data by using the feature gain coefficient, and obtains adapted point cloud features, and finally input the adapted point cloud features into the first point cloud semantic segmentation model and output a semantic segmentation result of the to-be-segmented point cloud data. Through the above process, the method and system provided in the application achieve the technical effects of improving the feature adaptation capability of the model to heterogeneous sensor point clouds and improving the accuracy and system reliability of environment perception in a cross-sensor scenario. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application will be briefly introduced below. In the present application, flowcharts are used to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the foregoing or the following operations are not necessarily performed in sequence. On the contrary, various steps can be processed in reverse order or simultaneously according to needs. Meanwhile, other operations can be added to these processes, or one or more steps of operations can be removed from these processes.

[0017] Figure 1 A flowchart of a semantic segmentation method based on 3D point clouds provided by the embodiments of the present application.

[0018] Figure 2 A structural diagram of a semantic segmentation system based on 3D point clouds provided by the embodiments of the present application.

[0019] Legend: first point cloud semantic segmentation model training module 10, judgment module 20, feature adaptive gain module 30, semantic segmentation module 40. DETAILED DESCRIPTION

[0020] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the specific embodiments, structures, features and effects according to the present application will be described in detail below with reference to the drawings and preferred embodiments.

[0021] The embodiments of the present application provide a semantic segmentation method based on 3D point clouds, as shown in Figure 1 The method comprises the following steps:

[0022] In step S100, a first point cloud semantic segmentation model is trained by using a first point cloud data set, and the first point cloud data set is collected by a first point cloud sensor.

[0023] Specifically, the first point cloud data set refers to an original data set serving as a model training benchmark, which is collected by a point cloud sensor of a specific model in a fixed scene and contains a large amount of 3D point cloud data with semantic labels. The first point cloud semantic segmentation model is a network model constructed based on a deep learning framework, which learns the mapping relationship between point cloud features and semantic categories in the data set through iterative training on the first point cloud data set, and has the basic ability to perform semantic segmentation on the same point cloud data.

[0024] In step S200, when performing semantic segmentation on the to-be-segmented point cloud data collected by a second point cloud sensor by using the first point cloud semantic segmentation model, it is judged whether to activate a feature adaptive gain control module. If not, the to-be-segmented point cloud data is directly input into the first point cloud semantic segmentation model, and the semantic segmentation result of the to-be-segmented point cloud data is output.

[0025] In particular, the second point cloud sensor refers to a different device from the first point cloud sensor, such as a different linear beam lidar, a different brand millimeter wave radar, and the like. The acquired point cloud data to be segmented by the second point cloud sensor has differences in local geometric feature contrast with the first point cloud data set due to differences in sensor hardware parameters and acquisition scenes. The feature adaptive gain control module is a functional module for adjusting the point cloud feature intensity. The core function is to eliminate the feature differences of point clouds of different sensors, so that the point cloud features to be segmented adapt to the input requirements of the first point cloud semantic segmentation model. This step determines whether the feature adaptive gain control module needs to be started for feature modulation through a preset judgment logic, to avoid redundant calculation when modulation is not needed. If the judgment result is that the feature adaptive gain control module needs to be activated, the point cloud data to be segmented is first input into the module for feature intensity modulation, and then the adapted point cloud features are input into the first point cloud semantic segmentation model for semantic segmentation. If the judgment result is that the feature adaptive gain control module does not need to be activated, the point cloud data to be segmented is directly input into the first point cloud semantic segmentation model, and the semantic segmentation result of the point cloud data to be segmented is output.

[0026] In a possible implementation, when performing semantic segmentation on the to-be-segmented point cloud data collected by the second point cloud sensor by using the first point cloud semantic segmentation model, it is determined whether to activate the feature adaptive gain control module, and step S200 further includes step S210: performing a test on different point cloud feature intensities for the first point cloud semantic segmentation model to generate a plurality of model test information corresponding to a plurality of point cloud feature intensity, wherein the quantization indicators of the point cloud feature intensity include local point density, curvature, and channel statistics of an early feature map obtained after shallow feature extraction on the point cloud, and the channel statistics include mean or variance; the shallow feature is calculated by inputting the point cloud data into the input layer of the first point cloud semantic segmentation model. Specifically, a test point cloud data set with different feature intensities is constructed, and 5-10 groups of test data with gradient changes in feature intensity are generated by performing data enhancement on the first point cloud data set, such as adding Gaussian noise to adjust the point density and using a Poisson surface reconstruction to change the local curvature. The quantization indicators are calculated, wherein the local point density is calculated by using a K-nearest neighbor algorithm to calculate the average distance of 10 neighboring points around each point, and the reciprocal of the distance is the local point density of the point; the average value of the two principal curvatures is taken as the curvature value of the point, and the average curvature of the region is also calculated; the shallow feature extraction is performed by inputting the test point cloud data into the input layer of the first point cloud semantic segmentation model to output a feature map with a dimension of (N, M), N is the number of point clouds, and M is the number of channels; the mean and variance of the feature value of each channel are calculated, the mean is the sum of all feature values divided by N, and the variance is the square sum of the difference between each feature value and the mean divided by N-1. Finally, each group of test data is input into the model, and the model test information such as segmentation accuracy, recall rate, and IOU is recorded to establish the corresponding relationship between the feature intensity indicators and the model performance.

[0027] In step S220, a point cloud feature intensity distribution space meeting a preset segmentation accuracy is extracted based on the plurality of model test information. Specifically, a preset segmentation accuracy threshold is set, such as IOU≥80%, accuracy≥85%, and model test information meeting the threshold is selected. The value range of the quantization indicators of the point cloud feature intensity corresponding to the qualified test information is extracted, for example, the value range of the local point density is 0.6-1.0, the value range of the curvature is 0.2-0.4, the value range of the channel mean of the feature map is 1.0-1.4, and the value range of the channel variance is 0.3-0.7. The value ranges of these indicators are combined to form a multi-dimensional space, such as a four-dimensional space: local point density×curvature×channel mean×channel variance, which is the point cloud feature intensity distribution space meeting the preset segmentation accuracy. When the four indicators of any point cloud data fall within the range, the model can directly output a result meeting the accuracy.

[0028] Step S230, collect the real-time point cloud feature intensity distribution of the point cloud data to be segmented, and determine whether the point cloud feature intensity distribution space is satisfied. If not, activate the feature adaptive gain control module, wherein the feature adaptive gain control module includes a point cloud intensity detector, a reference comparator, and a gain smoothing unit. The reference comparator stores a preset point cloud feature intensity distribution space. Specifically, the real-time feature intensity indicators of the point cloud data to be segmented are collected, and the calculation method is consistent with step S210. The reference comparator compares the real-time indicators one by one through the preset distribution space parameters, determines whether each indicator falls within the corresponding range, and determines that the distribution space is satisfied if all indicators satisfy the distribution space, otherwise, it is determined that the distribution space is not satisfied. The core of the point cloud intensity detector is a real-time calculation module, which adopts a parallel computing architecture at the hardware level or a multi-threaded computing at the software level. The gain smoothing unit pre-stores filtering algorithms such as moving average filtering and Gaussian filtering, which are directly called after the module is activated. If the real-time indicators do not satisfy the distribution space, the module activation signal is triggered, and the subsequent feature modulation process is started.

[0029] In a possible implementation, when the first point cloud semantic segmentation model is used to perform semantic segmentation on the point cloud data to be segmented collected by the second point cloud sensor, step S200 further includes step S240 of presetting a plurality of point cloud sensors based on historical point cloud segmentation requirements. Specifically, the types of point cloud sensors involved in the historical segmentation scenarios are sorted, classified according to sensor parameters and application scenarios, a common sensor model library is preset, for example, including 16-line, 32-line, 64-line, and 128-line lidar, and certain brand millimeter wave radar, depth camera, etc. For each sensor, record its hardware parameters, such as 16-line lidar scanning frequency 10 Hz, point cloud density 0.3 points / square centimeter, 128-line lidar scanning frequency 20 Hz, point cloud density 1.2 points / square centimeter, and historical point cloud feature data such as local point density and curvature statistical value, and establish a sensor parameter-feature association table.

[0030] For example: the preset sensor library includes: sensor A: 16-line lidar, local point density mean 0.4, curvature mean 0.1; sensor B: 32-line lidar, local point density mean 0.6, curvature mean 0.2; sensor C: 128-line lidar, local point density mean 0.8, curvature mean 0.3; sensor D: millimeter wave radar, local point density mean 0.2, curvature mean 0.05.

[0031] Step S250, point cloud difference analysis is performed on the plurality of point cloud sensors, a fixed gain coefficient is configured, and a sensor-fixed gain mapping is established. Specifically, the point cloud data set collected by each preset sensor is obtained, difference analysis is performed with the first point cloud data set, and the feature compensation factor of the two data sets in the local point density, curvature, and feature map channel statistics is calculated. The feature compensation factor = the first point cloud data set index mean value / the sensor data set index mean value. The difference value = (the sensor data set index mean value - the first point cloud data set index mean value) / the first point cloud data set index mean value) can also be calculated to assist in judging the feature attenuation / gain degree, and then the rationality of the feature compensation factor is verified based on the difference value. The fixed gain coefficient is designed based on the feature compensation factor, and the calculation of the gain coefficient follows the complementary principle, that is, the feature index of the sensor data set is multiplied by the coefficient, and the corresponding index mean value of the first point cloud data set is consistent. The model or unique identifier of each sensor is associated with a corresponding set of fixed gain coefficients, such as density, curvature, channel mean value, and channel variance gain coefficient, and is stored in the form of key-value pair to form a sensor-fixed gain mapping table.

[0032] Step S260, it is judged whether the sensor-fixed gain mapping contains the fixed gain coefficient corresponding to the second point cloud sensor, if yes, the original features of the to-be-segmented point cloud data are directly modulated by the corresponding fixed gain coefficient. Specifically, the unique identifier of the second point cloud sensor is obtained, such as reading the model and production batch information through the hardware interface of the sensor. The sensor-fixed gain mapping table is queried, and whether there is a corresponding key in the table is searched by calculating the hash value of the unique identifier. If there is, the corresponding fixed gain coefficient array is extracted, and the original features are modulated, wherein the local point density modulation is the original density value multiplied by the density gain coefficient, the curvature modulation is the original curvature value multiplied by the curvature gain coefficient, the feature map channel mean value modulation is the original mean value multiplied by the channel mean value gain coefficient, and the channel variance modulation is the original variance multiplied by the channel variance gain coefficient. The modulation process is realized by matrix multiplication, for example, the feature map is an (N, M) matrix, and the channel mean value gain coefficient is an M-dimensional vector. Multiply all elements of each channel of the feature map by the corresponding gain coefficient to obtain the modulated feature map.

[0033] In a possible implementation, the point cloud difference analysis is performed for the plurality of point cloud sensors, a fixed gain coefficient is configured, a sensor-fixed gain mapping is established, and step S250 further includes step S251, point cloud segmentation data acquisition is performed for the plurality of point cloud sensors, difference comparison is performed with the first point cloud data set, a gain modulation test is performed according to a comparison result by calling the first point cloud semantic segmentation model, and a plurality of fixed gain coefficients of the feature adaptive gain control module about the plurality of point cloud sensors are generated. Specifically, point cloud segmentation data of each preset sensor is collected, and the point cloud segmentation data includes original point cloud and a corresponding semantic label, and it is ensured that the data scene is consistent with the first point cloud data set, for example, both are urban road daytime scenes. The difference comparison adopts a point-by-point feature comparison manner, and the absolute error and the relative error of the local point density, the curvature and the like of the same target region, such as the same vehicle, and the Euclidean distance of feature map channel statistics are calculated. A gain modulation test scheme is designed, a search range of the gain coefficient is set based on the difference comparison result, a grid search method is used to perform traversal test on the gain coefficient of each sensor, each set of gain coefficients is applied to the point cloud data of the sensor, the first point cloud semantic segmentation model is input, and the segmentation IOU is recorded. A set of gain coefficients corresponding to the highest IOU is selected as the fixed gain coefficient of the sensor.

[0034] Step S252, the plurality of point cloud sensors are identified and associated with the plurality of fixed gain coefficients, and a sensor-fixed gain mapping is established. Specifically, a unique identifier, such as a sensor model + production batch, is assigned to each preset sensor, the identifier is used as a key, and the corresponding fixed gain coefficient is used as a value to construct a key-value pair data set. The mapping relationship is stored by using a hash table data structure, the key of the hash table is the hash value of the unique identifier of the sensor, and the value is the fixed gain coefficient array. At the same time, an update mechanism of the mapping table is established, when a new sensor type is added, the fixed gain coefficient of the new sensor type is calculated through step S251, and the hash table is automatically inserted. When the sensor parameters are changed, such as the point cloud density is changed due to firmware upgrade, the gain coefficient is recalculated and the value of the corresponding key is updated.

[0035] Step S300, if yes, the point cloud to be segmented is input into the feature adaptive gain control module, the feature adaptive gain control module generates a corresponding feature gain coefficient based on feature intensity information of the point cloud to be segmented, modulates original features of the point cloud to be segmented by using the feature gain coefficient, and obtains adapted point cloud features.

[0036] Specifically, the feature intensity information refers to a set of indexes capable of quantifying the local geometric feature contrast of the point cloud, such as local point density, curvature, etc. The feature gain coefficient is an adjustment parameter calculated according to the feature intensity information, such as an amplification coefficient, a reduction coefficient, used to adjust the intensity of the original features of the point cloud to be segmented. The original features refer to the shallow features extracted by the model input layer, such as the coordinate difference value of the points, the reflection intensity correlation features. The adapted point cloud features are the point cloud features after the original features are modulated by the gain coefficient, and the feature intensity distribution tends to be consistent with the feature distribution of the first point cloud dataset. The core of this step is to convert the heterogeneous sensor point cloud features into the feature distribution familiar to the model through dynamic adjustment of the module, solving the cross-sensor adaptation problem.

[0037] In a possible implementation, the feature adaptive gain control module generates a corresponding feature gain coefficient based on the feature intensity information of the point cloud data to be segmented, and step S300 further includes step S310 of calculating the point cloud feature intensity of each point in the point cloud data to be segmented by the point cloud intensity detector to generate the feature intensity information. Specifically, the point cloud intensity detector realizes real-time calculation by hardware acceleration or software optimization, and the specific steps are as follows: 1. Point cloud preprocessing: denoising and downsampling of the original point cloud to reduce the calculation amount; 2. Local point density calculation: for each point after preprocessing, use the K nearest neighbor algorithm to search for 10 nearest points around it, calculate the Euclidean distance between the point and each neighboring point, take the average of the 10 distances, and the local point density = 1 / average distance; 3. Curvature calculation: for each point, construct its local coordinate system, take 50 neighboring points around it to fit a plane, calculate the two principal curvatures of the plane by principal component analysis, and the curvature value = (first principal curvature + second principal curvature) / 2; 4. Feature map channel statistic calculation: input the preprocessed point cloud into the input layer of the first point cloud semantic segmentation model, output an (N, M) shallow feature map, and calculate the mean and variance of N feature values of each channel; 5. Feature intensity information integration: integrate the local point density, curvature, and channel mean and variance of the feature map of each point into a four-dimensional feature vector to form the feature intensity information.

[0038] Step S320, compare the feature intensity information with the point cloud feature intensity distribution space stored in the reference comparator, generate the feature gain coefficient, and at the same time, the gain smoothing unit identifies feature activation noise based on the feature gain coefficient and matches the gain smoothing filtering parameter. Specifically, the point cloud feature intensity distribution space stored in the reference comparator is the qualified range of each index, including the qualified range of local point density, the qualified range of curvature, the qualified range of feature map channel mean value, and the qualified range of feature map channel variance. The standard reference value of each index is pre-stored, that is, the mean value of the corresponding index of the first point cloud data set, which is denoted as the reference mean value. The ratio of the real-time feature intensity index to the corresponding reference mean value is calculated as the gain coefficient, wherein the density gain coefficient = local point density reference mean value / real-time local point density mean value, the curvature gain coefficient = curvature reference mean value / real-time curvature mean value, the channel mean value gain coefficient = feature map channel reference mean value / real-time feature map channel mean value, and the channel variance gain coefficient = feature map channel reference variance / real-time feature map channel variance.

[0039] The gain smoothing unit calculates the gradient change rate of the gain coefficient, that is, the absolute value of the difference between adjacent frame gain coefficients. If the gradient change rate exceeds a predetermined threshold, it is determined that there is feature activation noise, such as feature mutation caused by sensor burst interference. A plurality of sets of smoothing filtering parameters are pre-stored, such as the window size of the moving average filter and the standard deviation of the Gaussian filter. According to the noise intensity, the parameters are matched, such as using a sliding average filter with a window size of 3 for low noise, using a sliding average filter with a window size of 5 for medium noise, and using a Gaussian filter with a standard deviation of 0.5 for high noise.

[0040] Step S330, modulate the original features of the point cloud data to be segmented according to the feature gain coefficient, and filter the modulated point cloud with the gain smoothing filtering parameter to generate the adapted point cloud features. Specifically, a combination of point-by-point modulation and channel modulation is used. The point-by-point modulation is for local point density and curvature. The density value of each point is multiplied by the density gain coefficient, and the curvature value is multiplied by the curvature gain coefficient. The channel modulation is for the feature map. All elements of each channel of the feature map are multiplied by the mean value gain coefficient of the channel, and the variance is adjusted. The modulated variance = original variance x variance gain coefficient 2 . The modulated point cloud features are filtered. If a sliding average filter is used, the average value of the feature values of the neighboring points within the window size around each point is calculated as the final feature value of the point. If a Gaussian filter is used, the weight of each neighboring point is calculated according to the Gaussian function, and the final feature value of the point is obtained by weighted summation. The filtered feature value is mapped to the interval [0, 1] to avoid numerical overflow affecting model calculation, and finally the adapted point cloud features are generated.

[0041] Step S400, input the adapted point cloud feature into the first point cloud semantic segmentation model, and output the semantic segmentation result of the point cloud data to be segmented.

[0042] Specifically, the adapted point cloud feature is consistent with the first point cloud data set during model training in terms of feature intensity distribution, and can be effectively recognized by the first point cloud semantic segmentation model. The model determines the semantic category of each point cloud point through the learned feature-semantic label mapping relationship, such as distinguishing vehicles, pedestrians, and obstacles, and finally outputs the segmentation result containing the semantic label of each point, for example, storing the category ID of each point in an array form.

[0043] In a possible implementation, the method further includes step S500, the feature adaptive gain control module is located at the front end of the first point cloud semantic segmentation model for one-time modulation to obtain the adapted point cloud feature.

[0044] Specifically, the feature adaptive gain control module is deployed before the input layer of the first point cloud semantic segmentation model, forming a process of data input→modulation by the module→model inference. The module and the model realize data transmission through a data interface. The point cloud data to be segmented is first input into the feature adaptive gain control module, and after intensity detection, gain calculation, and modulation filtering, the adapted point cloud feature is directly output. The dimension and data format of the feature are completely consistent with the requirements of the input layer of the model. One-time modulation means that the data only needs to be processed once by the feature adaptive gain control module, without the need for repeated modulation, which ensures the inference efficiency and is suitable for scenarios where the sensor type changes little and the feature difference is stable.

[0045] In a possible implementation, the method further includes step S600, according to the number of network layers of the first point cloud semantic segmentation model, one feature adaptive gain control module is configured at each network layer, and feature extraction of the network layer is performed at each network layer, and one-time point cloud data modulation is performed.

[0046] Specifically, the network structure of the first point cloud semantic segmentation model is analyzed to determine the number and location of all feature extraction layers. For example, if the model contains an input layer, convolutional layer 1, convolutional layer 2, pooling layer 1, convolutional layer 3, fully connected layer 1, and output layer—a total of 5 feature extraction layers—then 5 feature adaptive gain control modules are configured. The parameters of each feature adaptive gain control module are adapted to the network layer. Based on the output feature dimension of the corresponding network layer, the feature intensity calculation dimension of the feature adaptive gain control module is adjusted to ensure that it can process the feature data of the corresponding layer. The reference comparator of each feature adaptive gain control module stores the intensity distribution space of the features of that layer. This intensity distribution space is obtained by statistically analyzing the index range of the output features of that layer during training on the first point cloud dataset. During model inference, after the point cloud to be segmented has undergone feature extraction by the previous network layer, the output feature data is first input into the corresponding configured feature adaptive gain control module for intensity detection, gain calculation, and modulation filtering. Then, it is input into the current network layer for feature extraction, and this process is iterated until all network layers have been processed. This approach is suitable for scenarios with large differences in features across sensors. By modulating layer by layer, it ensures that the features of each layer are adapted to the distribution during model training, thereby improving segmentation accuracy.

[0047] This application embodiment trains a first point cloud semantic segmentation model using a first point cloud dataset collected by a first point cloud sensor. When using this model to perform semantic segmentation on point cloud data to be segmented collected by a second point cloud sensor, it first determines whether the feature adaptive gain control module needs to be activated. If activation is required, the point cloud to be segmented is input into the module. The module generates feature gain coefficients based on the feature intensity information of the point cloud to be segmented and uses them to modulate the original features of the point cloud to be segmented to obtain adapted point cloud features. Finally, the adapted point cloud features are input into the first point cloud semantic segmentation model, and the semantic segmentation result of the point cloud data to be segmented is output. These technical means solve the technical problems of existing 3D point cloud-based semantic segmentation models being sensitive to differences in the feature distribution of point clouds from different sources and having weak cross-domain adaptability. This achieves the technical effect of improving the model's feature adaptation capability for point clouds from different sources and improving the accuracy of environmental perception and system reliability in cross-sensor scenarios.

[0048] In the above text, refer to Figure 1 A semantic segmentation method based on 3D point clouds according to embodiments of the present invention is described in detail. Next, reference will be made to... Figure 2 A semantic segmentation system based on 3D point clouds according to an embodiment of the present invention is described.

[0049] The 3D point cloud-based semantic segmentation system according to the embodiment of the present application is used to solve the technical problem that the existing 3D point cloud-based semantic segmentation model is sensitive to the feature distribution difference of heterogeneous sensor point clouds and has weak cross-domain adaptability, so as to improve the feature adaptation capability of the model to the heterogeneous sensor point clouds and improve the accuracy and system reliability of environment perception in the cross-sensor scene. The 3D point cloud-based semantic segmentation system comprises a first point cloud semantic segmentation model training module 10, a judgment module 20, a feature adaptive gain module 30, and a semantic segmentation module 40.

[0050] The first point cloud semantic segmentation model training module 10 is configured to train a first point cloud semantic segmentation model by using a first point cloud dataset collected by a first point cloud sensor. The judgment module 20 is configured to determine whether to activate a feature adaptive gain control module when performing semantic segmentation on to-be-segmented point cloud data collected by a second point cloud sensor by using the first point cloud semantic segmentation model. If not, the to-be-segmented point cloud data is directly input into the first point cloud semantic segmentation model, and a semantic segmentation result of the to-be-segmented point cloud data is output. The feature adaptive gain module 30 is configured to input the to-be-segmented point cloud into the feature adaptive gain control module if the determination result is yes. The feature adaptive gain control module generates a corresponding feature gain coefficient based on the feature intensity information of the to-be-segmented point cloud data, modulates the original features of the to-be-segmented point cloud data by using the feature gain coefficient, and obtains adapted point cloud features. The semantic segmentation module 40 is configured to input the adapted point cloud features into the first point cloud semantic segmentation model, and output the semantic segmentation result of the to-be-segmented point cloud data.

[0051] The judgment module 20 is specifically configured as follows: as described above, when performing semantic segmentation on to-be-segmented point cloud data collected by a second point cloud sensor by using the first point cloud semantic segmentation model, it is determined whether to activate a feature adaptive gain control module. The judgment module 20 can further comprise: a test unit configured to perform tests on different point cloud feature intensities of the first point cloud semantic segmentation model, and generate a plurality of model test information corresponding to a plurality of point cloud feature intensities; a point cloud feature intensity distribution space extraction unit configured to extract a point cloud feature intensity distribution space satisfying a preset segmentation accuracy based on the plurality of model test information; and a judgment unit configured to collect real-time point cloud feature intensity distribution of the to-be-segmented point cloud data, and determine whether the point cloud feature intensity distribution satisfies the point cloud feature intensity distribution space. If not, the feature adaptive gain control module is activated.

[0052] The test unit can further include: the quantification indicators of the point cloud feature intensity include local point density, curvature, and channel statistics of an early feature map obtained after shallow feature extraction on the point cloud, and the channel statistics include mean or variance; and the shallow features are calculated by inputting the point cloud data into an input layer of the first point cloud semantic segmentation model.

[0053] The judgment unit can further include: the feature adaptive gain control module includes a point cloud intensity detector, a reference comparator, and a gain smoothing unit, and the reference comparator stores a preset point cloud feature intensity distribution space.

[0054] The feature adaptive gain module 30 is specifically configured as follows: as described above, the feature adaptive gain control module generates a corresponding feature gain coefficient based on the feature intensity information of the to-be-segmented point cloud data, and the feature adaptive gain module 30 can further include: a point cloud feature intensity calculation unit for calculating the point cloud feature intensity of each point in the to-be-segmented point cloud data through the point cloud intensity detector to generate the feature intensity information; a feature gain coefficient generation unit for comparing the feature intensity information with the point cloud feature intensity distribution space stored in the reference comparator to generate the feature gain coefficient; a gain smoothing unit for identifying feature activation noise based on the feature gain coefficient and matching a gain smoothing filter parameter; a modulation filtering unit for modulating the original features of the to-be-segmented point cloud data according to the feature gain coefficient and filtering the modulated point cloud with the gain smoothing filter parameter to generate the adapted point cloud features.

[0055] The judgment module 20 can further include: a point cloud sensor preset unit for presetting a plurality of point cloud sensors based on historical point cloud segmentation requirements; a fixed gain coefficient configuration unit for performing point cloud difference analysis on the plurality of point cloud sensors, configuring a fixed gain coefficient, and establishing a sensor-fixed gain mapping; and a mapping modulation unit for judging whether the sensor-fixed gain mapping contains a fixed gain coefficient corresponding to the second point cloud sensor, and if so, directly modulating the original features of the to-be-segmented point cloud data with the corresponding fixed gain coefficient.

[0056] The point cloud difference analysis is performed on the plurality of point cloud sensors, a fixed gain coefficient is configured, a sensor-fixed gain mapping is established, and the fixed gain coefficient configuration unit can further include: a gain modulation test subunit configured to perform point cloud segmentation data acquisition on the plurality of point cloud sensors, perform difference comparison with the first point cloud data set, and perform gain modulation test according to the comparison result by calling the first point cloud semantic segmentation model to generate a plurality of fixed gain coefficients of the feature adaptive gain control module with respect to the plurality of point cloud sensors; and a mapping establishment subunit configured to identify and associate the plurality of point cloud sensors with the plurality of fixed gain coefficients to establish a sensor-fixed gain mapping.

[0057] The system can further include that the feature adaptive gain control module is located at the front end of the first point cloud semantic segmentation model to perform one-time modulation to obtain adapted point cloud features.

[0058] The system can further include that according to the number of network layers of the first point cloud semantic segmentation model, one feature adaptive gain control module is configured at each network layer, and feature extraction of the network layer is performed at each network layer to perform one-time point cloud data modulation.

[0059] The semantic segmentation system based on 3D point cloud provided in the embodiments of the present application can execute the semantic segmentation method based on 3D point cloud provided in any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0060] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, however, any number of different modules can be used and run on the user terminal and / or server, and each unit and module is only divided according to the functional logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy mutual differentiation, and do not limit the protection scope of the present application.

[0061] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with the preferred embodiment, however, it is not intended to limit the present application, and any person skilled in the art can make some changes or modifications to the above disclosed technical content to make equivalent embodiments with equivalent changes, as long as the changes do not deviate from the technical solution of the present application. Any modification, equivalent change and modification of the above embodiments according to the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. A semantic segmentation method based on 3D point clouds, characterized in that, include: A first point cloud semantic segmentation model is trained using the first point cloud dataset, which is collected by the first point cloud sensor. When using the first point cloud semantic segmentation model to perform semantic segmentation on the point cloud data to be segmented collected by the second point cloud sensor, it is determined whether the feature adaptive gain control module is activated. If not, the point cloud data to be segmented is directly input into the first point cloud semantic segmentation model, and the semantic segmentation result of the point cloud data to be segmented is output. If so, the point cloud to be segmented is input into the feature adaptive gain control module. The feature adaptive gain control module generates a corresponding feature gain coefficient based on the feature intensity information of the point cloud data to be segmented, and uses the feature gain coefficient to modulate the original features of the point cloud data to be segmented to obtain the adapted point cloud features. The adapted point cloud features are input into the first point cloud semantic segmentation model, and the semantic segmentation result of the point cloud data to be segmented is output.

2. The semantic segmentation method based on 3D point clouds as described in claim 1, characterized in that, When performing semantic segmentation on the point cloud data to be segmented collected by the second point cloud sensor using the first point cloud semantic segmentation model, the process of determining whether to activate the feature adaptive gain control module includes: The first point cloud semantic segmentation model is tested with different point cloud feature intensities, generating multiple model test information corresponding to multiple point cloud feature intensities; Based on the test information of the multiple models, a point cloud feature intensity distribution space that meets the preset segmentation accuracy is extracted; The real-time point cloud feature intensity distribution of the point cloud data to be segmented is collected, and it is determined whether the point cloud feature intensity distribution space is satisfied. If not, the feature adaptive gain control module is activated.

3. The semantic segmentation method based on 3D point clouds as described in claim 2, characterized in that, The quantitative indicators of point cloud feature intensity include local point density, curvature, and channel statistics of the early feature map obtained after shallow feature extraction of the point cloud, wherein the channel statistics include mean or variance. Shallow features are calculated by inputting point cloud data into the input layer of the first point cloud semantic segmentation model.

4. The semantic segmentation method based on 3D point clouds as described in claim 2, characterized in that, The feature adaptive gain control module includes a point cloud intensity detector, a reference comparator, and a gain smoothing unit. The reference comparator stores a preset point cloud feature intensity distribution space.

5. The semantic segmentation method based on 3D point clouds as described in claim 4, characterized in that, The feature adaptive gain control module generates corresponding feature gain coefficients based on the feature intensity information of the point cloud data to be segmented, including: The point cloud intensity detector calculates the point cloud feature intensity of each point in the point cloud data to be segmented, and generates the feature intensity information. The feature intensity information is compared with the point cloud feature intensity distribution space stored in the reference comparator to generate the feature gain coefficient. At the same time, the gain smoothing unit identifies feature activation noise based on the feature gain coefficient and matches the gain smoothing filter parameters. The original features of the point cloud data to be segmented are modulated according to the feature gain coefficient, and the modulated point cloud is filtered with the gain smoothing filter parameter to generate the adapted point cloud features.

6. The semantic segmentation method based on 3D point clouds as described in claim 5, characterized in that, When using the first point cloud semantic segmentation model to perform semantic segmentation on the point cloud data to be segmented collected by the second point cloud sensor, the method further includes: Multiple point cloud sensors are preset based on historical point cloud segmentation requirements; Point cloud difference analysis was performed on the various point cloud sensors, a fixed gain coefficient was configured, and a sensor-fixed gain mapping was established. Determine whether the sensor-fixed gain mapping includes the fixed gain coefficient corresponding to the second point cloud sensor. If so, directly modulate the original features of the point cloud data to be segmented using the corresponding fixed gain coefficient.

7. The semantic segmentation method based on 3D point clouds as described in claim 6, characterized in that, Point cloud difference analysis is performed on the various point cloud sensors, a fixed gain coefficient is configured, and a sensor-fixed gain mapping is established, including: Point cloud segmentation data is collected for the various point cloud sensors, and the difference is compared with the first point cloud dataset. Based on the comparison result, the first point cloud semantic segmentation model is called to perform gain modulation test, and multiple fixed gain coefficients of the feature adaptive gain control module for the various point cloud sensors are generated. The various point cloud sensors are identified and associated with the various fixed gain coefficients to establish a sensor-fixed gain mapping.

8. The semantic segmentation method based on 3D point clouds as described in claim 1, characterized in that, The feature adaptive gain control module is located at the front end of the first point cloud semantic segmentation model and performs a modulation to obtain adapted point cloud features.

9. The semantic segmentation method based on 3D point clouds as described in claim 1, characterized in that, Based on the number of network layers in the first point cloud semantic segmentation model, a feature adaptive gain control module is configured in each network layer. Feature extraction of each network layer is performed, and point cloud data modulation is performed once in each layer.

10. A semantic segmentation system based on 3D point clouds, characterized in that, The system is used to implement the semantic segmentation method based on 3D point clouds as described in any one of claims 1-9, and the system comprises: The first point cloud semantic segmentation model training module is used to train the first point cloud semantic segmentation model using the first point cloud dataset, which is collected by the first point cloud sensor. The judgment module is used to determine whether to activate the feature adaptive gain control module when performing semantic segmentation on the point cloud data to be segmented collected by the second point cloud sensor using the first point cloud semantic segmentation model. If not, the point cloud data to be segmented is directly input into the first point cloud semantic segmentation model and the semantic segmentation result of the point cloud data to be segmented is output. The feature adaptive gain module is used to input the point cloud to be segmented into the feature adaptive gain control module if the judgment result is yes. The feature adaptive gain control module generates a corresponding feature gain coefficient based on the feature intensity information of the point cloud data to be segmented, and uses the feature gain coefficient to modulate the original features of the point cloud data to be segmented to obtain the adapted point cloud features. The semantic segmentation module is used to input the adapted point cloud features into the first point cloud semantic segmentation model and output the semantic segmentation result of the point cloud data to be segmented.

Images