Domain adaptive lidar point cloud semantic segmentation method, device and storage medium

Abstract

The application discloses a domain adaptive laser radar point cloud semantic segmentation method and device and a storage medium, and comprises the following steps: step 1, a laser radar point cloud semantic segmentation network model is preliminarily trained through a target domain simulation sampling data alignment mode, the laser radar point cloud semantic segmentation network model is trained through a scene mixing and pseudo label correction combined mode based on the preliminarily trained laser radar point cloud semantic segmentation network model, and a trained laser radar point cloud semantic segmentation network model is obtained; and step 2, the target domain point cloud data is subjected to semantic segmentation through the trained laser radar point cloud semantic segmentation network model, and a semantic segmentation result of the target domain point cloud data is obtained. According to the method, the training of the laser radar point cloud semantic segmentation network model can be completed without or with only a small amount of target scene data annotation, and the semantic segmentation of the target domain laser radar point cloud data can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a domain-adaptive LiDAR point cloud semantic segmentation method. Background Technology

[0002] Point cloud semantic segmentation is a fundamental task in computer vision, with wide applications in robotics and automation. Semantic segmentation of LiDAR point clouds is particularly important for scene understanding in autonomous driving. To address this challenge, many current methods use manually labeled point cloud data to supervise the training of deep neural network models, such as those published in: B. Wu, et al., “SqueezeSeg: Convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3D LiDAR point cloud”, in ICRA, 2018; Y. Zhang, et al., “PolarNet: An improved grid representation for online LiDAR point clouds semantic segmentation”, in CVPR, 2020; X. Zhu, et al., “Cylindrical andasymmetrical 3D convolution networks for LiDAR segmentation”, in CVPR, 2021. These methods use the trained models to perform semantic segmentation on the acquired LiDAR point clouds during the testing phase. However, this fully supervised deep learning method relies on costly and limited manually labeled training data. Changes in real-world application scenarios (including changes in sensors and scene content) may lead to significant differences in data characteristics between point cloud data in the testing phase and point cloud data in the training phase, ultimately resulting in poor semantic segmentation test results on target domain point cloud data.In recent years, unsupervised domain adaptation semantic segmentation methods, such as those published in AAAI (S. Zhao et al., “ePointDA: An end-to-end simulation-to-real domain adaptation framework for LiDAR point clouds segmentation”) and CVPR (L. Yi et al., “Complete & label: A domain adaptation approach to semantic segmentation of LiDAR point clouds”), have received widespread attention. These methods utilize existing labeled source domain LiDAR point cloud data (referred to as source domain point cloud data) and unlabeled target domain LiDAR point cloud data (referred to as target domain point cloud data) to train deep neural network models, improving the test results of deep neural network models on target domain point cloud data. This method eliminates the need for manual data relabeling, reducing practical application costs. However, due to the lack of accurate guiding information, its current performance in LiDAR point cloud semantic segmentation is far from meeting practical needs.

[0003] In view of this, the present invention is hereby proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a domain-adaptive lidar point cloud semantic segmentation method, device, and storage medium that can accurately segment target domain lidar point cloud data under unsupervised or semi-supervised conditions, meet the needs of practical scenarios, and thus solve the aforementioned technical problems existing in the prior art.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] This invention provides a domain-adaptive lidar point cloud semantic segmentation method, comprising:

[0007] Step 1: Initially train the LiDAR point cloud semantic segmentation network model by aligning the target domain simulated sampling data. Based on the initially trained LiDAR point cloud semantic segmentation network model, train the LiDAR point cloud semantic segmentation network model by combining scene mixing and pseudo-label correction to obtain the trained LiDAR point cloud semantic segmentation network model.

[0008] Step 2: Use the trained lidar point cloud semantic segmentation network model to perform semantic segmentation on the target domain point cloud data to obtain the semantic segmentation results of the target domain point cloud data.

[0009] This invention also provides a processing apparatus, comprising:

[0010] At least one memory for storing one or more programs;

[0011] At least one processor is capable of executing one or more programs stored in the memory, such that when the processor executes one or more programs, the processor can implement the method of the present invention.

[0012] This invention further provides a readable storage medium storing a computer program that, when executed by a processor, can implement the method described in this invention.

[0013] Compared with existing technologies, the domain-adaptive lidar point cloud semantic segmentation method, device, and storage medium provided by this invention have the following advantages:

[0014] The LiDAR point cloud semantic segmentation network model is initially trained by aligning the target domain simulated sampling data. Source domain simulated sampling point cloud data conforming to the target domain sampling pattern is then used to train the LiDAR point cloud semantic segmentation network model. This reduces the spatial differences between the training data and the actual segmented target domain point cloud data, especially the point cloud density differences caused by different sensor line counts, thus minimizing the impact on the LiDAR point cloud semantic segmentation network model. Furthermore, training the LiDAR point cloud semantic segmentation network model using a combination of scene blending and pseudo-label correction reduces the interference of pseudo-label bias and noise on model training. This invention allows for the training of the LiDAR point cloud semantic segmentation network model with little or no target domain point cloud data annotation, enabling accurate semantic segmentation of target domain point cloud data in target scenes. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of a domain-adaptive lidar point cloud semantic segmentation method provided in an embodiment of the present invention.

[0017] Figure 2 Comparison of single-frame point cloud and multi-frame fused point cloud for the semantic segmentation method provided in this embodiment of the invention; wherein, (a) is a single-frame point cloud and (b) is a multi-frame fused point cloud.

[0018] Figure 3The diagram shows a comparison between the original source domain point cloud data and the simulated source domain sampling point cloud data of the domain-adaptive lidar point cloud semantic segmentation method provided in this embodiment of the invention; wherein, (a) is a schematic diagram of the original source domain point cloud data, and (b) is a schematic diagram of the simulated source domain sampling point cloud data of the simulated target domain sampling mode.

[0019] Figure 4 A schematic diagram of a data alignment training framework based on simulated scanning for the semantic segmentation method provided in this embodiment of the invention;

[0020] Figure 5 This is a schematic diagram of a self-training framework based on scene blending and pseudo-label correction for the semantic segmentation method provided in this embodiment of the invention.

[0021] Figure 6 The diagram shows scene-mixed point cloud data for the semantic segmentation method provided in this embodiment of the invention; wherein, (a) is a diagram of the real label of the source domain simulated sampled point cloud data, (b) is a diagram of the pseudo label of the source domain simulated sampled point cloud data, and (c) is a diagram of the label of the scene-mixed point cloud data.

[0022] Figure 7 This diagram illustrates the input target domain point cloud data, ground truth labels, and semantic segmentation results of the semantic segmentation method provided in this embodiment of the invention; wherein, (a) is the input point cloud image, (b) is the ground truth semantic label of the point cloud, and (c) is the semantic segmentation result image using this method. Detailed Implementation

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0024] First, the following explanations are provided for the terms that may be used in this article:

[0025] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0026] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0027] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0028] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0029] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience and simplification of description and do not imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0030] The domain-adaptive lidar point cloud semantic segmentation method provided by this invention will be described in detail below. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0031] like Figure 1As shown, this embodiment of the invention provides a domain-adaptive lidar point cloud semantic segmentation method, including:

[0032] Step 1: Initially train the LiDAR point cloud semantic segmentation network model (hereinafter referred to as the point cloud semantic segmentation network model) by aligning the target domain simulated sampling data. Based on the initially trained LiDAR point cloud semantic segmentation network model, train the LiDAR point cloud semantic segmentation network model by combining scene mixing and pseudo-label correction to obtain the trained LiDAR point cloud semantic segmentation network model.

[0033] Step 2: Use the trained LiDAR point cloud semantic segmentation network model to perform semantic segmentation on the target domain point cloud data (short for target domain LiDAR point cloud data) to obtain the semantic segmentation result of the target domain point cloud data.

[0034] In step 1 of the above method, the semantic segmentation network model of the LiDAR point cloud is initially trained according to the alignment method of the simulated sampling data of the target domain, including:

[0035] Step 11: By fusing the sequential point cloud of the source domain point cloud data (short for source domain lidar point cloud data) and the simulated sampling of the target domain, source domain simulated sampling point cloud data that conforms to the target domain sampling pattern is obtained.

[0036] Step 12: Train the lidar point cloud semantic segmentation network model using the source domain simulated sampling point cloud data that conforms to the target domain sampling pattern, and obtain the initially trained lidar point cloud semantic segmentation network model.

[0037] In step 11 above, source domain simulated sampling point cloud data conforming to the target domain sampling pattern is obtained by fusing sequential point clouds of source domain point cloud data and simulating sampling of the target domain in the following manner:

[0038] Step 111: Using the continuous point cloud sequence in the source domain point cloud data and the pose data of the laser scanning device, the multi-frame point cloud data in the source domain point cloud data is fused to obtain the dense point cloud data of the current scene.

[0039] Step 112: In the current scene where dense point cloud data is obtained, select a point to place a virtual lidar and use that point as the origin of the coordinate system. In the current scene, use the virtual lidar to simulate the target domain lidar and scan and resample according to the target domain sampling mode to obtain new single-frame target domain point cloud data as source domain simulated sampling point cloud data.

[0040] In step 112 above, new single-frame target domain point cloud data is obtained by resampling using a virtual lidar to simulate the target domain lidar in the target domain sampling mode, including:

[0041] Step 1121: Convert the Cartesian coordinates of each point in the dense point cloud data Q to spherical coordinates using the following calculation formula. The three-dimensional coordinates of each 3D point in Q in the Cartesian coordinate system with the virtual LiDAR as the origin are (x... i y i , z i The entire dense point cloud Q contains N Q A three-dimensional point, i.e., Q = {q} i |q i =(x i y i , z i ), i = 1, ..., N Q}, for each 3D point q i Transform the expression into spherical coordinates q′ using the following formula. i , denoted as (r i θ i , φ i ),

[0042]

[0043] Where, r i θ is the radius; i θ is the angle with the z-axis. i The range is the scanning line angle range of the laser scanning equipment [-θ] down θ up ];φ i φ is the horizontal rotation angle. i The range is [-π, π]; then we obtain the spherical coordinate representation Q' corresponding to Q, Q′={q′ i |q′ i =(r i θ i , φ i ), θ i ∈[-θ down θ up ], φ i ∈[-π,π],i=1,…,N Q};

[0044] Step 1122: Generate the corresponding target domain sampling pattern based on the angular resolution and line count of the virtual lidar, and obtain Np new scanning rays, each ray's direction determined by angle θ. i , φ i Define the total set of scanning rays as {(θ)} i , φ i | i = 1, ..., N P Based on these scan line directions, bilinear interpolation is performed in the (θ, φ) space of the dense point cloud data Q to calculate the distance r of each spatial point on each ray from the origin.i That is, to obtain the sphere radius {r} of the point cloud in the corresponding direction. i |i = 1, ..., N P};

[0045] Step 1123, obtain the point set spherical coordinates {(r) after sampling by the virtual lidar simulation target domain lidar scanning method. i θ i , φ i | i = 1, ..., N P} Convert to Cartesian coordinates to obtain new single-frame target domain point cloud data P = {p i |p i =(x i y i , z i ), i = 1, ..., N P}, P contains N p There are points, each point p i The three-dimensional coordinates are (x i y i , z i For each point p in this single frame of point cloud data i Find the nearest point in the dense point cloud data Q, and use the label of the nearest point as p. i The semantic tag, denoted as l i , l i It is a C-dimensional one-hot vector, where C is the total number of categories in the semantic segmentation, l i If only one of the C elements in the data has a value of 1 in the dimension corresponding to the true category of the point, and all other elements have a value of 0, then the total label data L = {l} corresponding to the single-frame target domain point cloud data P is obtained. i |l i ∈{0,1} 1×C i = 1, ..., N P}

[0046] In step 1 of the above method, the semantic segmentation network model of the lidar point cloud adopts any one of the following: U-shaped three-dimensional convolutional neural network based on sparse convolution, MinkowskiNet50, and MinkowskiNet101.

[0047] The semantic segmentation network model for LiDAR point clouds uses cross-entropy as the penalty function, and its formula is as follows:

[0048]

[0049] Where N is the total number of points; C is the number of semantic categories in the point cloud; It is an indicator of whether the i-th point belongs to the c-th category, and it can be either 0 or 1; It is the probability of predicting the i-th point as the c-th category.

[0050] The method described above for training a LiDAR point cloud semantic segmentation network model based on the alignment of target domain simulated sampling data utilizes the labeled point cloud sequence in the source domain point cloud data and the pose data of the LiDAR scanning device to perform coordinate transformation, fusing multiple consecutive sparse point cloud frames to obtain dense point cloud data for a single scene. Then, based on the sensor settings of the target domain point cloud data, a virtual LiDAR is simulated in the dense point cloud data to re-scan and sample. The next step uses the source domain simulated sampling point cloud data conforming to the target domain sampling pattern obtained from the simulated scan, along with labels, to train a deep neural network model for the LiDAR point cloud semantic segmentation network model. This method reduces the spatial differences between the training data and the target test data, especially the point cloud density differences caused by different sensor line counts, thus minimizing the impact on the LiDAR point cloud semantic segmentation network model.

[0051] In step 1 above, the LiDAR point cloud semantic segmentation network model is trained using scene blending and pseudo-label correction methods in the following manner:

[0052] The semantic segmentation network model of LiDAR point cloud obtained by training through the alignment method of sampled data in the target domain is used as the teacher segmentation network model, and a student segmentation network model is initialized with the model parameters of the teacher segmentation network model. The student segmentation network model has the same structure as the teacher segmentation network model.

[0053] Using a batch of training data consisting of one frame of simulated source domain point cloud data and one frame of target domain point cloud data, the teacher segmentation network model and the student segmentation network model are self-trained in the following manner:

[0054] In the first training step, the target domain point cloud data is input into the teacher segmentation network model for prediction to obtain intermediate layer features and initial segmentation results. Based on the obtained intermediate layer features and initial segmentation results, the class center vector and class weight map are calculated. The predicted probability map of the initial segmentation results is weighted by the class weight map to obtain a weighted predicted probability map. The class with the highest probability in the predicted probability map is taken as the pseudo label of the corrected target domain point cloud data.

[0055] In the second training step, the target domain point cloud data and the source domain simulated sampled point cloud data are mixed and augmented with scene effects before being input into the student segmentation network model to obtain the prediction results of the student segmentation network model. The pseudo-labels of the target domain point cloud data and the real labels of the source domain point cloud data are used for supervision, with cross-entropy as the penalty function. The gradient descent method is used to update the parameters of the student segmentation network model. Then, the parameters of the teacher segmentation network model are updated according to the exponential moving average of the parameters of the student segmentation network model.

[0056] Replace the next batch of training data and repeat the above training steps to update the teacher segmentation network model and train the student segmentation network model until training is complete. The final student segmentation network model is used as the trained LiDAR point cloud semantic segmentation network model.

[0057] During the first training step described above, the learner learning rate for both the teacher segmentation network model and the student segmentation network model was set to 1×10. -3 The number of category center vectors is chosen to be K=2; the probability threshold for generating pseudo-labels for target domain point cloud data is τ=0.3.

[0058] During the second training step described above, the learner learning rate for both the teacher segmentation network model and the student segmentation network model was set to 5×10. -4 The parameters of the teacher segmentation network model are updated based on the exponential moving average of the parameters of the student segmentation network model, where the parameter α of the exponential moving average is 0.999.

[0059] During the training process of the second training step described above, the target domain point cloud data and the source domain simulated sampled point cloud data are mixed and augmented in the following manner:

[0060] Scene blending is: combining all points O = {(x...} from the source domain simulated sampled point cloud data... i y i , z i )|l i.c =1, c∈C O The point cloud data is mixed with the scene point cloud H in the target domain point cloud data to synthesize a new point cloud data P = O∪H as the scene mixed point cloud data;

[0061] Among them, the foreground point cloud refers to the point cloud in which the foreground consists of small objects such as pedestrians, bicycles, and motorcycles, that is, the label c of the point belongs to set C. O = Point cloud of {pedestrians, bicycles, motorcycles}; point cloud H is the background point cloud after removing the foreground point cloud;

[0062] Data augmentation involves applying random rotation, translation, or scaling to the mixed point cloud data obtained from scene blending. Examples include random flipping of the point cloud about the x or y axis, random scaling within the range of [0.95, 1.05], and random rotation around the z-axis in the range of [-45°, 45°]. The augmented data is then used as training data to train the student segmentation network model.

[0063] This self-training method, combining scene mixing and pseudo-label correction, trains a semantic segmentation network model for LiDAR point clouds. The LiDAR point cloud semantic segmentation network model, trained using data alignment based on simulated target domain sampling data, serves as the initial teacher segmentation network model. Unlabeled target scene data is input into the teacher segmentation network model to obtain prediction results. The high-confidence portions of these predictions are used as pseudo-labels. A student segmentation network model with the same structure as the LiDAR point cloud semantic segmentation network model is then trained using these data and pseudo-labels. The teacher segmentation network model is then updated using the exponential moving average of the student segmentation network model's parameters. Repeating these steps several times allows the teacher segmentation network model to learn the contextual information of the target scene under pseudo-label supervision. This method enables the trained model to better adapt to the contextual relationships within the target scene, overcoming the problem of the geographical environment of the target domain point cloud data differing from the training data, which negatively impacts model performance. This training method can also be used to achieve domain adaptation under semi-supervised conditions.

[0064] In the self-training process described above, the scene-blending training strategy involves adding small object point clouds from the source domain simulated sampling point cloud data to the target domain point cloud data scene during the training of the LiDAR point cloud semantic segmentation network model. This creates a hybrid point cloud as input. Since these small object point clouds have correct annotations, they can guide the semantic segmentation model's learning. The category-aware pseudo-label correction training calculates the feature centers of each category in the current point cloud, then calculates the distance between each feature point and each category center. These distances are used as weights to correct the prediction probability map of the teacher segmentation network model, resulting in new predictions. Because category centers are less sensitive to outliers in the semantic segmentation results, this method can correct some noisy labels on the classification boundaries, effectively solving the problem of pseudo-label noise caused by initial model bias.

[0065] This invention also provides a processing device, comprising:

[0066] At least one memory for storing one or more programs;

[0067] At least one processor is capable of executing one or more programs stored in the memory, such that when the processor executes one or more programs, the processor can implement the methods described above.

[0068] The present invention further provides a readable storage medium storing a computer program that can implement the above-described method when executed by a processor.

[0069] In summary, the segmentation method of this invention can accurately segment point cloud data of the target domain under unsupervised or semi-supervised conditions.

[0070] To more clearly demonstrate the technical solution and its effects provided by the present invention, the domain-adaptive lidar point cloud semantic segmentation method provided by the present invention will be described in detail below with specific embodiments.

[0071] Example 1

[0072] like Figure 1 As shown, this embodiment provides a domain-adaptive LiDAR point cloud semantic segmentation method. This method generates pseudo-labels by simulating target domain sampling patterns and data mixing to train a LiDAR point cloud semantic segmentation network model for the target domain. Then, the trained LiDAR point cloud semantic segmentation network model for the target domain is used to perform semantic segmentation on the target domain point cloud data.

[0073] The training of the point cloud semantic segmentation network model mainly consists of two steps. The first step is training based on the data alignment method of simulated scanning. The training framework is described in [link to training framework]. Figure 4 The point cloud semantic segmentation network model is trained by fusing and resampling point clouds from source domain point cloud data sequences to obtain simulated sampled point clouds from the source domain that conform to the sampling pattern of the target domain. In this first step, the learning rate of the point cloud semantic segmentation network model optimizer is set to 1×10⁻⁶ during training. -3 ;

[0074] The second step is a self-training method based on scene blending and pseudo-label correction. See the training framework below. Figure 5 The point cloud semantic segmentation network model trained in the first step is used as the teacher segmentation network model. A student segmentation network model is initialized using the model parameters of this point cloud semantic segmentation network model. The structure of the student segmentation network model is consistent with that of the point cloud semantic segmentation network model. In the first training step, each batch of data contains one frame of source domain simulated sampled point cloud data and one frame of target domain point cloud data. The target domain point cloud data is first input into the teacher segmentation network model for prediction to obtain intermediate layer features and initial segmentation results. Then, the class centers and class weight maps are calculated using the intermediate layer features. The predicted probability map of the initial segmentation results is weighted using the class weight map. In the weighted prediction probability map, the category with the highest probability is selected as the pseudo-label of the corrected target domain point cloud data. In the second training step, the target domain point cloud data and the source domain simulated sampled point cloud data are mixed and augmented with scene, and then input into the student segmentation network model for prediction. The prediction results of the student segmentation network model are obtained. The pseudo-label of the target domain point cloud data and the real label of the source domain simulated sampled point cloud data are used for supervision. The parameters of the student segmentation network model are updated using gradient descent with cross-entropy as the penalty function. Then, the parameters of the teacher segmentation network model are updated according to the exponential moving average of the parameters of the student segmentation network model.

[0075] After the current batch of training data is completed, the training steps are repeated with the next batch of training data until the training is finished. The resulting student segmentation network model is used as the final trained point cloud semantic segmentation network model.

[0076] In the first step of training, the learner rate of the optimizer for the point cloud semantic segmentation network model is set to 1×10. -3 ;

[0077] In the second training step, the learner learning rate for both the teacher segmentation network model and the student segmentation network model was set to 5×10. -4 The parameter α for the exponential moving average is 0.999, the threshold τ for generating pseudo-labels for point cloud data in the target domain is 0.3, and the number of class center vectors is chosen to be K=2.

[0078] In the second step of training described above, during training, foreground targets (such as pedestrians, bicycles, etc.) in the source domain simulated sampling point cloud data are included, and the label c belongs to set C. O The point cloud O = {(x)} i y i , z i )|l i.c =1, c∈C O The point cloud data is mixed with the point cloud data scene of the target domain (the background point cloud after removing foreground objects) to synthesize a new point cloud data P = O∪H, which is the scene mixed point cloud data. The data augmentation methods used for the scene mixed point cloud data include any one of random rotation, translation, and scaling to generate training data.

[0079] The steps of the above method are explained in detail below.

[0080] (1) Training based on target domain simulated sampled data alignment:

[0081] The point cloud semantic segmentation network model used in this invention is a U-shaped 3D convolutional neural network, MinkowskiNet18A, based on sparse convolution. This basic segmentation network model can also use other more complex networks such as MinkowskiNet50 or MinkowskiNet101. The specific parameter settings for each convolutional layer in the point cloud semantic segmentation network model used in this embodiment are shown in the table below.

[0082]

[0083]

[0084]

[0085]

[0086] A LiDAR point cloud data to be segmented is first processed by voxelization and then input into a convolutional network. The input size is N×3, where N is the number of non-empty voxels, and the feature of each point is the three-dimensional coordinates (x, y, z).

[0087] LiDAR point cloud data is typically very sparse due to the characteristics of the sensors used for acquisition, and the sampling modes (such as line count and angle) vary significantly between different sensors. To transform existing source domain point cloud data into point cloud data conforming to the target domain sampling mode, this invention achieves this by simulating sampling of the source domain point cloud data according to the target domain sampling mode. First, using a continuous point cloud sequence from the source domain point cloud data and the pose data of the laser scanning device, multiple frames of point cloud data are fused to obtain the dense point cloud data Q = {q...} for the current scene. i |q i =(x i y i , z i ), i = 1, ..., N Q This largely solves the problem of excessively sparse point clouds in a single frame. A comparison of point clouds before and after fusion can be seen in [link / reference]. Figure 2 After obtaining the dense point cloud data, a location point c = (x) is selected within the scene of the dense point cloud data. o y o , z o The next step is to place a virtual LiDAR, and then re-simulate the scanning process of the virtual LiDAR with this location as the origin in the scene to obtain a new target domain single-frame point cloud as the source domain simulated sampling point cloud data. First, the point cloud coordinates need to be translated: The specific steps of virtual lidar scanning are as follows: convert the Cartesian coordinates in the dense point cloud Q to spherical coordinates to obtain a new point set Q′={q′ i |q′ i =(r i θ i , φ i ), θ i ∈[-θ down θ up ], φ i ∈[-π,π], i=1,...,N Q The spherical coordinates of each point are calculated as follows:

[0088]

[0089] To obtain a single-frame target domain point cloud P′={p′ that conforms to the target domain sampling pattern, i |p′ i =(r i θ i , φ i), i = 1, ..., N P}, where (θ, φ) in the spherical coordinates of each point represents the target domain sampling mode of the virtual lidar;

[0090] Therefore, this invention simulates the angular resolution and line count of a target domain lidar by setting a virtual lidar, and generates the corresponding sampling mode {(θ i , φ i | i = 1, ..., N P}, where Np is the total number of sampling points; then, bilinear interpolation is performed on the (θ, φ) space of the dense point cloud data Q to obtain the result for (θ) i , φ i The distance between points along the sampling direction {r} i |i = 1, ..., N P}; Finally, {(r i θ i , φ i | i = 1, ..., N P} Transform to Cartesian coordinates to obtain single-frame target domain point cloud data P = {p i |p i =(x i y i , z i ), i = 1, ..., N P}, and the label L = {l} corresponding to the obtained single-frame target domain point cloud data. i |l i ∈{0,1} 1×C i = 1, ..., N P Point clouds obtained from dense point cloud data using nearest neighbor interpolation can be obtained by labeling the data. Point clouds obtained from different sampling modes of the same scene are as follows: Figure 3 As shown.

[0091] When it is necessary to transfer a point cloud semantic segmentation model to a specific target domain point cloud data segmentation, the target domain simulated sampling data alignment method described above can be used to simulate sampling of the labeled source domain point cloud data according to the sampling pattern of the target domain point cloud data, obtaining new labeled data, and then retraining the point cloud semantic segmentation network model. When training the point cloud semantic segmentation network model, cross-entropy is used as the loss function for network training, with the specific formula as follows:

[0092]

[0093] Where C represents the number of categories, t i.c Let ∈{0,1} be the label of point i with respect to category c. This is the prediction result. The above method can reduce the impact of differences in data distribution and enable effective migration under different sensor settings.

[0094] (2) Self-training

[0095] The self-training method employed in this invention includes a trainable student segmentation network model g. S A teacher segmentation network model g with momentum update T Similarly, the student segmentation network model g S Teacher segmentation network model g T The structure is consistent with the point cloud semantic segmentation network model trained based on simulated scanning data alignment, and both can adopt existing point cloud segmentation network structures such as MinkowskiNet50 and MinkowskiNet101. The two models have the same structure, which includes two parts: feature extraction network f and segmentation network h, and can be represented as g(·)=h[f(·)].

[0096] Student segmentation network model g S Teacher segmentation network model g T The parameters are initialized from the parameters of the point cloud semantic segmentation network model obtained in the previous training step. During self-training, the teacher segmentation network model g is first... T Perform semantic segmentation prediction on the target domain point cloud data, that is, calculate the i-th point p. i The probability g of belonging to each category c T (p i [c], select the category with the highest probability as the pseudo-label for semantic segmentation of that point, then the one-hot vector of the corresponding label is... for:

[0097]

[0098] Where τ is the probability threshold for generating pseudo-labels.

[0099] Student segmentation network model g S The source domain simulated sampling point cloud data obtained using the labeled simulated target domain sampling pattern and the target domain point cloud data with pseudo-labels generated by the teacher segmentation network model are used for supervised training to update the parameters of the student segmentation network model. S Using cross-entropy as the penalty function again, the total penalty function is now:

[0100]

[0101] Among them, l A This represents the true label of the simulated point cloud data in the source domain obtained by simulating sampling using source domain point cloud data according to the target domain sampling mode; Pseudo-labels representing point cloud data in the target domain. and This represents the semantic segmentation results of the student segmentation network model on the source domain simulated sampled point cloud data and the target domain point cloud data.

[0102] The parameters of the teacher segmentation network model at the k-th iteration The parameters Θ of the student segmentation network model S The exponential moving average is:

[0103]

[0104] In each iteration, pseudo-labels are generated using the teacher segmentation network model, followed by training the student segmentation network model and updating the teacher segmentation network model. This process is repeated until training stops.

[0105] (3) Scenario-mixed training

[0106] When adopting a self-training framework, the learning performance of the model is significantly affected by the quality of pseudo-labels. To address the issue of low pseudo-label accuracy for small objects in the target domain point cloud data due to initial teacher segmentation network model bias, this embodiment of the invention employs a scene-blended training strategy. During training, foreground targets (such as pedestrians, bicycles, etc.) in the source domain simulated sampling point cloud data are used, with labels belonging to set C. O The point cloud O = {(x)} i y i , z i )|l i.c =1, c∈C O The point cloud data is mixed with the target domain point cloud data scene's point cloud H (background point cloud after removing foreground objects) to synthesize new point cloud data P = O∪H. After data augmentation, it is used for training. The actual application results are as follows: Figure 6 As shown, since the point cloud portion from the simulated point cloud data in the source domain has accurate labels, the mixed scene can contain more correctly labeled targets, which is beneficial to improving the segmentation accuracy of the point cloud semantic segmentation model on such objects. Considering that most real point cloud datasets contain straight road scenes, this synthesis method does not destroy contextual information.

[0107] (4) Category-aware pseudo-label correction

[0108] In practical applications, the pseudo-labels used during self-training contain a lot of noise, which causes the LiDAR point cloud semantic segmentation model to learn incorrect mapping relationships in the target domain, limiting further improvement in model accuracy. Therefore, this invention adopts a category-aware pseudo-label correction method to improve this problem. Category centers are the feature distribution centers of each category of data. Category centers are less sensitive to outliers and less affected by them, correctly representing the common features of a class of targets. The distance of each point to different category centers indicates the probability of it belonging to different categories. This information can be used to correct the incorrect labels of outliers. The specific pseudo-label correction method is as follows:

[0109] First, a target domain point cloud dataset P is input into the teacher segmentation network model to obtain intermediate layer features f. T (P) and initial pseudo-label (i.e., the initial segmentation result), for the point features of each class Use the K-means algorithm to calculate the K center vectors {e c，k |k=1,...,K}; In particular, when K=1, the method for calculating the category center is as follows:

[0110]

[0111] in, It is an indicator function that indicates whether the i-th point is predicted to be of the c-th class. After obtaining the class center vector, the minimum distance d between the features of each point in the point cloud and the class center vectors of different classes can be calculated. i，c As a measure of category affiliation, the corresponding category weight map W is further calculated, where w is the weight value of the i-th point relative to the c-th category. i，c The larger the value, the higher the probability that point i belongs to category c.

[0112]

[0113]

[0114] The class weight map W is used as a correction factor, along with the probability map g of the initial segmentation result. T (P) are multiplied to obtain a weighted probability map, and then the category with the highest probability is selected as the corrected pseudo-label. The process of this correction method is as follows: Figure 5 As shown, the teacher model refers to the teacher segmentation network model, the student model refers to the student segmentation network model, the target data refers to the target domain point cloud data, and the source data refers to the source domain simulated sampled point cloud data. These corrected pseudo-labels can be further used for supervised learning of the student segmentation network model. The method for generating the corrected pseudo-labels is as follows:

[0115]

[0116] This invention proposes a self-training framework to guide a semantic segmentation network in learning contextual information of a target scene. This enables effective transfer of the model from source domain point cloud data to target domain point cloud data, solving the domain shift problem caused by differences in scene content across different datasets, which is difficult to address through data alignment training strategies. Examples include domain shifts caused by variations in environment and street views across different geographical regions. Figure 7 The input target domain point cloud data, ground truth labels, and semantic segmentation prediction result diagram are shown for the semantic segmentation method provided in this embodiment of the invention.

[0117] In summary, the method of this invention, trained in the above manner, can achieve effective transfer of point cloud semantic segmentation network models between data in different application scenarios at low cost. This method has been validated on publicly available LiDAR point cloud semantic segmentation datasets. The cross-intersection over union (CUI) ratios of the point cloud semantic segmentation network models using the unsupervised domain adaptation method of this invention when transferred from the nuScenes dataset to the SemanticKITTI dataset and from the SemanticKITTI dataset to the nuScenes dataset reached 39.6% and 41.7%, respectively.

[0118] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A domain-adaptive semantic segmentation method for lidar point clouds, characterized in that, include: Step 1: Initially train the LiDAR point cloud semantic segmentation network model by aligning the target domain simulated sampling data. Based on the initially trained LiDAR point cloud semantic segmentation network model, further train the LiDAR point cloud semantic segmentation network model using a combination of scene blending and pseudo-label correction, resulting in a well-trained LiDAR point cloud semantic segmentation network model; including: Step 11: By fusing the sequential point clouds of the source domain point cloud data and simulating sampling of the target domain, source domain simulated sampling point cloud data that conforms to the target domain sampling pattern is obtained. Step 12: Train the lidar point cloud semantic segmentation network model using the source domain simulated sampling point cloud data that conforms to the target domain sampling pattern, and obtain the preliminary trained lidar point cloud semantic segmentation network model. In step 11, source domain simulated sampling point cloud data conforming to the target domain sampling pattern is obtained by fusing sequential point clouds of source domain point cloud data and simulating sampling of the target domain in the following manner: Step 111: Using the continuous point cloud sequence in the source domain point cloud data and the pose data of the laser scanning device, the multi-frame point cloud data in the source domain point cloud data is fused to obtain the dense point cloud data of the current scene. Step 112: In the current scene where dense point cloud data is obtained, select a point. Place a virtual lidar and use that point as the origin of the coordinate system. In the current scene, use the virtual lidar to simulate the target domain lidar and scan and resample according to the target domain sampling mode to obtain a new single frame of target domain point cloud data as the source domain simulated sampling point cloud data. In step 112, new single-frame target domain point cloud data is obtained by resampling using a virtual lidar to simulate a target domain lidar in the target domain sampling mode, including: Step 1121: Calculate the dense point cloud data using the following formula. The Cartesian coordinates of each point in Q are converted to spherical coordinates. The 3D coordinates of each 3D point in Q in the Cartesian coordinate system with the virtual LiDAR as the origin are: The entire dense point cloud Contains N Q A three-dimensional point, namely Each three-dimensional point Transform into spherical coordinates using the following calculation formula. , recorded as , ； Where, r i The radius is ; The angle with the z-axis, The range refers to the scanning line angle range of the laser scanning equipment. ; The horizontal rotation angle, The range is Then we obtain the spherical coordinate representation Q' corresponding to Q. ; Step 1122: Generate the corresponding target domain sampling pattern based on the angular resolution and line count of the virtual lidar, and obtain Np new scanning rays, the direction of each ray being determined by the angle. The total set of scan rays is defined as follows: Based on these scan line directions in dense point cloud data of Bilinear interpolation is performed in space to calculate the distance r from the origin to each spatial point on each ray. i That is, to obtain the sphere radius of the point cloud in the corresponding direction. ; Step 1123: Sample the point set sphere coordinates after the virtual lidar simulates the target domain lidar scanning method. Convert to Cartesian coordinates to obtain new single-frame target domain point cloud data. , Contains N p Each point The three-dimensional coordinates are For each point in this single frame of point cloud data In dense point cloud data Find the nearest point in the array and use the label of that nearest point as... The semantic tags are denoted as , It is a C-dimensional one-hot vector, where C is the total number of categories in the semantic segmentation. If only one of the C elements in the data has a value of 1 in the dimension corresponding to the true category of the point, and all other elements have a value of 0, then the single-frame target domain point cloud data is obtained. All corresponding tag data ; Step 2: Use the trained lidar point cloud semantic segmentation network model to perform semantic segmentation on the target domain point cloud data to obtain the semantic segmentation results of the target domain point cloud data.

2. The domain-adaptive lidar point cloud semantic segmentation method according to claim 1, characterized in that, In step 1, the semantic segmentation network model for lidar point cloud adopts any one of the following: a U-shaped three-dimensional convolutional neural network based on sparse convolution, MinkowskiNet50, or MinkowskiNet101. The semantic segmentation network model for LiDAR point clouds uses cross-entropy as the penalty function, and its formula is as follows: ； Where N is the total number of points; The number of semantic categories in the point cloud; Is the i-th point a class c? The identifier is either 0 or 1; It is to predict the i-th point as the i-th point. The probability of each category; It is the set of category identifiers for all N points; It is the set of predicted class probabilities for all N points.

3. The domain-adaptive lidar point cloud semantic segmentation method according to claim 1, characterized in that, In step 1, the LiDAR point cloud semantic segmentation network model is trained using scene blending and pseudo-label correction methods in the following manner: The semantic segmentation network model of LiDAR point cloud obtained by training through the alignment method of sampled data in the target domain is used as the teacher segmentation network model, and a student segmentation network model is initialized with the model parameters of the teacher segmentation network model. The student segmentation network model has the same structure as the teacher segmentation network model. Using a batch of training data consisting of one frame of simulated source domain point cloud data and one frame of target domain point cloud data, the teacher segmentation network model and the student segmentation network model are self-trained in the following manner: In the first training step, the target domain point cloud data is input into the teacher segmentation network model for prediction to obtain intermediate layer features and initial segmentation results. Based on the obtained intermediate layer features and initial segmentation results, the class center vector and class weight map are calculated. The predicted probability map of the initial segmentation results is weighted by the class weight map to obtain a weighted predicted probability map. The class with the highest probability in the predicted probability map is taken as the pseudo label of the corrected target domain point cloud data. In the second training step, the target domain point cloud data and the source domain simulated sampled point cloud data are mixed and augmented with scene effects before being input into the student segmentation network model to obtain the prediction results of the student segmentation network model. The pseudo-labels of the target domain point cloud data and the real labels of the source domain simulated sampled point cloud data are used for supervision, with cross-entropy as the penalty function. The gradient descent method is used to update the parameters of the student segmentation network model. Then, the parameters of the teacher segmentation network model are updated according to the exponential moving average of the parameters of the student segmentation network model. Replace the next batch of training data and repeat the above training steps to update the teacher segmentation network model and train the student segmentation network model until training is complete. The final student segmentation network model is used as the trained LiDAR point cloud semantic segmentation network model.

4. The semantic segmentation method for lidar point cloud data according to claim 3, characterized in that, During the training of the first training step, the learner learning rates of both the teacher segmentation network model and the student segmentation network model are set to a certain value; the number of class center vectors is selected as follows: ; Probability threshold for generating pseudo-labels for target domain point cloud data ; During the training process of the second training step, the optimizer learning rate of both the teacher segmentation network model and the student segmentation network model is set to... The parameters of the teacher's segmentation network model are updated based on the exponential moving average of the student segmentation network model parameters. .

5. The semantic segmentation method for lidar point cloud data according to claim 3, characterized in that, During the training process of the second training step, the target domain point cloud data and the source domain simulated sampled point cloud data are mixed and augmented in the following manner: Scene blending is: blending all points of the foreground point cloud from the source domain simulated sampling point cloud data. Point cloud of the scene in the target domain point cloud data Mix and synthesize new point cloud data As mixed point cloud data for a scene; Among them, the foreground point cloud refers to the point cloud with small objects such as pedestrians, bicycles, and motorcycles in the foreground, that is, the label c of the point belongs to the set. Point cloud of {pedestrians, bicycles, motorcycles}; point cloud To remove the background point cloud from the foreground point cloud; Data augmentation refers to applying any one of the following data augmentation methods—random rotation, translation, or scaling—to the scene blending point cloud data. Data augmentation is performed, and the augmented data is used as training data.

6. A processing apparatus, characterized in that, include: At least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor is able to implement the method of any one of claims 1-5.

7. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement the method described in any one of claims 1-5.

Images