Laser radar point cloud semantic segmentation method and system based on domain generalization

Abstract

The invention provides a domain generalization laser radar point cloud semantic segmentation method and system, and belongs to the field of laser radar semantic segmentation, and the method comprises the steps: obtaining point cloud data of a source domain, and carrying out the preprocessing; resampling the preprocessed source domain point cloud data to generate a resampling point cloud; generating a mixed point cloud through a cross-scene mixed enhancement strategy; inputting the initial point cloud data, the re-sampling point cloud data and the mixed point cloud data into a semantic segmentation network to extract point cloud features; unified alignment of point cloud features in a semantic space is realized by using a text encoder of a contrast language-image pre-training model; and carrying out model training by adopting a contrast loss function, selecting a target domain point cloud, inputting the target domain point cloud into the trained model, and outputting a point cloud semantic segmentation result. According to the method, through the synergistic effect of multi-density resampling, cross-scene mixing and unified semantic space alignment, the domain offset problem is effectively relieved, and the semantic segmentation precision and generalization ability of the model on an unknown target domain are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of laser radar semantic segmentation, and particularly relates to a domain generalization laser radar point cloud semantic segmentation method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the application and do not necessarily constitute prior art.

[0003] LiDAR-based three-dimensional semantic segmentation is a core technology for realizing fine perception of an environment in intelligent systems such as autonomous driving and robot navigation. This technology aims to assign a semantic class label (such as a vehicle, a pedestrian, a building, etc.) to each LiDAR point, thereby providing a decision-making system with rich structured information about a scene.

[0004] In recent years, with the development of deep learning technology, especially in the application of three-dimensional point cloud processing, LiDAR semantic segmentation models have achieved significant performance improvement on multiple closed benchmark datasets. However, these models are heavily dependent on the independent and identical distribution of source domains and target domains. However, in practical applications, when a model trained on a certain dataset or sensor configuration is directly deployed in another environment of a different domain, its performance usually drops sharply, which is collectively referred to as the domain shift problem.

[0005] To address the above challenges, domain generalization (DG) is proposed as a cutting-edge learning paradigm. The core goal is to learn a robust model that is independent of a specific domain during the training phase using data from one or more source domains, so that the model can maintain good performance on unseen target domains. Although domain generalization can improve the adaptability of the model on unseen target domains, cross-domain differences often lead to a decrease in model performance. Currently, there have been some research works to improve the performance decline problem in LiDAR semantic segmentation, such as introducing sparse invariant feature consistency constraints and semantic consistency constraints, and constructing models based on sparse consistency or semantic constraints. However, the above models have limited robustness to density changes and their performance significantly decreases in extremely sparse or extremely dense scenes. Moreover, the feature constraints and alignment process are easily affected by source domain data, leading to bias towards the source domain feature space. Density-aware methods, although to some extent, consider the point cloud distribution difference, are limited by the narrow density range in the training samples, making it difficult to cover more complex distribution patterns in actual scenes. At the same time, existing methods fail to fully consider the complex coupling relationship between density changes and scene structures under different LiDAR configurations, and are unable to learn semantic features extracted from different domain data into a domain-independent, semantically rich feature space, thereby resulting in insufficient model generalization ability. SUMMARY To overcome the shortcomings of the prior art, this invention provides a domain-generalized lidar point cloud semantic segmentation method and system. Based on multi-density point cloud resampling and cross-scene hybrid enhancement strategies, it achieves robust generalization performance across sensors and scenes through a unified semantic alignment mechanism.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a domain-generalized semantic segmentation method for lidar point clouds; A domain-generalized semantic segmentation method for lidar point clouds includes: Acquire point cloud data from the source domain and perform preprocessing; Resampling is used to resample the preprocessed source domain point cloud data to generate resampled point clouds with different density distributions. By employing a cross-scene hybrid enhancement strategy, two point clouds from different scenes that have undergone random resampling are divided into multiple non-overlapping regions and combined in an interleaved manner to generate a hybrid point cloud. The initial point cloud data, resampled point cloud data, and mixed point cloud data of the source domain are respectively input into the semantic segmentation network to extract point cloud features. The text encoder in the contrastive language-image pre-trained model encodes the category names of the point cloud semantic segmentation task into high-dimensional text embedding vectors. By calculating the semantic association between the point cloud features of each branch and their corresponding ground truth category text embedding vectors, consistent semantic supervision constraints are applied to the point cloud features of each branch in a unified semantic embedding space, thereby achieving unified alignment of point cloud features in the semantic space. The model is trained using a contrastive loss function. The target domain point cloud is selected and input into the trained model, and the point cloud semantic segmentation result is output.

[0007] As a further technical solution, the preprocessing process includes: The original LiDAR point cloud data in the source domain is acquired, and the attitude data of the original LiDAR is estimated by using the random sampling consistency plane fitting method. The original LiDAR parameter set is constructed by combining the inherent configuration parameters of the original LiDAR. For branches that require resampling, the point cloud is transformed from the sensor coordinate system to a unified global coordinate system using the original LiDAR attitude data. The point cloud is then stitched together in the global coordinate system. After stitching the original point cloud frame with several adjacent point cloud frames, the point cloud is transformed back to the original point cloud sensor coordinate system to obtain a high-density fused point cloud.

[0008] As a further technical solution, the preprocessed source domain point cloud data is resampled based on point cloud resampling to generate resampled point clouds with different density distributions, including: By randomly generating simulated LiDAR parameter sets to simulate different LiDAR configurations, according to the original LiDAR parameter set and the simulated LiDAR parameter set, the rotation matrix and the translation vector for realizing coordinate conversion are calculated, and the three-dimensional coordinates of the high-density fused point cloud are uniformly transformed to the simulated LiDAR coordinate system; In view of the index conflict existing in the coordinate system in the process of resampling the point cloud using the simulated LiDAR parameters, the z-buffer strategy is adopted to retain the point closest to the origin of the distance simulated LiDAR, and finally a resampled point cloud with different density distribution is generated.

[0009] As a further technical solution, the two frames of randomly resampled point clouds from different scenes are divided into multiple non-overlapping regions by a cross-scene mixed enhancement strategy, and a mixed point cloud is generated in an interleaved manner, including: From the obtained source domain point cloud data, two frames of randomly resampled point clouds from different scenes carrying corresponding semantic label information are selected; Based on the vertical pitch angle feature of the laser radar point cloud, the two frames of point clouds are respectively divided into multiple non-overlapping regions according to the vertical angle direction; An interleaved splicing manner is adopted to combine the divided regions to generate two different cross-scene mixed point clouds; In the process of generating the mixed point cloud, the semantic labels corresponding to each splicing region are synchronously assigned to the corresponding points in the mixed point cloud, so that each point in the mixed point cloud carries label information consistent with the semantic of the original region to which it belongs, and the generation of the cross-scene mixed point cloud is completed.

[0010] As a further technical solution, the obtained source domain initial point cloud data, resampled point cloud data, and mixed point cloud data are respectively input into a semantic segmentation network to extract point cloud features, including: A semantic segmentation network including an original point cloud branch, a resampled point cloud branch, and a mixed point cloud branch is built; wherein the mixed point cloud branch includes two sub-branches corresponding to the feature extraction of the two cross-scene mixed point clouds; The obtained source domain initial point cloud data, resampled point cloud data, and two mixed point cloud data are respectively input into the corresponding original point cloud branch, resampled point cloud branch, and two mixed point cloud sub-branches, and the point cloud data is spatially feature encoded through a sparse convolution layer, and the semantic features of the point cloud are extracted through multiple sparse convolution blocks in turn to generate an original point cloud feature map, a resampled point cloud feature map, and a first mixed point cloud feature map and a second mixed point cloud feature map.

[0011] As a further technical solution, the class names of the point cloud semantic segmentation task are encoded into high-dimensional text embedding vectors by using the text encoder of the contrastive language-image pre-training model, the semantic association between each branch point cloud feature and its corresponding true value class text embedding vector is calculated, and consistent semantic supervision constraints are applied to each branch point cloud feature in a unified semantic embedding space, so that the point cloud features are uniformly aligned in the semantic space, including: According to the preset categories of the point cloud semantic segmentation task, a set of class names is sorted to ensure that the class names are completely matched with the semantic categories labeled by the source domain point cloud; Each semantic class name after sorting is input into the text encoder of the contrastive language-image pre-training model, and the class name is converted into a fixed-dimensional high-dimensional text embedding vector through semantic encoding of the class name; The cosine similarity of the original point cloud feature map, the resampled point cloud feature map, the first mixed point cloud feature map and the second mixed point cloud feature map and the high-dimensional text embedding vector of the corresponding category is calculated to obtain the similarity value of each point cloud feature vector and the semantic category text embedding vector; During training, the feature map and the high-dimensional text embedding vector of the corresponding category are supervised by the point cloud semantic true value label using InfoNCE Loss, and the multi-branch point cloud feature is embedded into a unified category text feature space; during testing, the highest similarity value of the vector can be obtained. The predicted label.

[0012] As a further technical solution, the contrastive loss function adopts a weighted combination form of the multi-branch information noise contrast estimation contrastive loss function, as shown in the following formula:

[0013] Wherein, is the information noise contrast estimation loss function; is the original point cloud prediction feature; is the original point cloud branch true value text embedding; , is a hyperparameter; is the resampled point cloud prediction feature; is the resampled point cloud branch true value text embedding; is the mixed 1 point cloud prediction feature; mixed 2 point cloud prediction feature; is the mixed point cloud branch true value text embedding.

[0014] The second aspect of the present application provides a domain generalization laser radar point cloud semantic segmentation system.

[0015] A domain generalization laser radar point cloud semantic segmentation system, comprising: The point cloud data acquisition module is configured to acquire point cloud data of a source domain and perform preprocessing; The point cloud resampling module is configured to resample the preprocessed point cloud data of the source domain based on point cloud resampling to generate resampled point clouds with different density distributions; The mixed point cloud generation module is configured to divide two frames of randomly resampled point clouds from different scenes into multiple non-overlapping areas through a cross-scene mixed enhancement strategy, and combine the mixed point clouds in an interleaved manner. The multi-branch CLIP semantic feature alignment module is configured to input the acquired initial point cloud data of the source domain, the resampled point cloud data, and the mixed point cloud data into a multi-branch semantic segmentation network respectively to extract point cloud features, encode the class names of the point cloud semantic segmentation task into high-dimensional text embedding vectors using a text encoder of a contrastive language-image pre-training model, and apply consistent semantic supervision constraints to each branch point cloud feature in a unified semantic embedding space by calculating the semantic association between each branch point cloud feature and its corresponding true value class text embedding vector, thereby realizing the unified alignment of point cloud features in the semantic space. The point cloud semantic segmentation result output module is configured to perform model training using a contrastive loss function, input the target domain point cloud into the trained model, and output the point cloud semantic segmentation result.

[0016] The third aspect of the present application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the domain generalization laser radar point cloud semantic segmentation method according to the first aspect of the present application.

[0017] The fourth aspect of the present application provides an electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, wherein the processor implements the steps of the domain generalization laser radar point cloud semantic segmentation method according to the first aspect of the present application when executing the program.

[0018] The above one or more technical solutions have the following beneficial effects: The present application generates synthetic point cloud samples with different density distributions by fusing the source domain point cloud with adjacent frames based on a density-based resampling strategy and performing translation, rotation, and sampling according to randomly generated LiDAR configuration parameters. At the same time, cross-scene data mixing is performed through LaserMix, and the model learns domain-invariant features by making consistent predictions before and after mixing. This strategy significantly expands the distribution coverage of the source domain samples, allowing the model to access more diverse density distributions and scene features during the training process.

[0019] In domain generalization semantic segmentation, the inconsistency of source domain and target domain features often leads to the decline of model classification accuracy. The application uses the category embedding space of CLIP to uniformly project the multi-branch features from the original point cloud, the resampled point cloud and the LaserMix mixed point cloud, and simultaneously uses the InfoNCE loss constraint to realize effective differentiation and uniform mapping of the features at the semantic level. The CLIP aligned features can enhance the semantic separation ability of the feature space and improve the generalization stability of the model between different domains.

[0020] To improve the robustness and generalization ability of the model, the application adopts a multi-branch training strategy, including an original point cloud branch, a resampled point cloud branch and a LaserMix mixed point cloud branch, and fully utilizes the complementary information of different types of point clouds through joint training. The robustness of the trained model under different scene and density conditions is effectively improved. This strategy alleviates the technical problem that single data enhancement or single feature mapping cannot fully utilize the source domain data information.

[0021] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be known through the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application, illustrate the preferred embodiments of the application and assist in the explanation of the application. The drawings are not intended to limit the application in any way.

[0023] Figure 1 The method flowchart of the first embodiment.

[0024] Figure 2 The result schematic diagram for comparing the data enhancement method and the domain generalization method on the SemanticKITTI (K) and nuScenes (N) data sets using the average intersection over union in the first embodiment.

[0025] Figure 3 The result schematic diagram for comparing the domain generalization performance using the SemanticKITTI (K), nuScenes (N) and Waymo (W) data sets in the first embodiment.

[0026] Figure 4 The system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0028] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present application.

[0029] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0030] The general idea of the present application is as follows: The present application uses the LaserMix cross-scene mixing strategy to mix the resampled point clouds under different LiDAR configurations, thereby generating point cloud data with mixed density characteristics, and maintaining the consistency of the prediction results before and after mixing. By mapping the features of the original data, resampled data and mixed data to a unified and well-distinguished CLIP feature space, stable generalization in cross-domain scenes is achieved. Embodiment one The present embodiment discloses a domain generalization LiDAR point cloud semantic segmentation method, based on multi-density point cloud resampling and cross-scene mixing enhancement strategy, by constructing a multi-density, multi-scene consistent semantic feature space, the model realizes the robust generalization performance in different LiDAR configurations and complex scenes. Relieve the point cloud density inconsistency, semantic distribution drift and feature alignment difficulty caused by LiDAR sensor difference and scene distribution change and other problems.

[0031] Specifically, as shown in Figure 1 A domain generalization LiDAR point cloud semantic segmentation method, comprising: Step S1, acquiring point cloud data of a source domain and pre-processing.

[0032] The point cloud data of the source domain is obtained from a public LiDAR dataset, a LiDAR sensor installed on the top or around the vehicle. For example, the public LiDAR dataset can be obtained through the SemanticKITTI, nuScenes and other platforms. Then, the acquired source domain point cloud data is pre-processed, first set the parameters of the LiDAR sensor as , wherein is the number of laser beams, is the number of horizontal sampling, is the height of the LiDAR from the ground, is the LiDAR attitude direction vector, and are the upper and lower limit angles of the vertical field of view, respectively. By setting these parameters, an analog LiDAR system of any configuration can be constructed to realize point cloud resampling.

[0033] Further, a random sampling consistency plane fitting method is used to estimate the attitude data of the original LiDAR, the attitude data includes the attitude direction vector Height from ground installation , in combination with the inherent configuration parameters of the original LiDAR 、 、 、 Constructing the original LiDAR parameter set .

[0034] With the pose data of the original LiDAR, align the original point cloud frame and its adjacent several frames of point cloud to a unified world coordinate system, then splice and convert it back to the original point cloud sensor coordinate system, so as to obtain a high-density fused point cloud. At the same time, in order to avoid the interference of moving objects, exclude the points that obviously move in the adjacent frames during the registration process.

[0035] Step S2, resample the preprocessed source domain point cloud data based on point cloud resampling, to generate resampled point clouds with different density distributions.

[0036] By randomly generating a simulated LiDAR parameter set , simulate the sampling conditions of laser radars with different resolutions and fields of view. According to the original LiDAR parameter set and the simulated LiDAR parameter set , calculate the rotation matrix and translation vector that realize coordinate conversion, and unify the three-dimensional coordinates of the high-density fused point cloud to the coordinate system of the simulated LiDAR.

[0037] In view of the index conflict existing in the simulated LiDAR, the z-buffer strategy is adopted to retain the points closest to the origin of the simulated LiDAR, and finally generate resampled point clouds with different density distributions. For each three-dimensional point , it is converted to in the following way:

[0038]

[0039] wherein, is the vertical pitch angle; is the horizontal azimuth angle.

[0040] According to the number of laser beams and the number of horizontal sampling in the simulated LiDAR parameter set , calculate the vertical pitch angle and horizontal resolution of each laser beam, as follows:

[0041]

[0042] wherein, is the vertical pitch angle; is the horizontal resolution.

[0043] Next, the point is determined. The vertical index and the horizontal index in the coordinate system are:

[0044]

[0045] wherein, is the vertical index; is the horizontal index.

[0046] In the process of generating a simulated LiDAR coordinate system conforming to a random parameter setting, if an index conflict occurs, i.e., multiple points are mapped to the same pixel position , a z-buffer strategy is adopted to retain only the point closest to the origin of the simulated LiDAR, and finally a resampled point cloud conforming to the simulated LiDAR configuration and having different density distributions is generated.

[0047] In this embodiment, for point cloud data, if it is used for a non-original branch of model training (i.e., a training branch that is different from an input original point cloud), a random rotation operation around the z-axis is performed on the mixed point cloud with a probability of 50%, further enhancing the diversity of the mixed point cloud data distribution, and providing data support for subsequent model generalization ability improvement.

[0048] Step S3, through the cross-scene mixing enhancement strategy, the two frames of randomly resampled point clouds from different scenes are divided into multiple non-overlapping regions, and the mixed point cloud is generated in an interleaved manner.

[0049] From the source domain point cloud data set, two frames of randomly resampled point clouds to be mixed from different scenes are selected, denoted as point cloud and point cloud , and both frames of point clouds carry corresponding semantic label information.

[0050] Based on the vertical pitch angle feature of the laser radar point cloud, the point cloud and the point cloud are respectively divided into n non-overlapping regions according to the vertical angle direction, i.e. , . And the semantic labels of the point clouds in each region remain consistent with the semantic labels of the corresponding original point cloud region.

[0051] An interleaved splicing manner is used to combine the divided regions to generate two different cross-scene mixed point clouds: , , and the corresponding semantic labels are mixed synchronously.

[0052] In the process of generating the mixed point cloud and The corresponding semantic labels of each splicing area are synchronously given to the corresponding points in the mixed point cloud, so that each point of the mixed point cloud carries label information consistent with the semantic of the original area to which the point belongs, and the generation of the cross-scene mixed point cloud is completed.

[0053] In step S4, the obtained initial point cloud data of the source domain, the resampled point cloud data, and the mixed point cloud data are respectively input into a semantic segmentation network to extract point cloud features; a text encoder of a contrastive language-image pre-trained model is used to encode the class names of the point cloud semantic segmentation task into high-dimensional text embedding vectors, and by calculating the semantic correlation between each branch point cloud feature and the corresponding true value class text embedding vector, consistent semantic supervision constraints are applied to each branch point cloud feature in a unified semantic embedding space, so that the point cloud features are uniformly aligned in the semantic space.

[0054] A semantic segmentation network including three functional branches is built, which are an original point cloud branch, a resampled point cloud branch, and a mixed point cloud branch; MinkowskiNet is used as the backbone network, and the mixed point cloud branch includes two sub-branches corresponding to the feature extraction of the two cross-scene mixed point clouds.

[0055] The obtained initial point cloud data of the source domain is input into the original point cloud branch, and the spatial feature of the original point cloud is encoded by the sparse convolution layer of MinkowskiNet, and the geometric features and semantic information of the point cloud are extracted by a plurality of sparse convolution blocks in sequence to generate an original point cloud feature map, which provides stable source domain semantic information for the network.

[0056] The multi-density resampled point cloud data is input into the resampled point cloud branch; the same feature extraction process as the original point cloud branch is adopted, and the sparse convolution layer of MinkowskiNet is used to perform layer-by-layer feature conversion on the resampled point cloud to generate a resampled point cloud feature map, which enables the network to have robustness to the difference features of point clouds with different densities in the spatial structure.

[0057] The generated two kinds of mixed point cloud data and are respectively input into the two sub-branches of the mixed point cloud branch to generate a first mixed point cloud feature map and a second mixed point cloud feature map, respectively, which provides point cloud data with mixed scenes and densities, and further improves the robustness of the model to the scene distribution and the density distribution.

[0058] A contrastive language-image pre-training (CLIP) model is introduced, a text encoder module of the CLIP model is called, and parameters of the text encoder module are fixed to avoid parameter drift in the training process; according to preset categories (such as vehicles, pedestrians, drivable road surfaces, vegetation, etc.) of a point cloud semantic segmentation task, a category name set is sorted out to ensure that the category names are completely matched with semantic categories of source domain point cloud labels.

[0059] Each semantic category name sorted out is input into the CLIP text encoder, and the category names are semantically encoded by a word embedding layer and a Transformer encoder layer of the encoder to convert the category names in a natural language form into high-dimensional text embedding vectors of a fixed dimension; the text embedding vectors corresponding to all categories constitute a mapping table.

[0060] The original point cloud feature maps, the resampled point cloud feature maps, the first mixed point cloud feature maps and the second mixed point cloud feature maps output by the branches are respectively supervised by high-dimensional text embedding vectors of corresponding categories in the mapping table according to true value labels by using an InfoNCE loss; thus, semantic features of multi-density point clouds are uniformly mapped to a category text semantic space.

[0061] In step S5, a contrastive loss function is used for model training, target domain point clouds are selected and input into the trained model, and point cloud semantic segmentation results are output. Specifically, in the training phase, the text embedding vectors of the true values of the semantic labels are used to supervise the multi-branch enhanced data of the source domain data; after training, the target domain point clouds are directly input into the trained model to predict the point cloud semantic segmentation results.

[0062] Based on the output features of the multi-branch semantic segmentation network, a multi-branch information noise contrast estimation (InfoNCE) contrastive loss function is constructed, and the loss function expression is:

[0063] wherein, is an information noise contrast estimation loss function; is a predicted feature output by an original point cloud branch; is a true value text embedding of the original point cloud branch; , is a hyperparameter; is a resampled point cloud branch output true value; is a true value text embedding of the resampled point cloud branch; is a mixed 1 point cloud branch output feature; is a mixed 2 point cloud branch output feature; is a mixed point cloud branch true value text embedding.

[0064] Obtain target domain point cloud data that was not used in training, input it into the trained model, and output the point cloud semantic segmentation result.

[0065] Furthermore, to verify the effectiveness of the method of this invention, a performance comparison experiment was conducted with six advanced domain generalization methods on two different datasets. Specifically, the Adam optimization algorithm was used with an initial learning rate of 0.01 and a momentum parameter... , The learning rate decays by 0.98 after each epoch. The results are as follows: Figure 2 and Figure 3 As shown. Combined with Figure 2 and Figure 3 In cross-domain tests from SemanticKITTI to nuScenes (K→N) and from nuScenes to SemanticKITTI (N→K), the mIoU of the proposed method is improved by 5.9% and 10.9% respectively compared to DDFE, significantly outperforming the existing method DDFE and verifying the effectiveness of the proposed method in mitigating density and distribution differences between domains. It performs excellently across multiple datasets and scenarios, verifying the robustness of the method. The proposed method achieves superior performance in experiments on different LiDAR datasets such as SemanticKITTI, nuScenes, and Waymo. Compared with the existing method DGLSS, the proposed method achieves the highest mIoU in both the source and target domains, demonstrating its robustness under different point cloud densities and scenarios. Especially in N→K tasks with nuScenes (sparse point clouds) as the source domain, the performance improvement of the proposed method is even greater, fully demonstrating the advantages of multi-density resampling and cross-scenario hybrid strategies in sparse point clouds. Furthermore, the method of the present invention maintains high performance in the source domain, indicating that the method improves cross-domain performance without reducing source domain performance, demonstrating robustness and versatility under multiple scenarios and densities.

[0066] Example 2 This embodiment discloses a domain-generalized lidar point cloud semantic segmentation system; like Figure 4 As shown, a domain-generalized lidar point cloud semantic segmentation system includes: The point cloud data acquisition module is configured to acquire point cloud data from the source domain and perform preprocessing. The point cloud resampling module is configured to: resample the preprocessed source domain point cloud data based on point cloud resampling to generate resampled point clouds with different density distributions; The mixed point cloud generation module is configured to: divide two frames of randomly resampled point clouds from different scenes into a plurality of non-overlapping regions through a cross-scene mixed enhancement strategy, and combine the mixed point cloud in a staggered manner. The multi-branch CLIP semantic feature alignment module is configured to: input the obtained initial point cloud data, the resampled point cloud data, and the mixed point cloud data of the source domain into a multi-branch semantic segmentation network respectively, and extract point cloud features; encode the class names of the point cloud semantic segmentation task into high-dimensional text embedding vectors by using a text encoder of a contrastive language-image pre-training model, and apply consistent semantic supervision constraints to each branch point cloud feature in a unified semantic embedding space by calculating the semantic correlation between each branch point cloud feature and the corresponding true value class text embedding vector, so as to realize the unified alignment of the point cloud features in the semantic space. The point cloud semantic segmentation result output module is configured to: perform model training by using a contrastive loss function, input the point cloud of the target domain into the trained model, and output the point cloud semantic segmentation result.

[0067] Embodiment three The purpose of this embodiment is to provide a computer-readable storage medium.

[0068] A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the domain generalization laser radar point cloud semantic segmentation method of embodiment 1.

[0069] Embodiment four The purpose of this embodiment is to provide an electronic device.

[0070] An electronic device includes a memory, a processor, and a program stored on the memory and executable on the processor, and the processor executes the program to implement the steps of the domain generalization laser radar point cloud semantic segmentation method of embodiment 1.

[0071] The steps and methods of the above embodiments two, three and four are corresponding to the method of embodiment one, and the specific embodiments can be referred to the related description part of embodiment one. The term "computer-readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium that can store, encode or carry instruction sets for execution by a processor and make the processor execute any method in the present application.

[0072] Those skilled in the art should understand that the modules or steps of the present application described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0073] The specific embodiments of the present application described above in conjunction with the accompanying drawings are not intended to limit the protection scope of the present application, and those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A domain generalization method for semantic segmentation of lidar point clouds, characterized in that, The method comprises the following steps: Obtain point cloud data of a source domain and pre-process the data; Resample the pre-processed point cloud data of the source domain based on point cloud resampling to generate resampled point clouds with different density distributions; Divide two frames of point clouds from different scenes into multiple non-overlapping areas in an interleaved manner to generate mixed point clouds through a cross-scene mixed enhancement strategy; Input the obtained initial point cloud data, resampled point cloud data, and mixed point cloud data of the source domain into a semantic segmentation network respectively to extract point cloud features; Encode the class names of the point cloud semantic segmentation task into high-dimensional text embedding vectors using a text encoder in a contrastive language-image pre-training model, and impose consistent semantic supervision constraints on the branch point cloud features in a unified semantic embedding space by calculating the semantic association between the branch point cloud features and their corresponding true value class text embedding vectors, thereby realizing the unified alignment of the point cloud features in the semantic space; Train the model using a contrastive loss function, select point clouds of a target domain, and input them into the trained model to output the point cloud semantic segmentation results.

2. The domain generalization LiDAR point cloud semantic segmentation method of claim 1, wherein the pre-processing process comprises: Obtain the original LiDAR point cloud data of the source domain, estimate the attitude data of the original LiDAR using the random sampling consistency plane fitting method, and construct the original LiDAR parameter set in combination with the inherent configuration parameters of the original LiDAR; For branches that need to be resampled, convert the point cloud from the sensor coordinate system to the global coordinate system using the attitude data of the original LiDAR, splice the point cloud in the global coordinate system, convert the spliced point cloud of the original point cloud frame and its adjacent frames of point cloud back to the original point cloud sensor coordinate system to obtain high-density fused point cloud.

3. The domain generalization LiDAR point cloud semantic segmentation method of claim 1, wherein the resampling of the pre-processed point cloud data of the source domain based on point cloud resampling to generate resampled point clouds with different density distributions comprises: Simulate different LiDAR configurations by randomly generating simulated LiDAR parameter sets, calculate the rotation matrix and translation vector for coordinate conversion based on the original LiDAR parameter set and the simulated LiDAR parameter set, and unify the three-dimensional coordinates of the high-density fused point cloud to the simulated LiDAR coordinate system; To solve the index conflict in the coordinate system during the resampling of the point cloud using the simulated LiDAR parameters, use the z-buffer strategy to retain the points closest to the origin of the simulated LiDAR, and finally generate resampled point clouds with different density distributions.

4. The domain generalization LiDAR point cloud semantic segmentation method of claim 1, wherein the two frames of randomly resampled point clouds from different scenes are divided into multiple non-overlapping areas in an interleaved manner to generate mixed point clouds through a cross-scene mixed enhancement strategy, comprising: Select two frames of point clouds carrying corresponding semantic label information from different scenes from the obtained source domain point cloud data; Divide the two frames of point clouds into multiple non-overlapping areas according to the vertical angle direction based on the vertical pitch angle feature of the LiDAR point cloud; The divided regions are combined in a staggered splicing manner to generate two different cross-scene mixed point clouds; In the process of generating the mixed point cloud, the semantic labels corresponding to each splicing region are synchronously assigned to the corresponding points in the mixed point cloud, so that each point in the mixed point cloud carries label information consistent with the semantic of the original region to which the point belongs, and the generation of the cross-scene mixed point cloud is completed.

5. The domain generalization laser radar point cloud semantic segmentation method of claim 1, wherein the obtained initial point cloud data, the resampled point cloud data, and the mixed point cloud data of the source domain are respectively input into the semantic segmentation network to extract point cloud features, including: building a semantic segmentation network comprising an original point cloud branch, a resampled point cloud branch, and a mixed point cloud branch; wherein the mixed point cloud branch comprises two sub-branches corresponding to the feature extraction of the two cross-scene mixed point clouds; the obtained initial point cloud data, the resampled point cloud data, and the two mixed point cloud data of the source domain are respectively input into the corresponding original point cloud branch, the resampled point cloud branch, and the two mixed point cloud sub-branches, the point cloud data is spatially encoded by a sparse convolution layer, and the semantic features of the point cloud are sequentially extracted by a plurality of sparse convolution blocks to generate an original point cloud feature map, a resampled point cloud feature map, and a first mixed point cloud feature map and a second mixed point cloud feature map.

6. The domain generalization laser radar point cloud semantic segmentation method of claim 1, wherein the class names of the point cloud semantic segmentation task are encoded into high-dimensional text embedding vectors using a text encoder of a contrastive language-image pre-trained model, and the semantic association between the point cloud features of each branch and the corresponding true value class text embedding vectors is calculated to impose consistent semantic supervision constraints on the point cloud features of each branch in a unified semantic embedding space, thereby realizing the unified alignment of the point cloud features in the semantic space, including: According to the preset classes of the point cloud semantic segmentation task, the class name set is sorted to ensure that the class names are completely matched with the semantic classes annotated by the source domain point cloud; each semantic class name after sorting is input into the text encoder of the contrastive language-image pre-trained model, and the class names are converted into high-dimensional text embedding vectors of a fixed dimension by semantic encoding; the cosine similarity between the original point cloud feature map, the resampled point cloud feature map, and the first mixed point cloud feature map and the second mixed point cloud feature map and the high-dimensional text embedding vectors of the corresponding classes is calculated to obtain the similarity values of each point cloud feature vector and the semantic class text embedding vectors; during training, the point cloud feature vectors of the multi-branch vectors and the corresponding true value class high-dimensional text embedding vectors are supervised by the point cloud semantic true value labels using InfoNCE Loss, thereby prompting the point cloud features of the multi-branch to be embedded in a unified class text feature space; during testing, the predicted label can be obtained according to the highest similarity value of the vector.

7. The domain generalization laser radar point cloud semantic segmentation method of claim 1, wherein the contrastive loss function adopts a weighted combination form of the multi-branch information noise contrast estimation contrastive loss function, as shown in the following formula: including: wherein, Estimating loss function for information noise contrast; Predicting features for original point cloud; Ground truth text embedding for original point cloud branch; , Hyperparameters; Predicting features for resampled point cloud; Ground truth text embedding for resampled point cloud branch; Predicting features for mixed 1 point cloud; Predicting features for mixed 2 point cloud; Ground truth text embedding for mixed point cloud branch.

8. A domain generalized lidar point cloud semantic segmentation system, characterized in that, ​ The point cloud data acquisition module is configured to acquire point cloud data of a source domain and perform preprocessing; The point cloud resampling module is configured to perform resampling on the preprocessed point cloud data of the source domain based on point cloud resampling to generate resampled point clouds with different density distributions; The mixed point cloud generation module is configured to divide two frames of point clouds from different scenes into multiple non-overlapping areas through a cross-scene mixed enhancement strategy, and combine the areas in an interleaved manner to generate mixed point clouds; The multi-branch CLIP semantic feature alignment module is configured to input the acquired initial point cloud data, resampled point cloud data, and mixed point cloud data of the source domain into a multi-branch semantic segmentation network respectively to extract point cloud features; encode the class names of the point cloud semantic segmentation task into high-dimensional text embedding vectors using a text encoder of a contrastive language-image pre-training model, and apply consistent semantic supervision constraints to each branch point cloud feature in a unified semantic embedding space by calculating the semantic correlation between each branch point cloud feature and its corresponding true value class text embedding vector, thereby realizing the unified alignment of point cloud features in the semantic space; The point cloud semantic segmentation result output module is configured to perform model training using a contrastive loss function, input point clouds of a target domain into the trained model, and output point cloud semantic segmentation results.

9. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps of the domain generalization laser radar point cloud semantic segmentation method according to any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the domain generalization laser radar point cloud semantic segmentation method according to any one of claims 1-7.

Images