A multimodal point cloud segmentation method and system based on multi-level adversarial comparison

Abstract

The present invention proposes a multimodal point cloud segmentation method and system based on multi-level adversarial contrast, which relates to the field of image segmentation technology. The method preprocesses the acquired point cloud data set, generates a corresponding enhanced point cloud and 2D image for each point cloud, and obtains a training set; constructs a point cloud segmentation model, and performs comparative learning between the point cloud and the enhanced point cloud, and the point cloud and the 2D image based on the training set until a preset training round is reached, and saves the model with the smallest loss function or the best verification effect; inputs the point cloud to be segmented into the trained point cloud segmentation model for segmentation, and obtains a segmented image of the point cloud; the present invention introduces adversarial contrast and cross-modal multi-level contrast, and through the adversarial contrast between point cloud data and enhanced point cloud data, and the cross-modal multi-level contrast between 2D images and point cloud images, it helps the point cloud learn visual concepts, enrich the semantic information of the point cloud, and improve the expressive ability and distinguishability of features, thereby obtaining a more accurate segmented image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a multimodal point cloud segmentation method and system based on multi-level adversarial contrast. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] 3D vision is key to achieving fully autonomous robots. As robots move, they require three-dimensional data to help them perceive their surroundings, update their own positions, and construct a three-dimensional environment. Because point clouds are closer to the device's original representation, they are increasingly being used in radar-based autonomous driving. Point cloud segmentation is fundamental to understanding the three-dimensional environment and holds significant application value. However, due to the irregular structure of point clouds, manual labeling is very expensive.

[0004] Point clouds are transformation invariant, but when three-dimensional point clouds are applied to image segmentation methods, it is difficult to obtain accurate point cloud features, which affects the accuracy of the segmentation results. In addition, in existing point cloud segmentation methods, the boundary problem in the point cloud has not been solved, making it impossible to obtain high-quality segmentation boundaries. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a multimodal point cloud segmentation method and system based on multi-level adversarial contrast, introducing adversarial contrast and cross-modal multi-level contrast. Through the adversarial contrast between point cloud data and enhanced point cloud data, and cross-modal multi-level contrast between 2D images and point cloud images, it helps the point cloud learn visual concepts, enrich the semantic information of the point cloud, and improve the expressive ability and distinguishability of features, thereby obtaining a more accurate segmented image.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention provides a multimodal point cloud segmentation method based on multi-level adversarial comparison;

[0008] A multimodal point cloud segmentation method based on multi-level adversarial comparison, comprising:

[0009] The acquired point cloud dataset is preprocessed to generate corresponding enhanced point clouds and 2D images for each point cloud to obtain a training set.

[0010] Build a point cloud segmentation model and perform comparative learning between point cloud and enhanced point cloud, and point cloud and 2D image based on the training set until the preset training rounds are reached. Save the model with the smallest loss function or the best verification effect.

[0011] The point cloud to be segmented is input into the trained point cloud segmentation model for segmentation to obtain the segmented image of the point cloud.

[0012] Furthermore, the enhanced point cloud performs feature enhancement on the point cloud by using rotation, translation, and flipping;

[0013] The 2D images are captured from random camera perspectives of the point cloud acquisition device.

[0014] Furthermore, the point cloud segmentation model includes a feature extraction module and a segmentation prediction module.

[0015] Furthermore, the feature extraction module extracts point cloud features through comparative learning between point cloud and enhanced point cloud, and comparative learning between point cloud and 2D image.

[0016] Furthermore, in the comparative learning between the point cloud and the enhanced point cloud, the generator extracts features, the discriminator distinguishes whether the features come from the enhanced point cloud, reduces the distance between the point cloud features and the enhanced point cloud features, and increases the similarity between the point cloud features and the enhanced point cloud features.

[0017] Furthermore, the comparative learning of point clouds and 2D images performs multi-level granularity comparative constraints on the features of different modalities, establishes a connection between 2D images and point clouds, uses contrast loss to narrow the similarity between modalities, learns the visual feature information of 2D images, and finally obtains point cloud features.

[0018] Furthermore, the segmentation prediction module segments the point cloud based on the extracted point cloud features to obtain a segmented image.

[0019] A second aspect of the present invention provides a multimodal point cloud segmentation system based on multi-level adversarial comparison.

[0020] A multimodal point cloud segmentation system based on multi-level adversarial comparison includes a data processing unit, a model building unit and a point cloud segmentation unit:

[0021] The data processing unit is configured to: pre-process the acquired point cloud data set, generate a corresponding enhanced point cloud and 2D image for each point cloud, and obtain a training set;

[0022] The model building unit is configured to: build a point cloud segmentation model, perform comparative learning between the point cloud and the enhanced point cloud, and the point cloud and the 2D image based on the training set, until a preset training round is reached, and save the model with the minimum loss function or the best verification effect;

[0023] The point cloud segmentation unit is configured to: input the point cloud to be segmented into the trained point cloud segmentation model for segmentation, and obtain a segmented image of the point cloud.

[0024] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in a multimodal point cloud segmentation method based on multi-level adversarial contrast as described in the first aspect of the present invention.

[0025] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of a multimodal point cloud segmentation method based on multi-level adversarial contrast as described in the first aspect of the present invention are implemented.

[0026] One or more of the above technical solutions have the following beneficial effects:

[0027] This paper proposes a multimodal point cloud segmentation method based on multi-level adversarial contrast. Compared with previous segmentation methods that only use a single modality and ignore the transfer invariance of point clouds, this method shows excellent performance in image segmentation:

[0028] (1) The present invention introduces an adversarial contrast module and introduces adversarial training into the contrastive learning network, which enhances the feature extraction capability of the model and more effectively obtains point cloud features with transfer invariance.

[0029] (2) The present invention introduces a cross-modal multi-level comparison module, uses 2D images to enrich the features of point clouds, maintains the distribution consistency of different modalities through the comparison network, strengthens the information interaction between modalities, improves the expressiveness and distinguishability of features, and thus obtains a more accurate segmented image.

[0030] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 This is a flow chart of the method of the first embodiment.

[0033] Figure 2 This is a structural diagram of the point cloud segmentation model of the first embodiment.

[0034] Figure 3 This is a system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] Example 1

[0037] This embodiment discloses a multimodal point cloud segmentation method based on multi-level adversarial comparison;

[0038] like Figure 1 As shown in FIG, a multimodal point cloud segmentation method based on multi-level adversarial comparison includes:

[0039] Step S1: Preprocess the acquired point cloud dataset to generate corresponding enhanced point clouds and 2D images for each point cloud to obtain a training set.

[0040] In order to facilitate comparative learning in subsequent models, when constructing the training set, corresponding enhanced point clouds and 2D images are generated for each point cloud to form a corresponding image group of point cloud-enhanced point cloud-2D image.

[0041] The enhanced point cloud is obtained by performing feature enhancement methods such as rotation, translation, and flipping on the point cloud image, while the 2D image is captured from the random camera perspective of the point cloud acquisition device.

[0042] Before inputting into the model, the point cloud, enhanced point cloud and 2D image must be resized to meet the size requirements of the model. The size transformation is performed using the transforms algorithm in Python to transform them into a preset size. In this embodiment, the preset size is 224×224.

[0043] Step S2: Build a point cloud segmentation model. Based on the training set, perform comparative learning between point cloud and enhanced point cloud, and point cloud and 2D image until the preset training rounds are reached. Save the model with the smallest loss function or the best verification effect.

[0044] Figure 2 It is the structural diagram of the point cloud segmentation model, such as Figure 2 As shown, it includes a feature extraction module and a segmentation prediction module. The feature extraction module is used to extract features with better expression ability and discrimination, while the segmentation prediction module segments the point cloud based on the features.

[0045] The feature extraction module includes a 2D feature extraction submodule, a 3D feature extraction submodule, an adversarial contrast submodule, a cross-modal mapping submodule, and a cross-modal multi-level contrast submodule. It takes the corresponding image set of point cloud-enhanced point cloud-2D image as the input of the model and outputs the features of the point cloud. The following is a detailed description of the extraction process based on these five submodules.

[0046] (1) 2D feature extraction submodule: extract 2D features.

[0047] The 2D image is loaded and input into the customized 2D feature extraction network to extract the key feature information of the image and obtain the 2D features corresponding to the point cloud.

[0048] Among them, the customized 2D feature extraction network uses resnet50 as the base network and removes the last fully connected layer in the network to extract features from 2D images.

[0049] (2) 3D feature extraction submodule and adversarial contrast submodule: The cloud feature extraction submodule extracts point cloud features and enhanced point cloud features. Based on the contrastive learning of the adversarial contrast submodule, the similarity between the point cloud and the enhanced point cloud is utilized to reduce the distance between the point cloud features and the enhanced point cloud features, and increase the similarity between the point cloud features and the enhanced point cloud features.

[0050] Specifically, the 3D feature extraction submodule loads the point cloud and enhanced point cloud in pairs and inputs them into a customized 3D contrast learning feature extraction network to extract their respective key information as point cloud features and enhanced point cloud features, and introduces an adversarial contrast module into the network.

[0051] Among them, a customized 3D contrastive learning feature extraction network is used to construct a dual-branch contrastive learning network. The network of each branch has the same network structure, both use pointnet as the base network, share parameters, and the two branches are used to extract point cloud features and enhance point cloud features respectively.

[0052] In the adversarial contrast module, to ensure the similarity between the point cloud and the enhanced point cloud, the 3D contrast learning feature extraction network is regarded as the generator G, and a two-class discriminator D is set after the generator to distinguish whether the feature comes from the enhanced point cloud. During the training phase, the discriminator D attempts to distinguish whether each feature obtained by the generator comes from the point cloud or the enhanced point cloud as its optimization goal, while the generator G has the opposite optimization goal: to deceive the discriminator D, making it difficult for the discriminator D to distinguish the source of the feature, thereby further reducing the distance between the point cloud features and the enhanced point cloud features, and increasing the similarity between the point cloud features and the enhanced point cloud features; putting the two optimization goals together to form a maximum and minimum adversarial game, and conducting adversarial training, respectively constructing two loss functions: adversarial loss L1 and 3D contrast loss L2. The specific formula of adversarial loss L1 is:

[0053]

[0054] Among them, N represents the number of samples in each batch, i represents the i-th sample in the point cloud image; M is from point cloud or enhanced point cloud, D is the discriminator, G is the generator, x represents the sample image, y i,m It represents the sample label, that is, whether the i-th sample comes from the point cloud or the enhanced point cloud.

[0055] Before explaining the 3D contrast loss L2, let's first explain "positive samples": point cloud samples and enhanced point cloud samples with the same image labels in the point cloud and enhanced point cloud are positive samples. Use the 3D contrast loss L2 to calculate the contrast loss of positive samples. The specific formula is:

[0056]

[0057] Where N represents the number of samples in each batch, f c represents the 3D contrast learning feature extraction network, τ represents a temperature factor, i represents the i-th sample in the point cloud image, l represents the enhanced point cloud sample that is opposite to sample i, and k represents the k-th sample in the enhanced point cloud.

[0058] Based on the above L1 and L2, construct the adversarial contrast loss function:

[0059] L adv =L1+λ1L2 (3)

[0060] Among them, λ1 is a parameter.

[0061] (3) Cross-modal mapping module: maps the corresponding point cloud features and 2D features to the same feature space. The specific steps are as follows:

[0062] First, the point cloud features output from step (2) and the enhanced point cloud features are concatenated together to form a new point cloud feature vector.

[0063] Then, the point cloud feature vector and 2D features are input into a fully connected layer respectively and mapped into the same feature space through a nonlinear activation function.

[0064] Through the above operations, the point cloud features and 2D features are mapped to the same feature space, which facilitates the calculation of the contrast loss between different modalities.

[0065] (4) Cross-modal multi-level contrast module: After cross-modal mapping, the features of different modalities are subjected to multi-level granularity contrast constraints to ensure the consistency of distribution within the modality. At the same time, the connection between the 2D image and the point cloud is established. The contrast loss is used to narrow the similarity between the modalities, and the visual feature information of the 2D image is learned. The visual feature information is assigned to the point cloud data to increase the expressive power of the point cloud features, and finally a more accurate point cloud feature is obtained. The cross-modal contrast loss L3 is constructed, specifically:

[0066]

[0067] Where N is the number of samples in each batch, i is the sample number, τ is a temperature factor, g is a cross-modal mapping network, including two fully connected layers, h i,1 represents the i-th point cloud feature, hi,2 Indicates h i,1 h i,1 The corresponding 2D feature, h k,2 represents the kth 2D feature.

[0068] At the same time, fine-grained information is introduced into the point cloud and its corresponding 2D image to construct a multi-label dataset. Images with the same label in different modalities are considered as positive samples, and different weights are added to the positive samples based on the fine-grained information. This architecture can better characterize the intra-class hierarchy and ensure the consistency of the distribution of different modalities. At the same time, the contrast loss of the learning network with multi-level granularity comparison is performed on the features in the two modalities. The specific formula is as follows:

[0069]

[0070] Where τ is a temperature coefficient, F i,j is a control parameter that applies a penalty to subdivisions within the same category. i and h l Belong to one class, and the subdivided patient categories also belong to one class, F i,j is 1; when h i and h l Belong to one class, and when the patient category is subdivided into different classes, is the cosine similarity.

[0071] Based on the above L3 and L4, a cross-modal multi-level contrast loss is constructed:

[0072] L Mcon =L3+λ2L4 (6)

[0073] Among them, λ2 is a parameter.

[0074] The segmentation prediction module uses deconvolution technology to process the extracted point cloud features to obtain the segmentation prediction results. The pixel-by-pixel cross entropy loss is used to calculate the point cloud segmentation loss based on the gap between the predicted segmentation result and the actual segmentation point cloud. The closer the predicted value is to the actual value, the smaller the cross entropy loss between the two. The calculation formula of the point cloud segmentation loss is as follows:

[0075]

[0076] Among them, N is the number of input point clouds, m is the number of pixels in the segmented point cloud, and y ij is the prediction result of the m-th pixel in the i-th point cloud, is the true result of the markup.

[0077] In the training of the point cloud segmentation model, the point cloud segmentation loss L is used seg , adversarial contrast loss LAdv and cross-modal multi-level contrast loss L Mcon Add them together as the total loss function, specifically:

[0078] Loss = L seg +0.3L Adv +0.2L 7con (7)

[0079] Repeat the backpropagation training until the set number of training rounds is reached, and save the model with the smallest loss function or the best verification effect.

[0080] Step S3: Input the point cloud to be segmented into the trained point cloud segmentation model for segmentation to obtain a segmented image of the point cloud.

[0081] Before inputting into the model, it is also necessary to use the size transformation method in step S1 to transform the point cloud to be segmented into a preset size, and then input it into the trained point cloud segmentation model to obtain a segmented image of the point cloud.

[0082] Example 2

[0083] This embodiment discloses a multimodal point cloud segmentation system based on multi-level adversarial comparison;

[0084] like Figure 3 As shown, a multimodal point cloud segmentation system based on multi-level adversarial comparison includes a data processing unit, a model building unit and a point cloud segmentation unit:

[0085] The data processing unit is configured to: pre-process the acquired point cloud data set, generate a corresponding enhanced point cloud and 2D image for each point cloud, and obtain a training set;

[0086] The model building unit is configured to: build a point cloud segmentation model, perform comparative learning between the point cloud and the enhanced point cloud, and the point cloud and the 2D image based on the training set, until a preset training round is reached, and save the model with the minimum loss function or the best verification effect;

[0087] The point cloud segmentation unit is configured to: input the point cloud to be segmented into the trained point cloud segmentation model for segmentation, and obtain a segmented image of the point cloud.

[0088] Example 3

[0089] The purpose of this embodiment is to provide a computer-readable storage medium.

[0090] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a multimodal point cloud segmentation method based on multi-level adversarial comparison as described in the first embodiment of the present disclosure.

[0091] Example 4

[0092] The purpose of this embodiment is to provide an electronic device.

[0093] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of a multimodal point cloud segmentation method based on multi-level adversarial comparison as described in the first embodiment of the present disclosure are implemented.

[0094] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multimodal point cloud segmentation method based on multi-level adversarial contrast, characterized in that: include: Preprocess the acquired point cloud dataset to generate corresponding enhanced point clouds and 2D images for each point cloud to obtain a training set; Build a point cloud segmentation model and perform comparative learning between point cloud and enhanced point cloud, and point cloud and 2D image based on the training set until the preset training rounds are reached. Save the model with the smallest loss function or the best verification effect. The contrastive learning between the point cloud and the enhanced point cloud constructs a dual-branch contrastive learning network, wherein the network of each branch has the same network structure and shares parameters, and the two branches are used to extract point cloud features and enhance point cloud features respectively. The 3D contrastive learning feature extraction network is regarded as a generator, the generator extracts features, and the discriminator distinguishes whether the features come from the enhanced point cloud, reduces the distance between the point cloud features and the enhanced point cloud features, and increases the similarity between the point cloud features and the enhanced point cloud features. Two loss functions are constructed respectively, namely, adversarial loss and 3D contrast loss; The comparative learning between point clouds and 2D images performs multi-level granularity comparative constraints on features of different modalities, establishes a connection between 2D images and point clouds, uses contrast loss to narrow the similarity between modalities, learns visual feature information of 2D images, and ultimately obtains point cloud features; The point cloud to be segmented is input into the trained point cloud segmentation model for segmentation to obtain the segmented image of the point cloud.

2. The multimodal point cloud segmentation method based on multi-level adversarial contrast according to claim 1, characterized in that: The enhanced point cloud performs feature enhancement on the point cloud by using rotation, translation and flipping; The 2D images are captured from random camera perspectives of the point cloud acquisition device.

3. The multimodal point cloud segmentation method based on multi-level adversarial contrast according to claim 1, characterized in that: The point cloud segmentation model includes a feature extraction module and a segmentation prediction module.

4. The multimodal point cloud segmentation method based on multi-level adversarial contrast according to claim 3, characterized in that: The feature extraction module extracts point cloud features through comparative learning between point cloud and enhanced point cloud, and comparative learning between point cloud and 2D image.

5. The multimodal point cloud segmentation method based on multi-level adversarial contrast according to claim 3, characterized in that: The segmentation prediction module segments the point cloud based on the extracted point cloud features to obtain a segmented image.

6. A multimodal point cloud segmentation system based on multi-level adversarial comparison using the method of claim 1, characterized in that: Including data processing unit, model building unit and point cloud segmentation unit: The data processing unit is configured to: pre-process the acquired point cloud data set to generate a corresponding enhanced point cloud and 2D image for each point cloud to obtain a training set; The model building unit is configured to: build a point cloud segmentation model, perform comparative learning between the point cloud and the enhanced point cloud, and the point cloud and the 2D image based on the training set, until a preset training round is reached, and save the model with the smallest loss function or the best verification effect; The point cloud segmentation unit is configured to: input the point cloud to be segmented into a trained point cloud segmentation model for segmentation, and obtain a segmented image of the point cloud.

7. An electronic device, comprising: a memory for non-transitory storage of computer-readable instructions; as well as a processor for executing said computer-readable instructions, When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is executed.

8. A storage medium, characterized in that: Computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1 to 5 are executed.

Images