Three-dimensional point cloud data semantic category classification method and system based on multi-modal data

Abstract

The invention belongs to the technical field of point cloud data processing, and provides a three-dimensional point cloud data semantic category classification method and system based on multi-modal data, and the method comprises the steps: firstly preprocessing a point cloud data set, dividing the point cloud data set into a training set and a test set, constructing a point cloud semantic segmentation model, and carrying out the pre-training and optimization; using the training support set and the query set to carry out fine tuning training on the multi-modal prototype enhancement-based small sample point cloud semantic segmentation model to obtain a trained small sample point cloud semantic segmentation model, and inputting the test support set and the query set into the trained small sample point cloud semantic segmentation model to obtain a multi-modal prototype enhancement-based small sample point cloud semantic segmentation model; and the trained small sample point cloud semantic segmentation model performs semantic segmentation on the input point cloud, and outputs a segmentation result for calculating the segmentation performance. According to the method, efficient utilization of multi-modal information in a small sample point cloud semantic segmentation task is realized through deep mining of collaborative values of a point cloud geometric structure, label text semantic knowledge and 2D depth map boundary details, and a new path is provided for improving segmentation precision and generalization ability.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud data processing, and in particular relates to a method and system for semantic category classification of three-dimensional point cloud data based on multimodal data. Background Art

[0002] Point cloud semantic segmentation is a cutting-edge technology in computer vision, aiming to assign a semantic category to each point in a 3D point cloud. Point cloud semantic segmentation provides critical 3D scene understanding support for intelligent and automated technological applications, such as environmental perception for autonomous vehicles, navigation and obstacle avoidance for intelligent robots, and enhanced realism in virtual reality environments.

[0003] In recent years, with the development of GPU acceleration technology and increased computing power, deep learning has achieved breakthroughs in many fields. Driven by powerful deep networks and large-scale datasets, fully supervised point cloud semantic segmentation methods have made significant progress. However, traditional methods have significant limitations: not only are they highly dependent on large amounts of labeled data, making the data labeling process time-consuming and labor-intensive, but they are also unable to segment new categories that have never been seen before in open datasets.

[0004] To address these challenges, a series of small-shot point cloud semantic segmentation techniques have emerged in recent years. These techniques aim to efficiently extract semantic features from a small number of labeled point clouds (the support set), thereby providing strong support for the accurate segmentation of new target categories in unlabeled point clouds (the query set). The difficulty of small-shot point cloud semantic segmentation lies in how to effectively utilize the limited support set, including the 3D point cloud and the ground truth segmentation mask (GT mask). Although some recent small-shot point cloud semantic segmentation schemes have leveraged prototype learning methods to first extract a single 3D prototype or fuse multiple 3D prototypes from the 3D support information, and then assign categories to the query set by measuring the distance between the query set features and the support set prototypes, existing methods generally have limitations—they fail to fully exploit the heterogeneous multimodal information naturally present in the support set. This lack of information utilization significantly restricts the improvement of segmentation performance, making it difficult for the algorithm to achieve ideal performance in complex scenarios.

[0005] The above analysis reveals the following problems with existing techniques: 1) Few-shot support sets fail to fully capture intra-class diversity, typically encompassing only limited object regions within a class, whereas the textual labels in the ground truth mask contain rich semantic knowledge and category association information; 2) Given the scarcity of point clouds, the 3D support set only provides the geometric structure of objects, while the projected 2D depth map provides clearer boundary and shape information. Therefore, this paper provides a semantic category classification method and system for 3D point cloud data based on multimodal data. This method integrates ground truth segmentation masks (GT masks) with 2D depth maps to construct multimodal support data to address the representational deficiencies of the support set. Summary of the Invention

[0006] The purpose of the present invention is to overcome the existing defects and provide a semantic category classification method and system for three-dimensional point cloud data based on multimodal data. By deeply mining the synergistic value of point cloud geometric structure, label text semantic knowledge and 2D depth map boundary details, it can realize the efficient utilization of multimodal information in small sample point cloud semantic segmentation tasks, and provide a new path for improving segmentation accuracy and generalization ability.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] The first object of the present invention is to provide a semantic category classification method for three-dimensional point cloud data based on multimodal data, comprising:

[0009] S1, data construction and preprocessing: obtain the original point cloud data and perform preprocessing, divide the point cloud data into a training set containing known categories and a test set containing new categories, and construct a small sample support set with annotations and a query set to be segmented for the training set and test set respectively;

[0010] S2, pre-training and feature extraction: Build a deep learning-based point cloud semantic segmentation model, use the training set to pre-train the point cloud semantic segmentation model and optimize the network parameters, and use the optimized model to extract point cloud features of the query set and support set in the training set;

[0011] S3, model training: For each category in the support set of the training set, three modal prototype representations are generated: 3D prototype, 2D prototype, and text prototype; the multimodal prototypes of each category are integrated to construct a unified prototype representation and perform prototype optimization; the distance between the optimized prototype and the query set point cloud features extracted in step S2 is calculated, and the semantic category of each point in the query is predicted according to the nearest neighbor principle; a relationship consistency loss function between multimodal prototypes is introduced to maintain the consistency of the structural relationship between multimodal prototypes, and the overall model is optimized and trained in combination with the cross entropy loss, ultimately obtaining a trained small-sample point cloud semantic segmentation model based on multimodal prototype enhancement;

[0012] S4, model testing: Generate, fuse, and optimize multimodal prototypes for the support set in the test set according to step S3, use the distance measurement mechanism to perform semantic segmentation prediction on the query set in the test set, and evaluate the model performance with the mIoU indicator.

[0013] Furthermore, the raw point cloud data is obtained and preprocessed to divide the point cloud data into a training set containing known categories and a test set containing new categories, including:

[0014] Obtain a point cloud semantic segmentation dataset, where the point cloud semantic segmentation dataset includes two open source indoor point cloud segmentation datasets;

[0015] Perform data preprocessing on the point cloud semantic segmentation dataset, divide the sampling scene into several blocks, and sample a balanced number of points;

[0016] The semantic categories are divided into a training set containing known categories and a test set containing new categories.

[0017] Furthermore, the point cloud semantic segmentation model includes a feature extractor, an attention learner and a metric learner;

[0018] In the pre-training and optimization of the point cloud semantic segmentation model, the point cloud semantic segmentation model is iteratively trained using the training set, and the cross entropy loss function is used as the optimization objective function.

[0019] Furthermore, in step S3, the multimodal prototype relationship consistency loss function includes: distance relationship consistency loss function And angle relationship consistency loss function , which are used to constrain the distance relationship between the two prototypes and the angle relationship between the three prototypes to be consistent. The specific formula is:

[0020]

[0021]

[0022]

[0023] in, is the relationship consistency loss function between multimodal prototypes, represents the optimized multimodal prototype set, represents the set of originally generated multimodal prototypes; represents the optimized prototype, Represents the original generated prototype; Respectively represent the category number; Indicates the query sample number; Represents Huber loss, a regression loss function of smoothed mean absolute error; and Represent the Euclidean distance function and the cosine angle relationship function respectively; represents the equilibrium parameter; They represent 3D prototype set, text prototype set, and 2D prototype set respectively.

[0024] Furthermore, in the optimization training of the overall model by combining cross entropy loss, the loss function based on the consistency of the relationship between multimodal prototypes is used. And the traditional cross entropy loss function Calculate the overall loss function , the calculation formula is:

[0025] in, represents the equilibrium parameter.

[0026] Another object of the present invention is to provide a semantic category classification system for three-dimensional point cloud data based on multimodal data, comprising:

[0027] The data collection and preprocessing module is used to obtain the point cloud semantic segmentation dataset, preprocess the point cloud semantic segmentation dataset, and divide it into training set and test set;

[0028] Model building module, used to build a point cloud semantic segmentation model including feature extractor, attention learner and metric learner;

[0029] The model pre-training module is used to iteratively train the point cloud semantic segmentation model using the training set and optimize the point cloud semantic segmentation model using the cross entropy loss function as the optimization objective function;

[0030] Model training module, used to train the point cloud semantic segmentation model based on the multimodal prototype enhancement unit and the relationship consistency loss function between multimodal prototypes;

[0031] The model testing module is used to segment new categories in the test set using the trained small-sample point cloud semantic segmentation model, and use mIoU as the evaluation metric to evaluate the performance of the small-sample point cloud semantic segmentation model.

[0032] Furthermore, the multimodal prototype enhancement unit is used to generate a multimodal prototype for each category, including 3D, text and 2D prototypes, and fuse the multimodal prototypes of all categories and optimize the fused modal prototypes.

[0033] Another object of the present invention is to provide an electronic device comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, the method for semantic category classification of three-dimensional point cloud data based on multimodal data provided by the first object of the present invention is implemented.

[0034] Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the semantic category classification method for three-dimensional point cloud data based on multimodal data provided by the first object of the present invention.

[0035] Another object of the present invention is to provide a server comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the three-dimensional point cloud data semantic category classification method based on multimodal data provided by the first object of the present invention.

[0036] In combination with the above technical solutions, the present invention has the following beneficial effects compared with the prior art:

[0037] The present invention requires only a small number of manually annotated point clouds to train a small-sample point cloud segmentation model based on multimodal prototype augmentation. This model can generate more robust, comprehensive, and discriminative optimized prototypes for the support set and better assist in the segmentation of the query set. Furthermore, the present invention only requires a small number of annotated point clouds for new categories to improve the performance of small-sample point cloud segmentation.

[0038] Compared with the existing small sample point cloud segmentation technology that only relies on a small amount of annotations, the present invention proposes a multimodal prototype enhancement mechanism and constructs a cross-modal prototype interaction framework. Through multimodal visual joint representation, a more robust, comprehensive and discriminative optimized prototype is generated for the support set, effectively improving the query set segmentation effect. Specifically, firstly, through the generation of text prototypes, the text label information corresponding to the GT mask is introduced to construct high-order semantic associations between categories, and then the connection between point cloud data and semantic concepts is established, the semantic expression ability of the prototype is enhanced, and it has cross-category semantic reasoning capabilities, thereby improving the model's understanding and segmentation performance in complex semantic scenes; secondly, through the generation of 2D prototypes, the boundaries and shape details in the 2D depth map are modeled, and through multimodal prototype fusion, the optimized prototype has both point cloud spatial topology and refined edge perception capabilities, significantly improving the segmentation accuracy of fuzzy boundaries and overlapping objects; thirdly, the innovative introduction of multimodal prototype relationship consistency loss function constrains the prototype relationship within each modality to be consistent with the optimized prototype relationship, breaking through the performance bottleneck of traditional methods from the root.

[0039] Compared with existing small-sample segmentation technologies that rely solely on 3D point cloud information, this invention innovatively integrates ground truth segmentation masks (GT masks) with 2D depth maps to construct multimodal support data to make up for the representational deficiencies of the support set. By deeply mining the synergistic value of point cloud geometric structure, labeled text semantic knowledge, and 2D depth map boundary details, efficient utilization of multimodal information in small-sample point cloud semantic segmentation tasks is achieved, providing a new path to improve segmentation accuracy and generalization capabilities. By introducing text modality and fusing it with point cloud prototypes, effective alignment of point cloud prototypes with high-order semantic concepts is achieved, which not only enhances the semantic relevance between categories, but also significantly improves the semantic generalization and cross-category reasoning capabilities of the prototypes, thereby effectively enhancing the model's understanding and segmentation capabilities of complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0041] Figure 1 is a flow chart of a method for semantic category classification of three-dimensional point cloud data based on multimodal data provided by an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of a semantic category classification method for three-dimensional point cloud data based on multimodal data provided by an embodiment of the present invention;

[0043] Figure 3 This is a principle architecture diagram of the training model stage provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0045] Example 1:

[0046] like Figure 1 FIG. 1 is an embodiment of a method for semantic classification of three-dimensional point cloud data based on multimodal data provided by the present invention, which specifically includes the following steps:

[0047] S1, data construction and preprocessing: obtain the original point cloud data and perform preprocessing, divide the point cloud data into a training set containing known categories and a test set containing new categories, and construct a small sample support set with annotations and a query set to be segmented for the training set and test set respectively;

[0048] S2, pre-training and feature extraction: Build a deep learning-based point cloud semantic segmentation model, use the training set to pre-train the point cloud semantic segmentation model and optimize the network parameters, and use the optimized model to extract point cloud features of the query set and support set in the training set;

[0049] S3, model training: For each category in the support set of the training set, three modal prototype representations are generated: 3D prototype, 2D prototype, and text prototype; the multimodal prototypes of each category are integrated to construct a unified prototype representation and perform prototype optimization; the distance between the optimized prototype and the query set point cloud features extracted in step S2 is calculated, and the semantic category of each point in the query is predicted according to the nearest neighbor principle; a relationship consistency loss function between multimodal prototypes is introduced to maintain the consistency of the structural relationship between multimodal prototypes, and the overall model is optimized and trained in combination with the cross entropy loss, ultimately obtaining a trained small-sample point cloud semantic segmentation model based on multimodal prototype enhancement;

[0050] S4, model testing: Generate, fuse, and optimize multimodal prototypes for the support set in the test set according to step S3, use the distance measurement mechanism to perform semantic segmentation prediction on the query set in the test set, and evaluate the model performance with the mIoU indicator.

[0051] It should be noted that if Figure 2 As shown, the method of the present invention includes a training model phase and a testing model phase. First, the point cloud dataset is preprocessed and divided into a training set and a test set. In the training model phase, the present invention first uses the training set to pre-train a point cloud semantic segmentation network for extracting point cloud features and segmenting known categories. Subsequently, the present invention uses the training set to fine-tune a small sample point cloud semantic segmentation model, wherein: the multimodal prototype enhancement module generates 3D, 2D, and text prototypes for the support set and further generates an optimized prototype through multimodal fusion; the distance between the point cloud features of the query set and the optimized prototype is measured to assign a predicted category to the query point cloud; in the training phase, the entire model is optimized simultaneously using the traditional cross entropy loss function and the proposed multimodal prototype relationship consistency loss function, which constrains the relationship consistency between multimodal prototypes to alleviate the multimodal gap. In the testing model phase, the proposed small sample point cloud semantic segmentation model is used to segment new categories in the test set, including the following steps: using the multimodal prototype enhancement method to generate an optimized prototype for the test support set, and measuring the distance between the query set point cloud features and the optimized prototype to obtain the final segmentation result for evaluation.

[0052] Specifically, step S1 of the embodiment of the present invention includes the following steps:

[0053] S1.1. Obtaining a point cloud semantic segmentation dataset: This dataset includes two open-source indoor point cloud segmentation datasets: S3DIS and ScanNet. The S3DIS dataset collects point clouds from 272 rooms in six indoor environments, where each point is labeled with one of 13 semantic classes, including 12 specific semantic categories and unlabeled miscellaneous classes. The ScanNet dataset contains 1,513 point clouds from 707 different indoor scenes, with each point assigned to one of 20 semantic classes.

[0054] S1.2, Data Preprocessing: Due to the large number of points in the original room, this method adopts a data preprocessing strategy. The room is divided into blocks using a 1m×1m non-overlapping sliding window on the xy plane. This strategy results in 7,547 and 36,350 blocks in S3DIS and ScanNet, respectively. Subsequently, when training and testing this method, 2048 points are randomly sampled from each block. Each point is represented by a 9D vector, including XYZ coordinates, RGB values, and normalized 3D spatial coordinates.

[0055] S1.3, division of training set and test set: For each data set, this method uses a fixed background category and divides the semantic class into two non-overlapping category subsets, denoted as S0 and S1. This method adopts a cross-validation method, such as using a category subset S0 as a known category to construct a training set, and using another subset S1 as a new category to construct a test set, and vice versa. It is worth emphasizing that the same point cloud can appear in S0 and S1, but due to the different classes of interest, the annotation category of this point cloud is different;

[0056] S1.4, Constructing a Small-Shot Support and Query Sets: For each class in the training set, we construct the support and query sets using the following steps: We first randomly sample N classes from the training class. Then, based on the selected N classes, we randomly sample a small number of labeled point clouds and unsegmented point clouds as the support set S and query set Q. The ground-truth masks M in the support set and the labels L in the query set are modified based on the original point annotations to correspond to the selected training class. The test support and query sets are formed in a similar manner, except that we exhaust all combinations of the N target classes from the new class, rather than randomly selecting N classes.

[0057] Specifically, step S2 of the embodiment of the present invention includes the following steps:

[0058] S2.1. Constructing a Point Cloud Semantic Segmentation Network: The point cloud segmentation network used in this method consists of three modules: a feature extractor, an attention learner, and a metric learner. Specifically, the feature extractor is constructed using a dynamic graph CNN architecture (DGCNN) to extract local geometric features; the attention learner uses a self-attention network (SAN) to aggregate global contextual features of the point cloud; and the metric learner is composed of multi-layer perceptron (MLP) layers to map all points in the support set and query set into a manifold space. We concatenate the learned features of the three layers as the output of the point cloud semantic segmentation network.

[0059] S2.2, pre-training point cloud semantic segmentation network: using the training set, iteratively train the point cloud semantic segmentation network constructed in S2.1, set the batch size to 32, and train the entire dataset for 100 rounds;

[0060] S2.3, Optimization objective function: When optimizing the network, use the cross entropy loss function as the optimization objective function, use the Adam optimizer and set the learning rate to 0.001 for training;

[0061] Specifically, step S3 of the embodiment of the present invention includes the following steps:

[0062] S3.1, Multimodal prototype generation: In order to fully explore the potential of the support set S, the present invention uses the potential multimodal information in the support set to generate multimodal prototypes for each category, including 3D, text and 2D prototypes. (1) 3D prototype: The present invention uses the point cloud semantic segmentation network pre-trained in step S2 as a shared point cloud encoder to extract point cloud features of the support set and query set, and uses the mask average pooling (MAP) method to generate category-level 3D prototypes; (2) Text prototype: The present invention extracts the label names hidden in the support set GT mask as text information, such as {'background', 'wall', 'window'}, and uses the frozen text encoder of the Contrastive Language-Image Pre-Training (CLIP) pre-training model to extract reliable text feature representation; and further uses a trainable semantic projection network composed of a multi-layer perceptron (MLP) to generate a text prototype; (3) 2D prototype: The present invention first projects the original point cloud from 6 views ({up, down, left, right, front, back}) to generate a multi-view 2D depth image; then uses the frozen visual encoder of CLIP to extract reliable visual features; and further uses a trainable visual projection network composed of a multi-layer perceptron (MLP) to generate a 2D prototype. The multimodal prototype obtained in this step can be expressed as:

[0063]

[0064]

[0065]

[0066] in, Represents a set of 3D, text, and 2D prototypes, each of which contains a background prototype and N foreground category prototypes; Representation category 3D, text, and 2D prototypes; represents the N foreground categories extracted in step S1.4, Represents the prototype feature dimension;

[0067] S3.2, Multimodal prototype fusion: This invention uses a simple and easily scalable prototype fusion method, that is, directly using feature-level averaging operation to obtain the fusion prototype set , where each fusion prototype The calculation formula is:

[0068]

[0069] In order to further alleviate the difference in feature channel distribution between prototype and query features, this paper uses a query-guided prototype adaptation (QGPA) module to optimize the fused prototype. , which generates a set of optimized prototypes . Assume the query set contains Different query point clouds are generated in this step. The group optimization prototype can be expressed as:

[0070]

[0071] S3.3, predict the query set segmentation result: Use the point cloud semantic segmentation network pre-trained in step S2 to extract the features of each point in the query set, and use a common distance function (such as Euclidean distance or cosine distance) to measure the similarity between the query features and the optimized prototype obtained in step S3.2. Finally, the category of the prototype closest to each query point is assigned as the predicted segmentation category;

[0072] S3.4, Multimodal prototype relationship consistency loss function: Multimodal prototype relationship consistency loss function It is used to constrain the relationship between prototypes in each mode to be as consistent as possible with the relationship between the final optimized prototypes. Specifically, there are two loss functions: distance relationship consistency loss function And angle relationship consistency loss function , which are used to constrain the distance relationship between two prototypes and the angle relationship between three prototypes to be consistent. The specific formula is:

[0073]

[0074]

[0075]

[0076] in, represents the optimized prototype set generated in step S3.2, represents the different multimodal prototype sets generated in step S3.1; Respectively represent the category number; Indicates the query sample number; Represents Huber loss, a regression loss function of smoothed mean absolute error; and Represent the Euclidean distance function and the cosine angle relationship function respectively; represents the balance parameter, which is set to 2 in the present invention;

[0077] S3.5, overall optimization objective function: overall loss function To simultaneously use the traditional cross entropy loss function in the segmentation field and the multimodal prototype relationship consistency loss function described in step S3.3 , the specific formula is:

[0078]

[0079] in, Represents the balance parameter, which is set to 1 in the present invention.

[0080] Specifically, step S4 of the embodiment of the present invention is a testing phase, which includes the following steps:

[0081] S4.1, multimodal prototype generation: similar to step S3.1, generate a multimodal prototype set for the test support set;

[0082] S4.2, multimodal prototype fusion: similar to step S3.2, generate an optimized prototype set;

[0083] S4.3, split the test query set; similar to step S3.3, use the distance metric function to predict the test query set segmentation result;

[0084] S4.4, Segmentation Performance Evaluation: Using mIoU as the evaluation metric, the performance of the test set segmentation results was evaluated. This method is referred to as RHP when using only 3D and text prototypes, and as RHP+ when using 3D, 2D, and text prototypes simultaneously. The evaluation results on the S3DIS and ScanNet datasets are shown in Tables 1 and 2, respectively. Experimental results demonstrate that the proposed method for semantic category classification of 3D point cloud data based on multimodal data outperforms other existing algorithms.

[0085] Table 1 Comparison between existing small sample point cloud semantic algorithms on the S3DIS dataset

[0086]

[0087] Table 2 Comparison with existing small sample point cloud semantic algorithms on the ScanNet dataset

[0088]

[0089] In particular, step S4 of the present invention is similar to step S3, except that the input of step S3 is the training support set and the training query set, which are used to segment known categories; while the input of step S4 is the test support set and the test query set, which are used to segment new categories that did not appear in the training phase;

[0090] In particular, step S4 freezes all parameters of the model and does not calculate any loss function;

[0091] Preferably, in steps S3.3 and S4.3, cosine distance is used as the distance metric function;

[0092] In summary, the embodiments of the present invention require only a small amount of manually annotated point clouds to train a small-sample point cloud segmentation model based on multimodal prototype augmentation. This model can generate more robust, comprehensive, and discriminative optimized prototypes for the support set and better assist in the segmentation of the query set. Furthermore, the present invention only requires a small amount of annotated point clouds for new categories to improve small-sample point cloud segmentation performance.

[0093] Example 2:

[0094] An embodiment of the present invention provides a semantic category classification system for three-dimensional point cloud data based on multimodal data, comprising:

[0095] The data collection and preprocessing module is used to obtain the point cloud semantic segmentation dataset, preprocess the point cloud semantic segmentation dataset, and divide it into training set and test set;

[0096] Model building module, used to build a point cloud semantic segmentation model including feature extractor, attention learner and metric learner;

[0097] The model pre-training module is used to iteratively train the point cloud semantic segmentation model using the training set and optimize the point cloud semantic segmentation model using the cross entropy loss function as the optimization objective function;

[0098] Model training module, used to train the point cloud semantic segmentation model based on the multimodal prototype enhancement unit and the relationship consistency loss function between multimodal prototypes;

[0099] The model testing module is used to segment new categories in the test set using the trained small-sample point cloud semantic segmentation model, and use mIoU as the evaluation metric to evaluate the performance of the small-sample point cloud semantic segmentation model.

[0100] Furthermore, the multimodal prototype enhancement unit is used to generate a multimodal prototype for each category, including 3D, text and 2D prototypes, and fuse the multimodal prototypes of all categories and optimize the fused modal prototypes.

[0101] Example 3: An embodiment of the present invention provides an electronic device, comprising a processor and a memory storing a computer program. When the processor executes the computer program, the method for semantic category classification of three-dimensional point cloud data based on multimodal data provided in Example 1 of the present invention is implemented.

[0102] Example 4: The embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for semantic category classification of three-dimensional point cloud data based on multimodal data provided in Example 1 of the present invention is implemented.

[0103] Example 5: An embodiment of the present invention provides a server comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to enable the at least one processor to execute the three-dimensional point cloud data semantic category classification method based on multimodal data provided in Example 1 of the present invention.

[0104] It should be understood that, although the various steps in the flow charts of the various embodiments of the present invention are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified in the present invention, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the various embodiments may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0105] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0106] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A semantic category classification method for three-dimensional point cloud data based on multimodal data, characterized in that: The method comprises: S1, data construction and preprocessing: obtain the original point cloud data and perform preprocessing, divide the point cloud data into a training set containing known categories and a test set containing new categories, and construct a small sample support set with annotations and a query set to be segmented for the training set and test set respectively; S2, pre-training and feature extraction: Build a deep learning-based point cloud semantic segmentation model, use the training set to pre-train the point cloud semantic segmentation model and optimize the network parameters, and use the optimized model to extract point cloud features of the query set and support set in the training set; S3, model training: For each category in the support set of the training set, three modal prototype representations are generated: 3D prototype, 2D prototype, and text prototype; the multimodal prototypes of each category are integrated to construct a unified prototype representation and perform prototype optimization; the distance between the optimized prototype and the query set point cloud features extracted in step S2 is calculated, and the semantic category of each point in the query is predicted according to the nearest neighbor principle; a relationship consistency loss function between multimodal prototypes is introduced to maintain the consistency of the structural relationship between multimodal prototypes, and the overall model is optimized and trained in combination with the cross entropy loss, ultimately obtaining a trained small-sample point cloud semantic segmentation model based on multimodal prototype enhancement; S4, model testing: Generate, fuse, and optimize multimodal prototypes for the support set in the test set according to step S3, use the distance measurement mechanism to perform semantic segmentation prediction on the query set in the test set, and evaluate the model performance with the mIoU indicator.

2. The method for semantic classification of three-dimensional point cloud data based on multimodal data according to claim 1, characterized in that: The method of obtaining the original point cloud data and performing preprocessing to divide the point cloud data into a training set containing known categories and a test set containing new categories includes: Obtain a point cloud semantic segmentation dataset, where the point cloud semantic segmentation dataset includes two open source indoor point cloud segmentation datasets; Perform data preprocessing on the point cloud semantic segmentation dataset, divide the sampling scene into several blocks, and sample a balanced number of points; The semantic categories are divided into a training set containing known categories and a test set containing new categories.

3. The method for semantic classification of three-dimensional point cloud data based on multimodal data according to claim 1, characterized in that: The point cloud semantic segmentation model includes a feature extractor, an attention learner and a metric learner; In the pre-training and optimization of the point cloud semantic segmentation model, the point cloud semantic segmentation model is iteratively trained using the training set, and the cross entropy loss function is used as the optimization objective function.

4. The method for semantic classification of three-dimensional point cloud data based on multimodal data according to claim 1, characterized in that: In step S3, the multimodal prototype relationship consistency loss function includes: distance relationship consistency loss function And angle relationship consistency loss function , which are used to constrain the distance relationship between the two prototypes and the angle relationship between the three prototypes to be consistent. The specific formula is: ； ； ； in, is the relationship consistency loss function between multimodal prototypes, represents the optimized multimodal prototype set, represents the set of originally generated multimodal prototypes; represents the optimized prototype, Represents the original generated prototype; Respectively represent the category number; Indicates the query sample number; Represents Huber loss, a regression loss function of smoothed mean absolute error; and Represent the Euclidean distance function and the cosine angle relationship function respectively; represents the equilibrium parameter; They represent 3D prototype set, text prototype set, and 2D prototype set respectively.

5. The method for semantic classification of three-dimensional point cloud data based on multimodal data according to claim 4, characterized in that: In the optimization training of the overall model by combining cross entropy loss, the loss function based on the consistency of the relationship between multimodal prototypes is used. And the traditional cross entropy loss function Calculate the overall loss function , the calculation formula is: ； in, represents the equilibrium parameter.

6. A semantic category classification system for three-dimensional point cloud data based on multimodal data, characterized in that: The system comprises: The data collection and preprocessing module is used to obtain the point cloud semantic segmentation dataset, preprocess the point cloud semantic segmentation dataset, and divide it into training set and test set; Model building module, used to build a point cloud semantic segmentation model including feature extractor, attention learner and metric learner; The model pre-training module is used to iteratively train the point cloud semantic segmentation model using the training set and optimize the point cloud semantic segmentation model using the cross entropy loss function as the optimization objective function; Model training module, used to train the point cloud semantic segmentation model based on the multimodal prototype enhancement unit and the relationship consistency loss function between multimodal prototypes; The model testing module is used to segment new categories in the test set using the trained small-sample point cloud semantic segmentation model, and use mIoU as the evaluation metric to evaluate the performance of the small-sample point cloud semantic segmentation model.

7. The three-dimensional point cloud data semantic category classification system based on multimodal data according to claim 6, characterized in that: The multimodal prototype enhancement unit is used to generate a multimodal prototype for each category, including 3D, text and 2D prototypes, and fuse the multimodal prototypes of all categories and optimize the fused modal prototypes.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the method for semantic category classification of three-dimensional point cloud data based on multimodal data according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for semantic category classification of three-dimensional point cloud data based on multimodal data according to any one of claims 1 to 5 is implemented.

10. A server, characterized in that: It includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the three-dimensional point cloud data semantic category classification method based on multimodal data as described in any one of claims 1 to 5.

Images