Three-dimensional point cloud data semantic category classification method and system based on multi-modal data

Abstract

The application belongs to the technical field of point cloud data processing, and provides a three-dimensional point cloud data semantic category classification method and system based on multi-modal data. First, the point cloud data set is preprocessed and divided into a training set and a test set. A point cloud semantic segmentation model is constructed and pre-trained and optimized. A small sample point cloud semantic segmentation model based on multi-modal prototype enhancement is fine-tuned and trained using a training support set and a query set. The trained small sample point cloud semantic segmentation model is obtained. The test support set and the query set are input into the trained small sample point cloud semantic segmentation model. The trained small sample point cloud semantic segmentation model performs semantic segmentation on the input point cloud. The segmentation result is output and used to calculate the segmentation performance. The application deeply excavates the collaborative value of point cloud geometric structure, label text semantic knowledge and 2D depth map boundary details, realizes efficient utilization of multi-modal information in the small sample point cloud semantic segmentation task, and provides a new path for improving segmentation accuracy and generalization ability.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of point cloud data processing, and particularly relates to a three-dimensional point cloud data semantic category classification method and system based on multi-modal data. BACKGROUND

[0002] Point cloud semantic segmentation is a frontier technology in the field of computer vision, aiming to assign a semantic category to each point in a 3D point cloud. Point cloud semantic segmentation provides key three-dimensional scene understanding support for intelligent and automated technology applications, such as environmental perception for autonomous vehicles, navigation and obstacle avoidance for intelligent robots, and realistic enhancement for virtual reality environments.

[0003] In recent years, with the development of GPU acceleration technology and the improvement of computing power, deep learning has made breakthroughs in many fields. With the strong support of deep networks and large-scale data sets, fully supervised point cloud semantic segmentation methods have made significant progress. However, traditional methods have significant limitations: they are highly dependent on large-scale labeled data, resulting in time-consuming and labor-intensive data labeling processes, and in open set scenarios, they cannot segment new categories that have never appeared before.

[0004] To address the above challenges, a series of small-sample point cloud semantic segmentation technologies have emerged in recent years, aiming to efficiently extract semantic features from a small amount of labeled point clouds (support set) to provide strong support for accurate segmentation of new target categories in unlabeled point clouds (query set). The difficulty of small-sample point cloud semantic segmentation lies in how to efficiently utilize limited support sets, including 3D point clouds and ground truth segmentation masks (GT masks). Although some recent small-sample point cloud semantic segmentation schemes use prototype learning methods, they first extract a single 3D prototype or multiple 3D prototypes from 3D support information, and then measure the distance between query set features and support set prototypes to assign categories to the query set. However, existing methods generally have limitations— they fail to fully exploit the naturally occurring heterogeneous multi-modal information in the support set. This lack of information utilization greatly limits the improvement of segmentation performance, making the algorithm's performance in complex scenarios difficult to achieve ideal state.

[0005] Based on the above analysis, the existing technology has the following problems: 1) The small-sample support set is difficult to fully cover the intra-class diversity, usually containing only limited object regions within the class, while the text label of the GT mask has rich semantic knowledge and class association information; 2) Considering the scarcity of point clouds, the 3D support set only provides the geometric structure of the object, while the 2D depth map projected by it provides clearer boundary information and shape information. Therefore, the present application provides a three-dimensional point cloud data semantic category classification method and system based on multi-modal data, which integrates ground truth segmentation masks (GT masks) and 2D depth maps to construct multi-modal support data to compensate for the representation defects of the support set. SUMMARY

[0006] The purpose of the present application is to overcome the existing defects, provide a three-dimensional point cloud data semantic category classification method and system based on multi-modal data, by deeply mining the collaborative value of point cloud geometric structure, label text semantic knowledge and 2D depth map boundary details, realizing efficient utilization of multi-modal information in small sample point cloud semantic segmentation task, and providing a new path for improving segmentation accuracy and generalization ability.

[0007] In order to solve the above technical problems, the present application provides the following technical scheme:

[0008] The first purpose of the present application is to provide a three-dimensional point cloud data semantic category classification method based on multi-modal data, comprising:

[0009] S1, data construction and preprocessing: obtaining original point cloud data and preprocessing, dividing the point cloud data into a training set containing known categories and a test set containing new categories, constructing a small sample support set containing annotations and a query set to be segmented for the training set and the test set respectively;

[0010] S2, pre-training and feature extraction: constructing a point cloud semantic segmentation model based on deep learning, pre-training the point cloud semantic segmentation model using the training set and optimizing the network parameters, and extracting the point cloud features of the query set and the support set in the training set using the optimized model;

[0011] S3, model training: for each category of the support set in the training set, generate three modal prototype representations: 3D prototype, 2D prototype and text prototype; fuse the multi-modal prototypes of each category to construct a unified prototype representation and optimize the prototype; calculate the distance between the optimized prototype and the query set point cloud features extracted in step S2, and predict the semantic category of each point in the query according to the nearest neighbor principle; introduce a multi-modal prototype relationship consistency loss function to maintain the structural relationship consistency between the multi-modal prototypes, and combine the cross-entropy loss to optimize and train the overall model, and finally obtain a trained small sample point cloud semantic segmentation model based on multi-modal prototype enhancement;

[0012] S4, model testing: the support set in the test set is subjected to multi-modal prototype generation, fusion and optimization according to step S3, the query set in the test set is subjected to semantic segmentation prediction using the distance measurement mechanism, and the model performance is evaluated by the mIoU index.

[0013] Further, the obtaining of the original point cloud data and the preprocessing, and the division of the point cloud data into a training set containing known categories and a test set containing new categories, comprises:

[0014] Obtaining a point cloud semantic segmentation dataset, the point cloud semantic segmentation dataset containing two open source indoor point cloud segmentation datasets;

[0015] Data preprocessing is performed on the point cloud semantic segmentation dataset, and the sampled scene is divided into several blocks, and an equal number of points are sampled;

[0016] The semantic categories are divided into a training set containing known categories and a test set containing new categories.

[0017] Further, the point cloud semantic segmentation model comprises a feature extractor, an attention learner and a metric learner;

[0018] In the pre-training and optimization of the point cloud semantic segmentation model, the training set is used to iteratively train the point cloud semantic segmentation model, and the cross-entropy loss function is used as the optimization objective function.

[0019] Further, in step S3, the multi-modal prototype inter-relationship consistency loss function comprises: a distance relationship consistency loss function And an angle relationship consistency loss function , respectively used to constrain the distance relationship consistency of two prototypes and the angle relationship consistency among three prototypes, and the specific formula is:

[0020]

[0021]

[0022]

[0023] wherein, is the multi-modal prototype inter-relationship consistency loss function, represents the optimized multi-modal prototype set, represents the original generated multi-modal prototype set; represents the optimized prototype, represents the original generated prototype; respectively represent the category number; represents the query sample number; represents the Huber loss, which is a kind of smooth mean absolute error regression loss function; and respectively represent the Euclidean distance function and the cosine angle relationship function; represents the balance parameter; respectively represent the 3D prototype set, the text prototype set and the 2D prototype set.

[0024] Further, in the optimization training of the whole model based on the combination of the cross-entropy loss, the multi-modal prototype inter-relationship consistency loss function and the traditional cross-entropy loss function are used to calculate the total loss function , and the calculation formula is:

[0025] wherein, represents the equilibrium parameter.

[0026] Another object of the present application is to provide a three-dimensional point cloud data semantic category classification system based on multi-modal data, comprising:

[0027] A data collection and preprocessing module is configured to obtain a point cloud semantic segmentation dataset, preprocess the point cloud semantic segmentation dataset, and divide the point cloud semantic segmentation dataset into a training set and a test set.

[0028] A model construction module is configured to construct a point cloud semantic segmentation model comprising a feature extractor, an attention learner, and a metric learner.

[0029] A model pre-training module is configured to iteratively train the point cloud semantic segmentation model using the training set, and optimize the point cloud semantic segmentation model using a cross-entropy loss function as an optimization objective function.

[0030] A model training module is configured to train the point cloud semantic segmentation model based on a multi-modal prototype enhancement unit and a multi-modal prototype inter-relation consistency loss function.

[0031] A model testing module is configured to segment new categories in the test set using the small sample point cloud semantic segmentation model after training, and evaluate the performance of the small sample point cloud semantic segmentation model using mIoU as an evaluation index.

[0032] Further, the multi-modal prototype enhancement unit is configured to generate multi-modal prototypes for each category, including 3D, text, and 2D prototypes, fuse the multi-modal prototypes of all categories, and optimize the fused modal prototypes.

[0033] Another object of the present application is to provide an electronic device comprising a processor and a memory storing a computer program, wherein the processor executes the computer program to implement the three-dimensional point cloud data semantic category classification method based on multi-modal data provided by the first object of the present application.

[0034] Another object of the present application is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the three-dimensional point cloud data semantic category classification method based on multi-modal data provided by the first object of the present application.

[0035] Another object of the present application is to provide a server comprising at least one processor, and a memory connected in communication with the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor to cause the at least one processor to perform the method for classifying semantic categories of three-dimensional point cloud data based on multi-modal data according to the first object of the present application.

[0036] In combination with the above technical solution, the present application has the following beneficial effects compared with the prior art:

[0037] The present application only needs a small amount of manually annotated point cloud to train a small sample point cloud segmentation model based on multi-modal prototype enhancement, and the model can generate more robust, comprehensive and discriminative optimized prototypes for the support set and better assist the segmentation of the query set. In addition, the present application only needs to provide a small amount of annotated point cloud for new categories to improve the performance of small sample point cloud segmentation.

[0038] Compared with the existing small sample point cloud segmentation technology which only relies on a small amount of annotation, the present application proposes a multi-modal prototype enhancement mechanism and constructs a cross-modal prototype interaction framework. Through multi-modal visual joint representation, more robust, comprehensive and discriminative optimized prototypes are generated for the support set, effectively improving the segmentation effect of the query set. Specifically, first, by generating a text prototype, the text label information corresponding to the GT mask is introduced to construct the high-order semantic association between categories, and then the relationship between point cloud data and semantic concepts is established to enhance the semantic expression ability of the prototype and enable it to have cross-class semantic reasoning ability, thereby improving the understanding and segmentation performance of the model in complex semantic scenarios; second, by generating a 2D prototype, the boundary and shape details in the 2D depth map are modeled, and the multi-modal prototype fusion is performed to enable the optimized prototype to have point cloud spatial topology and fine edge perception ability, thereby significantly improving the segmentation accuracy of fuzzy boundaries and overlapping objects; third, a multi-modal prototype relationship consistency loss function is innovatively introduced to constrain the consistency of the prototype relationship in each modality and the optimized prototype relationship, thereby breaking through the performance bottleneck of traditional methods from the root.

[0039] Compared with the existing small sample segmentation technology which only relies on 3D point cloud information, the present application innovatively integrates the ground truth segmentation mask (GT mask) and the 2D depth map to construct multi-modal support data to make up for the representation defects of the support set. By deeply mining the collaborative value of point cloud geometric structure, label text semantic knowledge and 2D depth map boundary details, efficient utilization of multi-modal information in the small sample point cloud semantic segmentation task is realized, providing a new path for improving segmentation accuracy and generalization ability. By introducing a text modality and fusing it with the point cloud prototype, effective alignment of the point cloud prototype and high-order semantic concepts is realized, not only enhancing the semantic association between categories, but also significantly improving the semantic generalization ability and cross-class reasoning ability of the prototype, thereby effectively enhancing the understanding and segmentation ability of the model in complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and are meant to explain the application and are not intended to limit the application. In the drawings:

[0041] Figure 1 is a flow chart of a three-dimensional point cloud data semantic category classification method based on multi-modal data provided by an embodiment of the application;

[0042] Figure 2 is a principle diagram of a three-dimensional point cloud data semantic category classification method based on multi-modal data provided by an embodiment of the application;

[0043] Figure 3 is a principle architecture diagram of a training model stage provided by an embodiment of the application. DETAILED DESCRIPTION

[0044] The preferred embodiments of the application are described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to explain and illustrate the application, and are not used to limit the application.

[0045] Embodiment 1:

[0046] As shown in Figure 1 , it is one embodiment of a three-dimensional point cloud data semantic category classification method based on multi-modal data provided by the application, which specifically includes the following steps:

[0047] S1, data construction and preprocessing: obtaining original point cloud data and preprocessing, dividing the point cloud data into a training set containing known categories and a test set containing new categories, respectively constructing a small sample support set containing annotations and a query set to be segmented for the training set and the test set;

[0048] S2, pre-training and feature extraction: constructing a point cloud semantic segmentation model based on deep learning, pre-training the point cloud semantic segmentation model using the training set and optimizing the network parameters, and extracting the point cloud features of the query set and the support set in the training set using the optimized model;

[0049] S3, model training: for each class of the support set in the training set, generate prototype representations of three modalities: 3D prototype, 2D prototype and text prototype; fuse the multi-modal prototypes of each class to construct a unified prototype representation and perform prototype optimization; calculate the distance between the optimized prototype and the query set point cloud features extracted in step S2, and predict the semantic class of each point in the query according to the nearest neighbor principle; introduce a multi-modal prototype relationship consistency loss function to maintain the structural relationship consistency between the multi-modal prototypes, and combine the cross-entropy loss to optimize and train the overall model, and finally obtain a trained small sample point cloud semantic segmentation model based on multi-modal prototype enhancement;

[0050] S4, model testing: for the support set in the test set, perform multi-modal prototype generation, fusion and optimization according to step S3, use the distance measurement mechanism to perform semantic segmentation prediction on the query set in the test set, and evaluate the model performance with the mIoU index.

[0051] It should be noted that, as shown in Figure 2 The method of the present application includes a training model stage and a testing model stage. First, the point cloud data set is preprocessed and divided into a training set and a test set. In the training model stage, the present application first uses the training set to pre-train a point cloud semantic segmentation network for extracting point cloud features and segmenting known classes. Subsequently, the present application uses the training set to fine-tune a small sample point cloud semantic segmentation model, wherein: a multi-modal prototype enhancement module generates 3D, 2D and text prototypes for the support set and further generates optimized prototypes through multi-modal fusion; the distance between the point cloud features of the query set and the optimized prototype is measured to specify the predicted class for the query point cloud; in the training stage, the entire model is optimized by the traditional cross-entropy loss function and the proposed multi-modal prototype relationship consistency loss function, which constrains the relationship consistency between the multi-modal prototypes to alleviate the multi-modal gap. In the testing model stage, the proposed small sample point cloud semantic segmentation model is used to segment new classes in the test set, including the following steps: using the multi-modal prototype enhancement method to generate optimized prototypes for the test support set, measuring the distance between the query set point cloud features and the optimized prototype to obtain the final segmentation result and used to evaluate the effect.

[0052] Specifically, step S1 of the embodiment of the present application includes the following steps:

[0053] S1.1, Obtain point cloud semantic segmentation dataset: It contains two open source indoor point cloud segmentation datasets S3DIS and ScanNet. Among them, the S3DIS dataset collects point clouds of 272 rooms in 6 indoor environments, and each point is labeled with one of the 13 semantic classes, including 12 specific semantic categories and a no-label miscellaneous class; the ScanNet dataset contains 1513 point clouds, which come from 707 different indoor scenes, and each point is assigned to one of the 20 semantic classes;

[0054] S1.2, Data preprocessing: Because the number of points in the original room is large, the method adopts a data preprocessing strategy, which divides the room into blocks using a 1m x 1m non-overlapping sliding window in the xy plane, which obtains 7,547 and 36,350 blocks in S3DIS and ScanNet, respectively; then, when training and testing the method, 2048 points are randomly selected from each block, and each point is represented by a 9D vector, including XYZ coordinates, RGB values and normalized three-dimensional spatial coordinates;

[0055] S1.3, Divide the training set and the test set: For each dataset, the method fixedly uses a background class, and divides the semantic classes into two non-overlapping class subsets, denoted as S0 and S1. The method uses cross-validation, such as using one class subset S0 as the known class to construct the training set, and using the other subset S1 as the new class to construct the test set, and vice versa. It is worth emphasizing that the same point cloud can appear in S0 and S1, but due to the difference in the class of interest, the annotation class of this point cloud is different;

[0056] S1.4, Construct small sample support set and query set: For each class in the training set, we use the following steps to construct the support set and the query set: we first randomly select N classes from the training classes; then, according to the selected N classes, a small number of labeled point clouds and point clouds to be segmented are randomly selected as the support set S and the query set Q. The GT mask M in the support set and the label L in the query set are modified accordingly on the basis of the original point annotation to correspond to the selected training classes. The test support set and the query set are formed in a similar way, the difference lies in that we exhaust all combinations of N target classes from the new classes, instead of randomly selecting N classes.

[0057] Specifically, the step S2 of the embodiment of the present application comprises the following steps:

[0058] S2.1, constructing the point cloud semantic segmentation network: the point cloud segmentation network adopted by the method is composed of three modules: a feature extractor, an attention learner, and a metric learner. Specifically, the feature extractor is composed of a dynamic graph CNN architecture (DGCNN) and can extract local geometric features; the attention learner adopts a self-attention network (SAN) and can aggregate global context features of the point cloud; the metric learner is composed of a multi-layer perceptron (MLP) layer and can map all points of the support set and the query set to a manifold space. We connect the learned three layers of features together as the output of the point cloud semantic segmentation network;

[0059] S2.2, pre-training the point cloud semantic segmentation network: using the training set, iteratively train the point cloud semantic segmentation network constructed in S2.1, set the batch size to 32, and train the entire data set for 100 rounds;

[0060] S2.3, optimizing the objective function: when optimizing the network, use the cross-entropy loss function as the optimization objective function, use the Adam optimizer and set the learning rate to 0.001 for training;

[0061] Specifically, the step S3 of the embodiment of the application comprises the following steps:

[0062] S3.1, multi-modal prototype generation: in order to fully exploit the potential of the support set S, the application generates multi-modal prototypes for each class using the potential multi-modal information in the support set, including 3D, text and 2D prototypes. (1) 3D prototype: the application uses the point cloud semantic segmentation network pre-trained in step S2 as a shared point cloud encoder to extract point cloud features of the support set and the query set, and uses the mask average pooling (MAP) method to generate a class-level 3D prototype; (2) text prototype: the application extracts the label name hidden in the support set GT mask as text information, such as {‘background’, ‘wall’, ‘window’}, uses the frozen text encoder of the pre-trained contrastive text-image pair (Contrastive Language-Image Pre-Training, abbreviated as CLIP) to extract reliable text feature representation, and further uses a trainable semantic projection network composed of a multi-layer perceptron (MLP) to generate a text prototype; (3) 2D prototype: the application first projects the original point cloud from 6 perspectives ({top, bottom, left, right, front, back}) to generate multi-view 2D depth images; then uses the frozen visual encoder of CLIP to extract reliable visual features; and further uses a trainable visual projection network composed of a multi-layer perceptron (MLP) to generate a 2D prototype. The multi-modal prototypes obtained in this step can be represented as:

[0063]

[0064]

[0065]

[0066] wherein, denotes 3D, text, 2D prototype sets, each set contains one background prototype and N foreground class prototypes; denotes class of 3D, text, 2D prototypes; denotes N foreground classes extracted in step S1.4, denotes prototype feature dimension;

[0067] S3.2, multi-modal prototype fusion: the present application uses a simple and easy-to-expand prototype fusion method, i.e. directly using feature-level average operation to obtain the fused prototype set wherein each fused prototype The calculation formula is:

[0068]

[0069] In order to further alleviate the feature channel distribution difference between the prototype and the query set feature, the present application uses a query-guided prototype adaptation (QGPA) module to optimize the fused prototype. For each query point cloud This module can generate a set of optimized prototypes Suppose the query set contains different query point clouds, this step generates sets of optimized prototypes, which can be represented as:

[0070]

[0071] S3.3, predicting query set segmentation results: using the point cloud semantic segmentation network pre-trained in step S2 to extract the features of each point in the query set, and using common distance functions (such as Euclidean distance or cosine distance) to measure the similarity between the query features and the optimized prototypes obtained in step S3.2, finally specifying the class of the prototype closest to each query point as the predicted segmentation class;

[0072] S3.4, multi-modal prototype relationship consistency loss function: the multi-modal prototype relationship consistency loss function is used to constrain the relationship between prototypes within each modality to be as consistent as possible with the relationship between the final optimized prototypes. Specifically, it contains two loss functions: distance relationship consistency loss function and angle relationship consistency loss function , which are respectively used to constrain the distance relationship consistency of two prototypes and the angle relationship consistency among three prototypes. The specific formula is:

[0073]

[0074]

[0075]

[0076] wherein, represents the optimized prototype set generated in step S3.2, represents the different multi-modal state prototype set generated in step S3.1; respectively represent the class number; represents the query sample number; represents the Huber loss, which is a kind of smooth mean absolute error regression loss function; and respectively represent the Euclidean distance function and the cosine angle relationship function; represents the balance parameter, which is set to 2 in the present application;

[0077] S3.5, overall optimization objective function: overall loss function is the traditional cross-entropy loss function used in the segmentation field and the multi-modal prototype relationship consistency loss function described in step S3.3 , the specific formula is:

[0078]

[0079] wherein, represents the balance parameter, which is set to 1 in the present application.

[0080] Specifically, step S4 of the embodiment of the present application is a test phase, including the following steps:

[0081] S4.1, multi-modal prototype generation: similar to step S3.1, generate a multi-modal prototype set for the test support set;

[0082] S4.2, multi-modal prototype fusion: similar to step S3.2, generate an optimized prototype set;

[0083] S4.3, segmentation test query set; similar to step S3.3, use the distance measurement function to predict the segmentation result of the test query set;

[0084] S4.4, evaluate segmentation performance: use mIoU as evaluation indicator to evaluate the performance of the segmentation result of the test set. The present application is referred to as RHP when only using 3D and text prototypes, and is referred to as RHP+ when using 3D, 2D and text prototypes. The evaluation results on the S3DIS and ScanNet datasets are described in Tables 1 and 2, respectively. The experimental results show that the three-dimensional point cloud data semantic category classification method based on multi-modal data proposed in the present application is superior to other existing algorithms.

[0085] Table 1 Comparison between the present small sample point cloud semantic algorithm and the existing small sample point cloud semantic algorithm on the S3DIS dataset

[0086]

[0087] Table 2 Comparison between the present small sample point cloud semantic algorithm and the existing small sample point cloud semantic algorithm on the ScanNet dataset

[0088]

[0089] In particular, step S4 of the present application is similar to step S3, the difference being that the input of step S3 is the training support set and the training query set for segmenting known categories; while the input of step S4 is the test support set and the test query set for segmenting new categories not appearing in the training phase;

[0090] In particular, step S4 freezes all parameters of the model and does not calculate any loss function;

[0091] Preferably, cosine distance is used as the distance metric function in steps S3.3 and S4.3;

[0092] In summary, the embodiment of the present application only needs a small amount of manually labeled point clouds to train a small sample point cloud segmentation model based on multi-modal prototype enhancement, which can generate more robust, comprehensive and discriminative optimized prototypes for the support set and better assist the segmentation of the query set. In addition, the present application only needs to provide a small amount of labeled point clouds for new categories to improve the performance of small sample point cloud segmentation.

[0093] Embodiment 2:

[0094] The embodiment of the present application provides a three-dimensional point cloud data semantic category classification system based on multi-modal data, comprising:

[0095] A data collection and preprocessing module is configured to obtain a point cloud semantic segmentation dataset, preprocess the point cloud semantic segmentation dataset, and divide the point cloud semantic segmentation dataset into a training set and a test set;

[0096] A model construction module is configured to construct a point cloud semantic segmentation model comprising a feature extractor, an attention learner and a metric learner;

[0097] The model pre-training module is configured to iteratively train the point cloud semantic segmentation model using a training set and optimize the point cloud semantic segmentation model using a cross-entropy loss function as an optimization objective function.

[0098] The model training module is configured to train the point cloud semantic segmentation model based on the multi-modal prototype enhancement unit and a multi-modal prototype inter-relation consistency loss function.

[0099] The model testing module is configured to segment new classes in a test set using the trained small sample point cloud semantic segmentation model and evaluate the performance of the small sample point cloud semantic segmentation model using mIoU as an evaluation index.

[0100] Further, the multi-modal prototype enhancement unit is configured to generate multi-modal prototypes for each class, including 3D, text and 2D prototypes, fuse the multi-modal prototypes of all classes, and optimize the fused modal prototypes.

[0101] Embodiment 3: The electronic device provided by the embodiment of the present application comprises a processor and a memory storing a computer program, and the processor implements the method for classifying semantic categories of three-dimensional point cloud data based on multi-modal data provided by the embodiment 1 of the present application when executing the computer program.

[0102] Embodiment 4: The computer readable storage medium provided by the embodiment of the present application stores a computer program, and the computer program is executed by a processor to implement the method for classifying semantic categories of three-dimensional point cloud data based on multi-modal data provided by the embodiment 1 of the present application.

[0103] Embodiment 5: The server provided by the embodiment of the present application comprises at least one processor and a memory in communication connection with the processor, the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to enable the at least one processor to execute the method for classifying semantic categories of three-dimensional point cloud data based on multi-modal data provided by the embodiment 1 of the present application.

[0104] It should be understood that, although each step in the flowchart of each embodiment of the present application is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified in the present application, the execution of these steps has no strict sequence limitation, and these steps can be executed in other orders. Moreover, at least a part of the steps in each embodiment can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least a part of other steps or sub-steps or stages of other steps.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0106] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for limiting the present application, although the above-mentioned embodiments of the present application are described in detail, for those skilled in the art, it still can be modified, or part of the technical features of the equivalent replacement of the technical solutions recorded in the above-mentioned embodiments. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for semantic class classification of three-dimensional point cloud data based on multi-modal data, characterized in that, The method comprises: S1, data construction and preprocessing: obtaining original point cloud data and preprocessing, dividing the point cloud data into a training set containing known categories and a test set containing new categories, and constructing a small sample support set containing annotations and a query set to be segmented for the training set and the test set respectively; S2, pre-training and feature extraction: constructing a point cloud semantic segmentation model based on deep learning, pre-training the point cloud semantic segmentation model using the training set and optimizing the network parameters, and extracting the point cloud features of the query set and the support set in the training set using the optimized model; S3, model training: for each category of the support set in the training set, generating three kinds of modal prototype representations: 3D prototype, 2D prototype and text prototype; fusing the multi-modal prototypes of each category to construct a unified prototype representation and optimizing the prototype; calculating the distance between the optimized prototype and the query set point cloud features extracted in step S2, and predicting the semantic category of each point in the query according to the nearest neighbor principle; introducing a multi-modal prototype relationship consistency loss function to maintain the structural relationship consistency between the multi-modal prototypes, and combining a cross-entropy loss to optimize and train the overall model, finally obtaining a trained multi-modal prototype enhanced small sample point cloud semantic segmentation model; The multimodal inter-prototype relationship consistency loss function comprises: a distance relationship consistency loss function and an angle relationship consistency loss function respectively used for constraining the distance relationship consistency of two prototypes and the angle relationship consistency among three prototypes. S4, model testing: generating, fusing and optimizing the multi-modal prototypes of the support set in the test set according to step S3, using a distance measurement mechanism to predict the semantic segmentation of the query set in the test set, and evaluating the model performance with mIoU index.

2. The method of claim 1, wherein, The method comprises: Obtaining a point cloud semantic segmentation dataset, the point cloud semantic segmentation dataset containing two open source indoor point cloud segmentation datasets; Data preprocessing is performed on the point cloud semantic segmentation dataset, the sampling scene is divided into several blocks, and an equal number of points are sampled; The semantic categories are divided into a training set containing known categories and a test set containing new categories.

3. The method of claim 1, wherein, The point cloud semantic segmentation model comprises a feature extractor, an attention learner and a metric learner; In the pre-training and optimization of the point cloud semantic segmentation model, the training set is used to iteratively train the point cloud semantic segmentation model, and a cross-entropy loss function is used as the optimization objective function.

4. The method of claim 1, wherein, In step S3, the multi-modal prototype relationship consistency loss function has the following specific formula: ； ； ； wherein, is a multi-modal prototype inter-consistency loss function, denotes an optimized multi-modal prototype set, denotes an original generated multi-modal prototype set; denotes an optimized prototype, denotes an original generated prototype; denotes a class number, respectively; denotes a query sample number; denotes a Huber loss, which is a kind of smooth mean absolute error regression loss function; and denote a Euclidean distance function and a cosine angle relationship function, respectively; denotes a balance parameter; denote a 3D prototype set, a text prototype set, and a 2D prototype set, respectively.

5. The method of claim 4, wherein, The combined cross-entropy loss optimizes the whole model in the training of the whole model based on a relationship consistency loss function between multi-modal prototypes and a traditional cross-entropy loss function The total loss function is calculated , and the calculation formula is: ； wherein represents the equilibrium parameter.

6. A system for semantic class classification of three-dimensional point cloud data based on multi-modal data, the system comprising: The system comprises: A data collection and preprocessing module for obtaining a point cloud semantic segmentation dataset, preprocessing the point cloud semantic segmentation dataset, and dividing it into a training set and a test set; A model construction module for constructing a point cloud semantic segmentation model comprising a feature extractor, an attention learner and a metric learner; A model pre-training module for iteratively training the point cloud semantic segmentation model using the training set, and optimizing the point cloud semantic segmentation model using a cross-entropy loss function as the optimization objective function; The model training module is configured to train the point cloud semantic segmentation model based on a multi-modal prototype enhancement unit and a multi-modal prototype inter-relation consistency loss function; the multi-modal prototype enhancement unit is configured to generate multi-modal prototypes for each category, including 3D, text and 2D prototypes, and fuse the multi-modal prototypes of all categories, and optimize the fused modal prototypes; the multi-modal prototype inter-relation consistency loss function includes a distance relation consistency loss function and an angle relation consistency loss function , which are respectively configured to constrain the distance relation consistency of two prototypes and the angle relation consistency among three prototypes. A model testing module for segmenting new categories in the test set using the trained small sample point cloud semantic segmentation model, and evaluating the performance of the small sample point cloud semantic segmentation model using mIoU as the evaluation index.

7. An electronic device comprising a processor and a memory having a computer program stored therein, characterized in that The computer program is executed by the processor to implement the method for classifying semantic categories of three-dimensional point cloud data based on multi-modal data according to any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method for classifying semantic categories of three-dimensional point cloud data based on multi-modal data according to any one of claims 1 to 5.

9. A server, characterized by: The computer program is executed by the processor to implement the method for classifying semantic categories of three-dimensional point cloud data based on multi-modal data according to any one of claims 1 to 5.

Images