Few-sample 3D point cloud semantic segmentation method and system
By constructing a DGCNN backbone network, expansion attention block and multi-view prototype fusion module, the problem of low semantic segmentation accuracy of 3D point clouds in a small sample scenario is solved, and higher quality sample representativeness and model generalization capabilities are achieved.
Patent Information
- Application Number
- CN202510300999.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-14
- Publication Date
- 2025-06-27
AI Technical Summary
The prior art is difficult to ensure the accuracy of 3D point cloud semantic segmentation in a small sample scenario, especially when the sample information is small, it is difficult to understand and extract high-dimensional features, resulting in poor distinction and dependence between the foreground and the background.
By obtaining point cloud samples and screening, DGCNN backbone network, expansion attention block and multi-view prototype fusion module are built to form a semantic segmentation network for a small sample point cloud. The network uses high-quality sample set training to generate sample prototype features, and adjusts network parameters through multi-view prototype fusion module to ultimately realize semantic segmentation of point clouds.
It significantly improves the quality and representativeness of the sample, enhances the receptive field and generalization capabilities of the model, and can more accurately identify and segment objects in 3D point clouds, especially under the conditions of few samples.
Smart Images
Figure CN120219741A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of 3D point cloud semantic segmentation, and particularly to a few-shot 3D point cloud semantic segmentation method and system. Background Art
[0002] In the practical application of point cloud semantic segmentation tasks, researchers and workers may also encounter the problem of lacking sufficient labeled datasets, and accurately labeling a three-dimensional point cloud dataset requires a large amount of human and time costs.
[0003] In addition, when facing new datasets, existing models are often difficult to directly migrate and apply, which requires the models to be retrained to adapt to new tasks, further increasing the consumption of time and computing resources.
[0004] The problems existing in the prior art mainly include the following aspects:
[0005] (1) How to effectively process and recognize new target categories with only a very small number of samples for learning: The complexity of this task lies in that not only the objects of these rare categories need to be accurately recognized, but also under the condition of extremely limited learning samples, the foreground and background of the objects need to be distinguished and the dependency relationship of local point features needs to be extracted, which requires the model to capture very subtle feature differences. In this context, the distinction between foreground and background information and their dependency relationship become extremely crucial. Previous methods often perform poorly in this regard because they are difficult to understand and extract high-dimensional features related to the task at a deep level, especially when the sample information is scarce.
[0006] (2) The random selection method of samples leads to lack of representativeness of data: The current mainstream few-shot point cloud semantic segmentation sample selection strategy is mainly random selection. However, when this method is applied to the few-shot learning task of three-dimensional point cloud data, its limitations begin to become significant. Due to the inherent structural complexity and increased spatial dimensions of three-dimensional point cloud data, there are higher requirements for the quality of samples. The features of three-dimensional point cloud data include but are not limited to spatial distribution, density change, and relative position between objects. These characteristics require a more refined sample selection method to ensure learning efficiency and model performance. The random selection method may result in unrepresentative samples in this case, thereby affecting the model's learning ability for new categories.
[0007] (3)Style diversity of data in the same category: Differences in size, color, etc. of the same category in a scene may also have an adverse impact on model learning. Taking the specific furniture category "bed" as an example, although all beds are classified into the same category, there may be significant differences in design styles and appearance features among them, such as the distinction between classical and modern minimalist styles. In few-shot learning tasks, this inherent style difference may have a significant impact on the training effect of the model because the model needs to learn sufficient generalization ability from a very small number of samples to identify category instances that have not been seen before. Summary of the Invention
[0008] The purpose of the present invention is to provide a few-shot 3D point cloud semantic segmentation method and system to solve the problem that it is difficult to ensure the segmentation accuracy in the case of few-shot scenes for existing point cloud semantic segmentation methods.
[0009] The above object of the present application is achieved through the following technical solutions:
[0010] S1: Obtain point cloud samples and perform screening to obtain a high-quality sample set;
[0011] S2: Construct a few-shot point cloud semantic segmentation network through a DGCNN backbone network, dilated attention blocks, and a multi-view prototype fusion module;
[0012] S3: Input the high-quality sample set into the few-shot point cloud semantic segmentation network for training to obtain a sample prototype feature set;
[0013] S4: Obtain the prediction results of each prototype in the sample prototype feature set through the predictor in the multi-view prototype fusion module; adjust the parameters of the few-shot point cloud semantic segmentation network through the prediction results;
[0014] S5: Obtain the point cloud sample to be segmented; input the point cloud sample to be segmented into the few-shot point cloud semantic segmentation network with adjusted parameters to obtain a segmentation result, and complete the semantic segmentation of the few-shot 3D point cloud.
[0015] Optionally, step S1 includes:
[0016] Let the set of sample blocks to be selected in the point cloud sample be Calculate the proportion c of the target category points in the total number of points in the sample block k , when the proportion c k exceeds the set threshold t k , mark it as a high-quality sample;
[0017] Construct a high-quality sample set through the high-quality samples of each target category point.
[0018] Optionally, step S3 includes:
[0019] The expansion attention block includes: a feature extractor EC, a multi-layer perceptron MLP, and a two-layer self-attention module SA;
[0020] Input the high-quality samples in the high-quality sample set into the DGCNN backbone network of the few-shot point cloud semantic segmentation network for feature extraction to obtain the support feature and query feature f0 to be processed;
[0021] Process the query feature f0 through the feature extractor EC of the few-shot point cloud semantic segmentation network to obtain the pre-learned feature f1;
[0022] Use the multi-layer perceptron MLP to perform feature encoding on the feature f1 to obtain the high-dimensional feature f2;
[0023] Use the two-layer self-attention module SA to process the feature f1 to obtain the weighted feature f3;
[0024] Fuse the features f1, f2, and f3 to obtain the support or query encoded feature f4.
[0025] Optionally, step S3 further includes:
[0026] Obtain the foreground prototype f g and the background prototype b g ;
[0027] Through the foreground prototype b g and the background prototype f g , combined with the three-layer multi-head attention module MA of the few-shot point cloud semantic segmentation network, obtain the deeply fused background prototype
[0028] Fuse the deeply fused background prototype with the foreground prototype f g to obtain the sample prototype feature p of the high-quality sample;
[0029] Construct a sample prototype feature set through the sample prototype feature of each high-quality sample as follows:
[0030]
[0031] where s i represents the i-th high-quality sample, and n is the number of high-quality samples; represents the sample prototype feature of the i-th high-quality sample.
[0032] Optionally, step S4 includes:
[0033] Use the predictor to obtain the prediction result {r 1 …ri …r n}, and calculate its corresponding cross-entropy loss {L 1 …L i …L n};
[0034] Find the mean value of the cross-entropy loss as the total loss value;
[0035] Adjust the parameters of the few-shot point cloud semantic segmentation network through the total loss value.
[0036] A few-shot 3D point cloud semantic segmentation system, the system includes: a construction module, a model training module, and a point cloud sample segmentation module;
[0037] The construction module, the model training module, and the point cloud sample segmentation module are sequentially connected;
[0038] The construction module is used to obtain point cloud samples and perform screening to obtain a high-quality sample set;
[0039] The construction module is also used to construct a few-shot point cloud semantic segmentation network through a DGCNN backbone network, a dilated attention block, and a multi-view prototype fusion module;
[0040] The model training module is used to input the high-quality sample set into the few-shot point cloud semantic segmentation network for training to obtain a sample prototype feature set;
[0041] The model training module is also used for the predictor in the multi-view prototype fusion module to obtain the prediction results of each prototype in the sample prototype feature set; adjust the parameters of the few-shot point cloud semantic segmentation network through the prediction results;
[0042] The point cloud sample segmentation module is used to obtain the point cloud sample to be segmented; input the point cloud sample to be segmented into the few-shot point cloud semantic segmentation network with adjusted parameters to obtain a segmentation result, and complete the semantic segmentation of the few-shot 3D point cloud.
[0043] An electronic device, including a processor, a memory, a user interface, and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes a few-shot 3D point cloud semantic segmentation method.
[0044] A computer-readable storage medium, the computer-readable storage medium stores instructions, and when the instructions are executed, a few-shot 3D point cloud semantic segmentation method is executed.
[0045] The beneficial effects brought by the technical solution provided by this application are:
[0046] 1. By analyzing the proportion of sample blocks among samples, the sample blocks with higher proportions are screened out, marked as high-quality samples and preferentially selected, thereby improving the training effect. The screening of high-quality samples aims to select representative and diverse samples from limited samples to address the problem of sample unevenness. Different from the conventional random sample selection strategy, this component significantly improves the quality and representativeness of samples, providing a more effective learning basis for the model.
[0047] 2. The dilated attention block deeply explores the effective information of the foreground and background, significantly expanding the receptive field of the model. Different from previous processing methods, this module effectively distinguishes and understands various detailed changes in the scene by utilizing the principle of dilated convolution. By reusing the attention module for the features extracted by attention, the receptive field of the network can be increased, and the problem of uneven information distribution can be solved more deeply.
[0048] 3. Use high-quality samples in different blocks to generate prototypes from multiple perspectives and use each prototype for prediction. Subsequently, calculate the cross-entropy loss function for multiple prediction results and take the average. By integrating prototypes generated from different perspectives and samples, a comprehensive class representation is formed, thereby enhancing the generalization and recognition ability of the model for different styles under the same category. Thus, prototype features with stronger generalization ability are obtained, reducing the impact of inter-class diversity on the model.
[0049] Fully mine the available information in limited data to improve the accuracy of the prototype feature descriptor. The dilated attention block focuses on the separation of foreground and background information, providing a basis for high-quality prototypes. The multi-perspective prototype fusion module starts from the samples themselves and intra-class differences, further enhancing the generalization ability of the prototypes. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The following will further illustrate the present application in conjunction with the drawings and embodiments. In the drawings:
[0051] Figure 1 is the step diagram in the embodiment of the present application;
[0052] Figure 2 is the structural diagram of the dilated attention block in the embodiment of the present application;
[0053] Figure 3 is the multi-perspective prototype fusion module diagram in the embodiment of the present application;
[0054] Figure 4 is the schematic diagram of the electronic device structure in the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0055] For a clearer understanding of the technical features, objectives, and effects of the present application, the specific embodiments of the present application will now be described in detail with reference to the drawings.
[0056] An embodiment of the present application provides a few-shot 3D point cloud semantic segmentation method.
[0057] Please refer to Figure 1 , Figure 1 which is a step diagram of a few-shot 3D point cloud semantic segmentation method in an embodiment of the present application, including:
[0058] S1: Obtain point cloud samples and perform screening to obtain a high-quality sample set;
[0059] S2: Construct a few-shot point cloud semantic segmentation network through a DGCNN backbone network, a dilated attention block, and a multi-view prototype fusion module;
[0060] S3: Input the high-quality sample set into the few-shot point cloud semantic segmentation network for training to obtain a sample prototype feature set;
[0061] S4: Obtain the prediction result of each prototype in the sample prototype feature set through the predictor in the multi-view prototype fusion module; adjust the parameters of the few-shot point cloud semantic segmentation network through the prediction result;
[0062] S5: Obtain the point cloud sample to be segmented; input the point cloud sample to be segmented into the few-shot point cloud semantic segmentation network with adjusted parameters to obtain a segmentation result, and complete the semantic segmentation of the few-shot 3D point cloud.
[0063] In another embodiment, the few-shot point cloud semantic segmentation network (EAMP-Net) of the present application includes the following two key parts: 1. A dilated attention block is designed, which can effectively capture depth details and strengthen the separation between foreground and background. 2. The proposed method introduces a multi-view prototype fusion module, which can significantly improve the quality of prototypes and the accuracy of target object feature descriptors. Experimental results on two benchmark datasets, S3DIS and ScanNet, show that the proposed method has achieved excellent performance in the few-shot point cloud semantic segmentation task.
[0064] In another embodiment, features first pass through a multi-layer edge convolution module, and then are fused through a self-attention block and a multi-layer perceptron. Subsequently, a high-quality support sample is screened out by a high-quality sample screening component via a preset rule, and multiple contrast prototypes are generated through a dilated attention block, and the prototype samples of the target object from multiple angles are fused. Finally, class prediction is achieved by calculating the Gaussian similarity between the query sample and the prototype matrix.
[0065] Step S1 includes:
[0066] Let the set of sample blocks to be selected in the point cloud sample be Calculate the proportion c of the target category points in the total number of points in the sample block k, when the proportion c k exceeds the set threshold t k , mark it as a high-quality sample;
[0067] Construct a high-quality sample set through the high-quality samples of each target category point.
[0068] In one embodiment, the data used in the present invention is the public datasets S3DIS and ScanNet. The categories of the target category points include: sofas, chairs, tables, etc.
[0069] As an embodiment, since there are differences in the proportions of different categories in the sample blocks (for example, walls are more likely to obtain a larger point proportion relative to chairs), different thresholds should be set for different categories. For the convenience of calculation, multiply the total proportion of each category in the point cloud by 0.8 as the threshold for each category. Form a high-quality sample set Q. The algorithm is as follows:
[0070]
[0071] where represents the number of points of the target category in the block where it is located; represents the total number of points of the target category in the block where it is located; p k represents the number of points of the target category in the dataset; p s represents the total number of points of the target category in the dataset; s i represents the set of sample blocks to be selected; represents that there exists a category sample k, whose c k is greater than t k , and can be used as a high-quality sample.
[0072] Step S3 includes:
[0073] The dilation attention block includes: a feature extractor EC, a multi-layer perceptron MLP, and a two-layer self-attention module SA;
[0074] Input the high-quality samples in the high-quality sample set into the DGCNN backbone network of the few-shot point cloud semantic segmentation network for feature extraction to obtain the support feature and query feature f0 to be processed;
[0075] Process the query feature f0 through the feature extractor EC of the few-shot point cloud semantic segmentation network to obtain the pre-learned feature f1;
[0076] Use the multi-layer perceptron MLP to perform feature encoding on the feature f1 to obtain the high-dimensional feature f2;
[0077] Use the two-layer self-attention module SA to process the feature f1 to obtain the weighted feature f3;
[0078] Fuse the features f1, f2, and f3 to obtain the support or query encoded feature f4.
[0079] As an example, the algorithm for the support or query encoded feature f4 is as follows:
[0080] f1 = EC(f0)
[0081] f2 = MLP(f1)
[0082] f3 = SA(f1) 2
[0083]
[0084] As an example, the specific structure of the dilated attention block is as Figure 2 shown, where N is the number of points, EC is the feature extractor, SA is the self-attention, and MLP is the multi-layer perceptron. Self-attention is a mechanism widely used in sequence data processing. It is a method that enables the model to weight and emphasize information at different positions in the sequence, thereby improving the understanding and representation of the dynamics within the sequence.
[0085] Step S3 further includes:
[0086] Obtain the foreground prototype f g and the background prototype b g ;
[0087] Through the foreground prototype b g and the background prototype f g , combined with the three-layer multi-head attention module MA of the few-shot point cloud semantic segmentation network, obtain the deeply fused background prototype
[0088] Fuse the deeply fused background prototype with the foreground prototype f g to obtain the sample prototype feature p of the high-quality sample;
[0089] Construct a sample prototype feature set through the sample prototype feature of each high-quality sample as follows:
[0090]
[0091] where s i represents the i-th high-quality sample, and n is the number of high-quality samples; represents the sample prototype feature of the i-th high-quality sample.
[0092] As an example, for the encoded support feature f4, first obtain the foreground and background prototypes b obtained through the above dilated attention blockg With f g Subsequently, a three-layer multi-head attention module MA is used to obtain the depth-fused background prototype Finally, the foreground is fused with the depth-fused background prototype to obtain the sample prototype feature p of this iteration. Ultimately, the category prediction can be achieved by calculating the feature similarity between the prototype feature and the query feature. The implementation algorithm is as follows:
[0093]
[0094] As an embodiment, the specific structure is as Figure 3 shown, where K is the number of prototypes and MA is the multi-head attention. The multi-head attention is an extension of the self-attention mechanism. When processing sequence data, it can simultaneously focus on different subspaces of the sequence, enabling the model to capture multiple different aspects of the data in parallel, thereby obtaining richer and more complex dependency relationships and improving the performance and flexibility of the model.
[0095] Step S4 includes:
[0096] Using a predictor to obtain the prediction result {r 1 / r i / r n} corresponding to each sample prototype, and calculating its corresponding cross-entropy loss {L 1 …L i …L n};
[0097] Calculating the mean value of the cross-entropy loss as the total loss value;
[0098] Adjusting the parameters of the few-shot point cloud semantic segmentation network through the total loss value.
[0099] A few-shot 3D point cloud semantic segmentation system, the system includes: a construction module, a model training module, and a point cloud sample segmentation module;
[0100] The construction module, the model training module, and the point cloud sample segmentation module are sequentially connected in series;
[0101] The construction module is used to obtain point cloud samples and perform screening to obtain a high-quality sample set;
[0102] The construction module is also used to construct a few-shot point cloud semantic segmentation network through a DGCNN backbone network, a dilation attention block, and a multi-view prototype fusion module;
[0103] The model training module is used to input the high-quality sample set into the few-shot point cloud semantic segmentation network for training to obtain a sample prototype feature set;
[0104] The model training module is also used for the predictor in the multi-view prototype fusion module to obtain the prediction results of each prototype in the sample prototype feature set; and adjust the parameters of the few-shot point cloud semantic segmentation network based on the prediction results.
[0105] The point cloud sample segmentation module is used to obtain the point cloud sample to be segmented; input the point cloud sample to be segmented into the few-shot point cloud semantic segmentation network with adjusted parameters to obtain the segmentation result, thereby completing the semantic segmentation of the few-shot 3D point cloud.
[0106] This application also discloses an electronic device. Referring to Figure 4 , Figure 4 FIG. is a schematic structural diagram of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.
[0107] Among them, the communication bus 502 is used to realize the connection and communication between these components.
[0108] Among them, the user interface 503 may include a display screen, and optionally, the user interface 503 may further include a standard wired interface and a wireless interface.
[0109] Among them, the network interface 504 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).
[0110] This application also discloses a computer-readable storage medium, which stores multiple instructions adapted to be loaded by a processor to execute the above-mentioned few-shot 3D point cloud semantic segmentation method.
[0111] The above are only exemplary embodiments of the present disclosure, and the scope of the present disclosure cannot be limited thereby. That is, any equivalent changes and modifications made according to the teachings of the present disclosure still fall within the scope covered by the present disclosure.
[0112] This application aims to cover any variations, uses, or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include well-known common knowledge or conventional technical means in the technical field not recorded in the present disclosure. The description and embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.
Claims
1. A few-sample 3D point cloud semantic segmentation method, characterized in that: The method comprises the following steps: S1: Obtain point cloud samples and screen them to obtain high-quality sample sets; S2: Construct a few-shot point cloud semantic segmentation network through the DGCNN backbone network, dilated attention block and multi-view prototype fusion module; S3: Input the high-quality sample set into the few-sample point cloud semantic segmentation network for training to obtain the sample prototype feature set; S4: Obtain the prediction results of each prototype in the sample prototype feature set through the predictor in the multi-view prototype fusion module; adjust the parameters of the few-sample point cloud semantic segmentation network based on the prediction results; S5: Obtain point cloud samples to be segmented; input the point cloud samples to be segmented into the few-sample point cloud semantic segmentation network with adjusted parameters to obtain the segmentation result, and complete the semantic segmentation of the few-sample 3D point cloud.
2. The method for semantic segmentation of 3D point clouds with few samples according to claim 1, characterized in that: Step S1 includes: Suppose the sample block set to be selected in the point cloud sample is Calculate the proportion of target category points in the total number of points in the sample block c k , when the proportion c k Exceeds the set threshold t k When , it is marked as a high-quality sample; A high-quality sample set is constructed through high-quality samples of each target category point.
3. The method for semantic segmentation of 3D point clouds with few samples according to claim 1, characterized in that: Step S3 includes: The expanded attention block includes: feature extractor EC, multi-layer perceptron MLP and two-layer self-attention module SA; Input the high-quality samples in the high-quality sample set into the DGCNN backbone network of the few-sample point cloud semantic segmentation network for feature extraction, and obtain the support features to be processed and the query features f0; The query feature f0 is processed through the feature extractor EC of the few-shot point cloud semantic segmentation network to obtain the pre-learned feature f1; Use multi-layer perceptron MLP to encode feature f1 and obtain high-dimensional feature f2; Use a two-layer self-attention module SA to process feature f1 and obtain weighted feature f3; Features f1, f2 and f3 are fused to obtain the supporting or query encoding feature f4.
4. The method for semantic segmentation of 3D point clouds with a small number of samples according to claim 3, characterized in that: Step S3 also includes: By supporting or querying the encoded feature f4, we get the foreground prototype f g And background prototype b g ; By foreground prototype b g And background prototype f g , combined with the three-layer multi-head attention module MA of the few-sample point cloud semantic segmentation network, to obtain the deep fusion background prototype Deep Fusion Background Prototype Prototype with prospects Perform fusion to obtain the sample prototype feature p of high-quality samples; Construct a sample prototype feature set through the sample prototype features of each high-quality sample as follows: Among them, s i represents the i-th high-quality sample, and n is the number of high-quality samples; Represents the sample prototype feature of the i-th high-quality sample.
5. The method for semantic segmentation of 3D point clouds with few samples according to claim 4, characterized in that: Step S4 includes: Using the predictor, obtain the prediction result corresponding to each sample prototype {r 1 …r i …r n }, and calculate the corresponding cross entropy loss {L 1 …L i …L n }; Find the mean of the cross entropy loss As the total loss value; Adjust the parameters of the few-shot point cloud semantic segmentation network through the total loss value.
6. A few-sample 3D point cloud semantic segmentation system, used to implement a few-sample 3D point cloud semantic segmentation method according to any one of claims 1 to 5, characterized in that: The system includes: a construction module, a model training module and a point cloud sample segmentation module; The construction module, the model training module and the point cloud sample segmentation module are connected in sequence; The construction module is used to obtain point cloud samples and screen them to obtain a high-quality sample set; The building module is also used to construct a few-shot point cloud semantic segmentation network through the DGCNN backbone network, dilated attention block, and multi-view prototype fusion module; The model training module is used to input the high-quality sample set into the few-sample point cloud semantic segmentation network for training to obtain the sample prototype feature set; The model training module is also used as the predictor in the multi-view prototype fusion module to obtain the prediction results of each prototype in the sample prototype feature set; the parameters of the few-sample point cloud semantic segmentation network are adjusted based on the prediction results; The point cloud sample segmentation module is used to obtain point cloud samples to be segmented; the point cloud samples to be segmented are input into the few-sample point cloud semantic segmentation network with adjusted parameters to obtain the segmentation results, thus completing the semantic segmentation of the few-sample 3D point cloud.
7. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed by a computer, the method according to any one of claims 1 to 5 is executed.