Small sample point cloud semantic segmentation method, network, storage medium and processor

Abstract

The present application is applicable to the field of point cloud semantic segmentation technology, and provides a small sample point cloud semantic segmentation method, network, storage medium and processor. The small sample point cloud semantic segmentation method uses label propagation to extract pseudo-prototype features that are adapted to the query set data, thereby obtaining prototype features that are adapted to the query set data, and performs feature calibration by extracting the relationship between the prototype and the query set data. Prototype expansion effectively utilizes the distribution information of the query set data and the prototype information of the support set. This further improves the adaptability of the prototype to the query set data. Therefore, the present invention can obtain prototype features that are adapted to the query set, achieve effective segmentation of the point cloud scene, and reduce the impact of misjudgment results on the expanded prototype.

Description

Technical Field

[0001] The present application belongs to the technical field of point cloud semantic segmentation, and in particular relates to a small sample point cloud semantic segmentation method, network, storage medium and processor. Background Art

[0002] Current small-sample point cloud semantic segmentation techniques primarily utilize prototype features from annotated support data to propagate labels on query data to obtain corresponding point cloud labels. However, due to the limited amount of annotated data, there is a discrepancy between the support and query data. Therefore, using decision boundaries derived solely from the support data makes it difficult to achieve accurate segmentation on the query data. Existing techniques have shortcomings. Summary of the Invention

[0003] The purpose of this application is to provide a small sample point cloud semantic segmentation method, network and storage medium and processor, which can flexibly cooperate with the multiple functions assigned by secondary users and regulate the corresponding system resources to derive the reachable rate area for the information security scenario of the cognitive system and solve the technical problem of establishing a secure coexistence relationship between authorized users and cognitive users.

[0004] On the one hand, the present application provides a small sample point cloud semantic segmentation method, comprising the following steps:

[0005] s1. Input the support set point cloud and the query set point cloud into the feature extractor to obtain the corresponding support set features and query set features;

[0006] s2. The support set features and the query set features are input into the prototype expansion module to obtain an expanded multi-prototype;

[0007] s3. The amplified multiple prototypes and the query set features are input into a feature calibration module to obtain calibration prototype features and calibration query features;

[0008] s4. The distance between the calibration query feature and the calibration prototype feature is measured by a label propagation module, and label propagation is performed to achieve semantic segmentation of the query set point cloud.

[0009] On the other hand, the present application also provides a small sample point cloud semantic segmentation network using the above semantic segmentation method, including a sequentially connected feature extractor, a prototype amplification module, a feature calibration module and a label propagation module;

[0010] The feature extractor extracts features from the support set point cloud and the query set point cloud, and outputs corresponding support set features and query set features;

[0011] The prototype expansion module combines the support set features and the query set features to obtain expanded multiple prototypes;

[0012] The feature calibration module exchanges information between the multiple prototypes and the query set features to obtain calibrated prototype features and calibrated query features;

[0013] The label propagation module measures the distance between the calibration query feature and the calibration prototype feature to perform label propagation, thereby achieving semantic segmentation of the query set point cloud.

[0014] On the other hand, the present application also provides a storage medium, which stores a program file that can implement the above-mentioned small sample point cloud semantic segmentation method.

[0015] On the other hand, the present application also provides a processor, which is used to run a program, wherein the program executes the above-mentioned small sample point cloud semantic segmentation method when running.

[0016] This application proposes a small-sample point cloud semantic segmentation method, network, storage medium, and processor. This application utilizes label propagation to extract pseudo-prototype features that are adapted to the query set data, thereby obtaining prototype features that are adapted to the query set data. Feature calibration is then performed by extracting the relationship between the prototype and the query set data. Prototype expansion effectively utilizes the distribution information of the query set data and the prototype information of the support set, further improving the adaptability of the prototype to the query set data. Therefore, the present invention can obtain prototype features that are adapted to the query set and achieve effective segmentation of point cloud scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of the small sample point cloud semantic segmentation method of this application;

[0018] Figure 2 This is a schematic diagram of the architecture of the prototype amplification module of this application;

[0019] Figure 3 It is a schematic diagram of the architecture of the feature calibration module of this application;

[0020] Figure 4 Schematic diagram of the main steps of the small sample point cloud semantic segmentation method in this application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-4 It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0022] The following describes the specific implementation of this application in detail with reference to specific embodiments:

[0023] Example 1:

[0024] Figure 1-4 The implementation process of the small sample point cloud semantic segmentation method provided in Example 1 of the present application is shown. For ease of explanation, only the part related to the embodiment of the present application is shown, which is detailed as follows:

[0025] On the one hand, the present application provides a small sample point cloud semantic segmentation method, comprising the following steps:

[0026] s1. Input the support set point cloud and the query set point cloud into the feature extractor to obtain the corresponding support set features and query set features;

[0027] s2. The support set features and the query set features are input into the prototype expansion module to obtain an expanded multi-prototype;

[0028] s3. The amplified multiple prototypes and the query set features are input into a feature calibration module to obtain calibration prototype features and calibration query features;

[0029] s4. The distance between the calibration query feature and the calibration prototype feature is measured by a label propagation module, and label propagation is performed to achieve semantic segmentation of the query set point cloud.

[0030] Specifically, the solution process of the present invention is as follows Figure 1 As shown. The support set data and query set data are passed through a shared feature extractor to obtain the corresponding support set features f s and query set features f q Afterwards, we will support the set feature f s and query set features f q Input into the prototype expansion module to obtain multiple prototypes p. At the same time, the expanded multiple prototypes p and the query set features f q Input to the feature calibration module to obtain the calibration prototype features and calibration query features Finally, semantic segmentation of point cloud is achieved through label propagation.

[0031] Furthermore, the step s2 includes the following steps:

[0032] s21. The support set features are aggregated with the support set mask to obtain multiple prototypes of the support set;

[0033] s22. According to the Euclidean distance from the query set feature to the multi-prototype of the support set, assign each query set feature the label of the multi-prototype of the support set closest to it; obtain a pseudo label of the query set feature;

[0034] s23. The query set features are aggregated with the pseudo labels of the query set to obtain a pseudo prototype;

[0035] s24. The multiple prototypes of the support set are combined with the pseudo-prototype to obtain the expanded multiple prototypes.

[0036] The structure of the prototype amplification module is as follows Figure 2 As shown in Figure 2. We input the support set features and query set features into the prototype augmentation module. The support set features are aggregated to obtain the support set's multiple prototypes. Through feature aggregation, we obtain the corresponding prototype features. Next, based on the Euclidean distance from the query set features to the support set's multiple prototypes, we assign each query set feature the label of the closest support set's multiple prototype, obtaining a pseudo-label for the query set feature. The query set features are then aggregated again to obtain the pseudo-prototype. By combining the support set's multiple prototypes with the pseudo-prototypes, we obtain augmented multiple prototypes.

[0037] Furthermore, the feature aggregation includes the following steps:

[0038] j1. Extract similar features based on labels;

[0039] j2. Perform the farthest point sampling on the same features to obtain the initial cluster center;

[0040] j3. Assign the feature to the nearest cluster center based on its distance to the initial cluster center;

[0041] j4. Take the average of the features belonging to each cluster center as the final prototype feature.

[0042] Furthermore, the feature calibration module realizes the information flow between the multi-prototype information and the query set features through two shared residual cross-attention modules, and then outputs the calibrated prototype features and the calibrated query features.

[0043] Furthermore, the residual cross attention module obtains attention features and query vector features through three convolution kernels, adopts the attention mechanism to extract the relationship matrix between input features, and uses the query vector to extract channel information.

[0044] Specifically, as attached Figure 3 As shown, Figure 3 Part (a) shows the specific architecture of the feature calibration module, which realizes the information flow between prototype information and query set features through two shared residual attention modules; Figure 3 Part (b) shows the specific structure of the residual cross attention module, which obtains the attention features and query vector features through three convolution kernels.

[0045] We input the amplified prototype into the feature calibration module, as shown in the attached Figure 3 The feature calibration module consists of two shared residual cross attention modules to achieve the prototype feature p and the query set feature fq The residual cross attention module uses the attention mechanism to extract the relationship matrix between input features and uses the query vector to extract channel information. Taking prototype calibration as an example, its calculation method is as follows:

[0046]

[0047] where Q p =k1(p),K p =k2(f q ), V p =k1(p), C is the number of channels of the feature. p represents the query vector during prototype calibration, K p represents the key vector during prototype calibration, V p A vector of values ​​representing the prototype calibration.

[0048] Characteristic calibration module through Q p , K p Get the relationship matrix between the input features and combine it with V p Get attention features to strengthen the connection between prototype and query set features; through Q p Extract information between feature channels and adjust attention features to obtain better feature distribution.

[0049] After the feature calibration module, we get the calibrated prototype features With queryset features

[0050] Furthermore, the step s4 includes the following steps:

[0051] s41. The calibration prototype features and the calibration query features are used as nodes of a graph;

[0052] s42. The prototype labels of the calibration prototype features are continuously propagated on the graph to obtain the corresponding label values ​​of each node;

[0053] s43. Using a Gaussian similarity matrix to represent the similarity between each of the nodes, and normalizing it to achieve label propagation;

[0054] s44. Extract node information corresponding to the query set feature, normalize the node information to obtain a prediction corresponding to each query set feature, and implement semantic segmentation of the query set point cloud.

[0055] Specifically, the calibrated prototype features and the calibrated query features are used as nodes v in the graph. The prototype labels are continuously propagated across the graph to obtain the corresponding label values ​​for each node. This propagation process relies on the adjacency matrix of the nodes in the graph. A Gaussian similarity matrix is ​​used to represent the similarity between nodes and is normalized to achieve label propagation. The node information corresponding to the query set features is then extracted and normalized to obtain the prediction corresponding to each query set feature. This achieves semantic segmentation of the query set point cloud.

[0056] Among them, the loss function includes the loss between the pseudo label and the true label during label propagation and the loss between the final predicted value and the true label.

[0057] The present invention proposes a scheme for prototype expansion and feature calibration. To address the problem that prototypes with a small amount of support set data find it difficult to obtain an excellent decision boundary on query set data, we propose to expand the prototypes using query set data. At the same time, considering that the expanded prototypes rely on the prototypes of the support set data, and the prototypes of the support set data may have misjudgment results when assigning labels to the query set data. To reduce the impact of misjudgment results on the expanded prototypes, we propose to calibrate the features using the relationship between the prototype and the query set data. To further reduce the impact of misjudgment data, we add a label assignment loss function on the basis of the original segmentation loss function.

[0058] Example 2:

[0059] On the other hand, the present application also provides a small sample point cloud semantic segmentation network using the above semantic segmentation method, including a sequentially connected feature extractor, a prototype amplification module, a feature calibration module and a label propagation module;

[0060] The feature extractor extracts features from the support set point cloud and the query set point cloud, and outputs corresponding support set features and query set features;

[0061] The prototype expansion module combines the support set features and the query set features to obtain expanded multiple prototypes;

[0062] The feature calibration module exchanges information between the multiple prototypes and the query set features to obtain calibrated prototype features and calibrated query features;

[0063] The label propagation module measures the distance between the calibration query feature and the calibration prototype feature to perform label propagation, thereby achieving semantic segmentation of the query set point cloud.

[0064] Furthermore, the prototype amplification module includes a first feature aggregation unit, a label assignment unit, a second feature aggregation unit, and a combination unit connected in sequence;

[0065] The first feature aggregation unit performs feature aggregation on the support set features based on the support set mask to obtain multiple prototypes of the support set;

[0066] The label assignment unit assigns each query set feature the label of the multi-prototype of the support set closest to it based on the Euclidean distance from the query set feature to the multi-prototype of the support set, thereby obtaining a pseudo label of the query set feature;

[0067] The second feature aggregation unit performs feature aggregation on the query set features based on the pseudo labels of the query set to obtain a pseudo prototype;

[0068] The combining unit combines the multiple prototypes of the support set with the pseudo prototype to obtain an augmented multiple prototype;

[0069] The feature calibration module realizes the information flow between prototype information and query set features through two shared residual cross-attention modules.

[0070] Specifically, the feature calibration module contains two shared residual cross attention modules to achieve the prototype feature p and the query set feature f q The residual cross attention module uses the attention mechanism to extract the relationship matrix between input features and uses the query vector to extract channel information. Taking prototype calibration as an example, its calculation method is as follows:

[0071]

[0072]

[0073] The query vector on time, K p represents the key vector during prototype calibration, V p A vector of values ​​representing the prototype calibration.

[0074] After the feature calibration module, we get the calibrated prototype features With queryset features

[0075] The label propagation module uses the calibrated prototype features and the calibrated query features as nodes v in the graph. The prototype labels are continuously propagated across the graph to obtain the corresponding label values ​​for each node. This propagation process relies on the adjacency matrix of the nodes in the graph. We use a Gaussian similarity matrix to represent the similarity between nodes and normalize it to achieve label propagation. The node information corresponding to the query set features is then extracted and normalized to obtain the corresponding prediction for each query and feature. The loss function includes the loss between the pseudo-label and the true label during label propagation, and the loss between the final prediction and the true label.

[0076] Example 3:

[0077] On the other hand, the present application also provides a storage medium, which stores a program file that can implement the above-mentioned small sample point cloud semantic segmentation method.

[0078] Those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc.

[0079] Example 4:

[0080] On the other hand, the present application also provides a processor, which is used to run a program, wherein the program executes the above-mentioned small sample point cloud semantic segmentation method when running.

[0081] In the embodiments of the present application, the small sample point cloud semantic segmentation method can be implemented by corresponding hardware or software units. Each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit. This is not intended to limit the present application. The specific implementation of each unit can be referred to the description of Example 1 and will not be repeated here.

[0082] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A small sample point cloud semantic segmentation method, characterized by: The steps include: s1. Input the support set point cloud and the query set point cloud into the feature extractor to obtain the corresponding support set features and query set features; s2. The support set features and the query set features are input into the prototype expansion module to obtain an expanded multi-prototype; s3. The amplified multiple prototypes and the query set features are input into a feature calibration module to obtain calibration prototype features and calibration query features; s4. Measure the distance between the calibration query feature and the calibration prototype feature through the label propagation module, perform label propagation, and achieve semantic segmentation of the query set point cloud; The step s2 comprises the following steps: s21. The support set features are aggregated with the support set mask to obtain multiple prototypes of the support set; s22. According to the Euclidean distance from the query set feature to the multi-prototype of the support set, assign each query set feature the label of the multi-prototype of the support set closest to it; obtain a pseudo label of the query set feature; s23. The query set features are aggregated with the pseudo labels of the query set to obtain a pseudo prototype; s24. The multiple prototypes of the support set are combined with the pseudo-prototype to obtain the expanded multiple prototypes.

2. The semantic segmentation method according to claim 1, wherein: The feature aggregation includes the following steps: j1. Extract similar features based on labels; j2. Perform the farthest point sampling on the same features to obtain the initial cluster center; j3. Assign the feature to the nearest cluster center based on its distance to the initial cluster center; j4. Take the average of the features belonging to each cluster center as the final prototype feature.

3. The semantic segmentation method according to claim 1, wherein: The feature calibration module realizes the information flow between the multi-prototype information and the query set features through two shared residual cross-attention modules, and then outputs the calibrated prototype features and the calibrated query features.

4. The semantic segmentation method according to claim 3, wherein: The residual cross attention module obtains attention features and query vector features through three convolution kernels, adopts the attention mechanism to extract the relationship matrix between input features, and uses the query vector to extract channel information.

5. The semantic segmentation method according to claim 4, wherein: The step s4 comprises the following steps: s41. The calibration prototype features and the calibration query features are used as nodes of a graph; s42. The prototype labels of the calibration prototype features are continuously propagated on the graph to obtain the corresponding label values ​​of each node; s43. Using a Gaussian similarity matrix to represent the similarity between each of the nodes, and normalizing it to achieve label propagation; s44. Extract node information corresponding to the query set feature, normalize the node information to obtain a prediction corresponding to each query set feature, and implement semantic segmentation of the query set point cloud.

6. A small sample point cloud semantic segmentation device using the semantic segmentation method according to any one of claims 1 to 5, characterized in that: It includes a sequentially connected feature extractor, a prototype amplification module, a feature calibration module and a label propagation module; The feature extractor extracts features from the support set point cloud and the query set point cloud, and outputs corresponding support set features and query set features; The prototype expansion module combines the support set features and the query set features to obtain expanded multiple prototypes; The feature calibration module exchanges information between the multiple prototypes and the query set features to obtain calibrated prototype features and calibrated query features; The label propagation module measures the distance between the calibration query feature and the calibration prototype feature to perform label propagation, thereby achieving semantic segmentation of the query set point cloud.

7. The semantic segmentation device according to claim 6, wherein: The prototype amplification module includes a first feature aggregation unit, a label assignment unit, a second feature aggregation unit and a combination unit connected in sequence; The first feature aggregation unit performs feature aggregation on the support set features based on the support set mask to obtain multiple prototypes of the support set; The label assignment unit assigns each query set feature a label of the multi-prototype of the support set that is closest to the query set feature based on the Euclidean distance from the query set feature to the multi-prototype of the support set; Get pseudo labels for query set features; The second feature aggregation unit performs feature aggregation on the query set features based on the pseudo labels of the query set to obtain a pseudo prototype; The combining unit combines the multiple prototypes of the support set with the pseudo prototype to obtain an augmented multiple prototype; The feature calibration module realizes the information flow between prototype information and query set features through two shared residual cross-attention modules.

8. A storage medium, characterized in that: The storage medium stores a program file capable of implementing the small sample point cloud semantic segmentation method described in any one of claims 1 to 5.

9. A processor, characterized in that: The processor is used to run a program, wherein the program executes the small sample point cloud semantic segmentation method according to any one of claims 1 to 5 when running.

Images