A few-shot point cloud classification method based on multi-modal pulse fusion neuron

Abstract

The present application relates to a kind of based on multi-modal pulse fusion neuron's few sample point cloud classification method, including to three-dimensional point cloud data is carried out queue sampling time series coding, constructs point cloud time series sequence;Perspective projection is carried out to point cloud time series sequence, generates multi-view depth map and constitutes image mode;Multi-layer pulse perception machine and pulse residual block are constructed;Based on multi-layer pulse perception machine, combined with sampling grouping module and mixed pooling, obtain point cloud mode feature;Based on multi-layer pulse perception machine and pulse residual block, obtain image mode feature;Based on multi-modal pulse fusion neuron obtains multi-modal fusion feature;Based on cosine similarity, obtain enhanced support set feature and enhanced query set feature, and then calculate support prototype vector and obtain few sample point cloud classification result according to its with enhanced query set feature Euclidean distance matrix.The present application compared to existing few sample point cloud classification method, effectively improve the accuracy of point cloud classification, reduce the network's computing energy consumption.

Description

Technical Field

[0001] This invention relates to the field of computer vision 3D point cloud data classification technology, and in particular to a few-sample point cloud classification method based on multimodal spiking fusion neurons. Background Technology

[0002] 3D point clouds, as a high-precision spatial information carrier, can accurately describe the geometric shape, spatial location, and scale characteristics of objects, and have irreplaceable application value in fields such as autonomous driving, industrial inspection, and robot navigation. However, the inherent disorder, sparsity, and massive size of point cloud data pose challenges to feature extraction and semantic understanding. Point cloud classification, as one of the core tasks of point cloud processing, directly affects the reliability of downstream applications, but existing technologies still have many limitations in complex scenarios. Among these, few-sample point cloud classification is a key problem that urgently needs to be solved in practical applications. Traditional point cloud classification methods mostly rely on artificial neural networks (ANNs), requiring large-scale labeled data for fully supervised training. However, in emerging fields (such as new industrial defect detection and rare object recognition), labeled samples are often extremely scarce, leading to a significant decrease in the model's generalization ability.

[0003] In contrast, multimodal networks offer an effective way to improve the classification performance of point clouds with few samples. A single modality is insufficient to fully characterize object attributes, while combining texture and contour information from 2D images allows for cross-modal information complementarity. Existing multimodal methods, such as cross-attention and self-attention mechanisms, fuse 3D point cloud features with 2D image features, improving classification accuracy to some extent. However, these methods are mostly based on ANN architectures, resulting in high computational cost and energy consumption, making them difficult to deploy on resource-constrained embedded devices (such as vehicle terminals and portable detection instruments). Spiking Neural Networks (SNNs), as a novel network model simulating the activity of biological neurons, possess event-driven computational characteristics, performing operations only during pulse signal transmission. This naturally offers advantages in low energy consumption and high efficiency, providing a new approach to solving the aforementioned energy consumption problem. SNNs encode temporal and spatial information through pulse sequences, exhibiting unique advantages in processing dynamic data. However, the application of SNN in point cloud processing is still in the exploratory stage: on the one hand, the unstructured nature of point clouds is difficult to directly adapt to the pulsed processing mode of SNN; on the other hand, existing SNN research focuses on single-modal point cloud processing and lacks a few-sample classification scheme that combines multimodal fusion, thus failing to fully realize the potential of SNN in low-energy scenarios.

[0004] In summary, existing technologies have several limitations: in scenarios with few samples, traditional methods rely on large-scale data and lack generalization ability; multimodal fusion methods, based on ANN architectures, are energy-intensive and difficult to adapt to resource-constrained devices; and there is a technological gap in the application of SNNs in point cloud multimodal few-sample classification, failing to effectively combine multimodal information with the advantages of spiking computation. This invention, by exploring SNN-based multimodal few-sample classification methods, will promote the expansion of SNN applications in multimodal scenarios and help deepen the understanding of multimodal spiking neural networks and their application prospects. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a few-sample point cloud classification method based on multimodal spiking fusion neurons. This method solves the problems of traditional point cloud classification methods relying on large-scale data and having insufficient generalization ability in few-sample scenarios; multimodal fusion methods being limited by artificial neural network (ANN) architecture, resulting in high energy consumption and difficulty in adapting to resource-constrained devices; and the application of spiking neural networks (SNNs) in multimodal few-sample point cloud classification lacking the ability to effectively combine multimodal information with the advantages of spiking computation.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a few-sample point cloud classification method based on multimodal spiking fusion neurons, comprising the following steps: S1. Using 3D point cloud data as the point cloud modality, perform queue-style sampling temporal encoding on the 3D point cloud data to obtain the point set at each time moment and construct the point cloud temporal sequence. S2. For each point set in the point cloud time series, perform perspective projection from each preset observation viewpoint to generate a multi-view depth map corresponding to each point set, thus forming an image modality. S3. Construct a multilayer pulse sensor and a pulse residual block; S4. Construct a point cloud modal feature extraction module; extract features from the point cloud temporal sequence based on a multilayer pulse perceptron, and combine the sampling grouping module and hybrid pooling to construct local neighborhoods and perform pooling fusion to obtain point cloud modal features; S5. Construct an image modal feature extraction module; perform feature extraction based on a multilayer pulse perceptron and pulse residual blocks, and combine max pooling and downsampling to obtain image modal features; S6. Based on the multimodal spiking fusion neuron inherited from the LIF spiking neuron, the point cloud modal features and image modal features are spliced ​​and weighted to obtain multimodal fusion features; S7. Split the multimodal fusion features into support set features and query set features. Apply a cosine similarity-based attention mechanism to the support set features and query set features to generate enhanced support set features and enhanced query set features. Then calculate the support prototype vectors for each category. Obtain the category probability distribution of the query samples based on the Euclidean distance matrix between the enhanced query set features and the support prototype vectors to obtain the category prediction results of the few-sample point cloud.

[0007] Furthermore, in step S1, the specific process includes the following steps: S11. Perform farthest point sampling on the 3D point cloud data, and select... The initial core point set is constructed from 10 core points, and a mask is created to mark the remaining unsampled points. The initial core point set is used as the... A point set of moments; S12. Perform farthest point sampling again on the remaining point set to obtain... The set of newly added sampling points at each moment; S13. Merge the newly added sampling point set with the point set from the previous time step, following the first-in, first-out principle. Remove the earliest sampled points from the previous time step and add the newly added points to form a single point count. The new set of points, which serves as the set of points at the current moment; S14. Remove the points sampled at this moment from the current set of remaining points, update the set of remaining points for the next iteration; S15, For each moment Iteratively execute steps S12-S14 to obtain each time step. The point set constitutes the point cloud temporal sequence.

[0008] Furthermore, in step S2, the specific process includes the following steps: S21. Preset a set of virtual camera parameters and observation angles, and perform perspective projection on the point set at each moment in the point cloud time series from each preset observation angle to render and generate a two-dimensional multi-view depth map corresponding to each point set. S22. The multi-view depth maps corresponding to the point sets at all times are combined into a multi-view, multi-time depth map set, which constitutes an image modality.

[0009] Further, step S3 specifically includes: the multilayer spiking sensor and the spiking residual block are constructed by replacing the activation functions of the multilayer spiking sensor and the residual block with LIF spiking neurons, and the forward propagation process of the LIF spiking neurons is described by the following leak-integral-fire model: ; ; ; in, for Input at any moment; for Intermediate variables of membrane potential at time t; for Membrane potential at time t; for The pulse signal output at any given time; To reset the voltage; The threshold for issuance; The membrane time constant; Let be a step function, defined as: ; The input parameters for the step function are, here, Intermediate variable of membrane potential at time t With distribution threshold The difference.

[0010] Furthermore, in step S4, the specific process includes the following steps: S41. Use a multilayer pulse perceptron to perform initial feature transformation and pulse coding on the input point cloud time sequence to obtain pulse-coded point cloud features; S42. The pulse-coded point cloud features are fed into the sampling grouping module for downsampling and local neighborhood construction to obtain the local neighborhood of the point cloud. The sampling grouping module selects the center point by the farthest point sampling method and finds the neighborhood of each center point by the K-nearest neighbor algorithm, dividing the input pulse-coded point cloud features into several local regions, and finally obtaining the local neighborhood of the point cloud. S43. Feed the local neighborhood of the point cloud into a multilayer pulse perceptron again to extract the local neighborhood features of the point cloud mode. S44. The hybrid pooling process performs max pooling and average pooling on the local neighborhood features of the point cloud, and adds the two pooling results to generate local aggregated features of the point cloud. S45. Finally, a multilayer pulse perceptron is used to extract global features from the local aggregated features of the point cloud to generate point cloud modal features.

[0011] Furthermore, in step S5, the specific process includes the following steps: S51. Use a multilayer pulse perceptron to perform initial feature transformation and pulse coding on the input multi-view depth map to obtain pulse-coded depth features; S52. Perform max pooling and downsampling on the pulse coding depth features to generate pooled depth features; S53. Pooling depth features are further extracted through two pulse residual blocks to obtain image modal features.

[0012] Furthermore, in step S5, the forward propagation process of the multimodal spiking fusion neuron is described by a leak-integration-fire model: S61, Regarding the previous moment membrane potential Accumulate membrane potential to obtain the current Intermediate variable of membrane potential at time t ; S62. The intermediate variable of membrane potential at the current moment. Intermediate variables of membrane potential segmented into point cloud modes and image modes , A splicing-style weighted fusion is performed to obtain the intermediate variable of membrane potential after multimodal weighted fusion. ; S63, using step function Based on the issuance threshold right Perform pulse generation to obtain the pulse signal output at the current moment. ; S64, based on right Perform membrane potential decay and use reset voltage right Perform membrane potential reset to obtain the membrane potential at the current moment. .

[0013] Furthermore, in step S7, the specific process includes the following steps: S71. Split the multimodal fusion features and use the batches with category labels as support set features. Unlabeled batches are used as query set features. ; S72, Regarding support set features Calculate its features with the query set. The cosine similarity matrix between the features is used to generate an attention weight matrix, which is then applied to the query set features. The process is performed to obtain attention-enhanced support features, which are then compared with the support set features. By concatenating the features and passing them through a linear transformation layer and skip connections, we obtain the enhanced support set features. ; S73, Regarding query set features Perform a process similar to step S71 to obtain enhanced query set features. The above process can be expressed as the formula: ; ; ; in This is the cosine similarity matrix; , These are linear transformation layers used in the construction processes of enhancing support set features and enhancing query set features, respectively. Indicates a splicing operation; S74. Average the enhanced support set features for each category to generate support prototype vectors for each category; S75. Calculate the Euclidean distance matrix between the enhanced query set features and the support prototype vectors of each category. Input the negative value of the distance matrix into the Softmax function to obtain the category probability distribution of the query sample belonging to each category.

[0014] Furthermore, the present invention also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a few-sample point cloud classification method based on multimodal spiking fusion neurons.

[0015] Furthermore, the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a few-sample point cloud classification method based on multimodal spiking fusion neurons.

[0016] By employing the above technical solution, the present invention provides a few-sample point cloud classification method based on multimodal spiking fusion neurons, which has at least the following beneficial effects: (1) The present invention uses a spiking neural network (SNN) instead of a traditional artificial neural network (ANN); the SNN processes information through a process of pulse firing and membrane potential integration based on biological mechanisms, and has the characteristics of event-driven and sparse activation, which significantly reduces the computation and memory access overhead, is suitable for deployment in embedded and edge devices, and realizes low-power and high-performance point cloud processing; (2) This invention converts static point clouds into point cloud temporal sequences with temporal relationships by designing queue-style sampling temporal coding, effectively preserving the spatial structure and temporal evolution information of the original point cloud and enhancing the model's ability to model local geometric relationships; (3) The present invention generates a two-dimensional multi-view depth map by multi-view projection, providing complementary image modal information and enriching feature representation; (4) By designing a multimodal pulse fusion neuron, this invention realizes feature splicing and adaptive weighted fusion of point cloud modality and image modality during pulse firing, fully combining multimodal information and the energy consumption advantage of pulsed computation, enhancing the consistency and discriminability of cross-modal features, effectively improving the classification robustness in low-sample scenarios, and significantly reducing the additional computational overhead caused by multimodal information. (5) This invention enhances the feature interaction between the support set and the query set by introducing an attention mechanism based on cosine similarity, optimizes the category prototype generation process, and further improves the generalization ability of the model in the case of few samples. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a few-sample point cloud classification method based on multimodal spiking fusion neurons according to the present invention; Figure 2 This is a schematic diagram of the forward propagation process of the multimodal spiking fusion neuron of the present invention; Figure 3 A framework diagram of a few-sample point cloud classification model constructed for embodiments of the present invention; Figure 4 This is a feature distribution map extracted from the PointNet model, a traditional single-modal artificial neural network, on the ModelNet40 dataset. Figure 5 This is a feature distribution map obtained on the ModelNet40 dataset using the few-shot classification method proposed in this embodiment of the invention. Detailed Implementation

[0018] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0019] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0020] Please refer to Figures 1-5 This illustration demonstrates a specific implementation of this embodiment. This embodiment constructs a temporal sequence of point clouds from 3D point cloud data, generates image modalities through perspective projection, constructs a multilayer pulse perceptron and pulse residual blocks to acquire point cloud modal features and image modal features, and achieves multimodal feature fusion based on multimodal pulse fusion neurons. A cosine similarity-based attention mechanism is introduced to optimize category prototype generation, thus achieving few-shot point cloud classification. Compared to existing few-shot point cloud classification methods, this effectively improves the accuracy of point cloud classification and reduces the computational energy consumption of the network.

[0021] Please refer to Figure 1 This embodiment proposes a few-sample point cloud classification method based on multimodal spiking fusion neurons, which includes the following steps: S1. Using 3D point cloud data as the point cloud modality, perform queue-style sampling temporal encoding on the 3D point cloud data to obtain the point set at each time moment and construct the point cloud temporal sequence. As a preferred embodiment of step S1, the specific process includes the following steps: S11. Receive the input 3D point cloud data, with dimensions of... ,in For batch size, For the number of points in the point cloud, Define the number of point cloud feature channels; define the total number of time steps for queued sampling temporal coding. With the number of core points ; Perform farthest point sampling (FPS) on the 3D point cloud data and select The initial core point set is constructed from 10 core points, and a mask is created to mark the remaining unsampled points. The initial core point set is used as the... A point set of moments; S12. Perform farthest point sampling again on the remaining point set to obtain... The set of newly added sampling points at each moment; S13. Merge the newly added sampling point set with the point set from the previous time step, following the first-in, first-out principle. Remove the earliest sampled points from the previous time step and add the newly added points to form a single point count. The new set of points, which serves as the set of points at the current moment; S14. Remove the points sampled at this moment from the current set of remaining points, update the set of remaining points for the next iteration; S15, For each moment Iteratively execute steps S12-S14 to obtain each time step. The set of points, consisting of dimension The point cloud time series.

[0022] In this embodiment, by designing queue-style sampling temporal coding, static point clouds are converted into point cloud temporal sequences with temporal relationships, which effectively preserves the spatial structure and temporal evolution information of the original point clouds and enhances the model's ability to model local geometric relationships.

[0023] S2. For each point set in the point cloud time series, perform perspective projection from each preset observation viewpoint to generate a multi-view depth map corresponding to each point set, thus forming an image modality. As a preferred embodiment of step S2, the specific process includes the following steps: S21. Preset a set of virtual camera parameters and observation angles, and perform perspective projection on the point set at each moment in the point cloud time series from each preset observation angle to render and generate a two-dimensional multi-view depth map corresponding to each point set. S22. The multi-view depth maps corresponding to the point sets at all times are combined into a multi-view, multi-time depth map set, constituting an image modality; the dimension of the depth map set can be represented as... ,in For the number of viewpoints, These represent the height and width of the depth map, respectively.

[0024] In this embodiment, a two-dimensional multi-view depth map is generated by multi-view projection, providing complementary image modal information and enriching feature representation.

[0025] S3. Construct a multilayer pulse sensor and a pulse residual block; As a preferred embodiment of step S3, it specifically includes: the multilayer spiking sensor and the spiking residual block are constructed by replacing the activation functions of the multilayer spiking sensor and the residual block with LIF spiking neurons, and the forward propagation process of the LIF spiking neurons is described by the following leak-integral-fire model: ; ; ; in, for Input at any moment; for Intermediate variables of membrane potential at time t; for Membrane potential at time t; for The pulse signal output at any given time; To reset the voltage; The threshold for issuance; The membrane time constant; Let be a step function, defined as: ; The input parameters for the step function are, here, Intermediate variable of membrane potential at time t With distribution threshold The difference.

[0026] In this embodiment, a spiking neural network (SNN) is used instead of a traditional artificial neural network (ANN). SNN processes information through a biological mechanism-based pulse firing and membrane potential integration process. It has the characteristics of event-driven and sparse activation, which significantly reduces the computation and memory access overhead. It is suitable for deployment in embedded and edge devices and achieves low-power and high-performance point cloud processing.

[0027] S4. Construct a point cloud modal feature extraction module; extract features from the point cloud temporal sequence based on a multilayer pulse perceptron, and combine the sampling grouping module and hybrid pooling to construct local neighborhoods and perform pooling fusion to obtain point cloud modal features; As a preferred embodiment of step S4, the specific process includes the following steps: S41. Use a multilayer pulse perceptron to perform initial feature transformation and pulse coding on the input point cloud time sequence to obtain pulse-coded point cloud features; S42. The pulse-coded point cloud features are fed into the sampling grouping module for downsampling and local neighborhood construction to obtain the local neighborhood of the point cloud. The sampling grouping module selects center points using the farthest point sampling method (FPS) and searches for the neighborhood of each center point using the K-nearest neighbor algorithm (KNN), dividing the input pulse-coded point cloud features into several local regions, ultimately obtaining the local neighborhood of the point cloud. The dimension of the obtained local neighborhood of the point cloud is... ,in The number of center points in the farthest point sampling method. The number of nearest neighbors of the center point in the K-nearest neighbor algorithm; S43. Feed the local neighborhood of the point cloud into a multilayer pulse perceptron again to extract the local neighborhood features of the point cloud mode. S44. The hybrid pooling process performs max pooling and average pooling on the local neighborhood features of the point cloud, and adds the two pooling results to generate local aggregated features of the point cloud. S45. Finally, a multilayer pulse perceptron is used to extract global features from the local aggregated features of the point cloud to generate point cloud modal features.

[0028] S5. Construct an image modal feature extraction module; perform feature extraction based on a multilayer pulse perceptron and pulse residual blocks, and combine max pooling and downsampling to obtain image modal features; As a preferred embodiment of step S5, the specific process includes the following steps: S51. Use a multilayer pulse perceptron to perform initial feature transformation and pulse coding on the input multi-view depth map to obtain pulse-coded depth features; S52. Perform max pooling and downsampling on the pulse coding depth features to generate pooled depth features; S53. Pooling depth features are further extracted through two pulse residual blocks to obtain image modal features.

[0029] S6. Based on the multimodal spiking fusion neuron inherited from the LIF spiking neuron, the point cloud modal features and image modal features are spliced ​​and weighted to obtain multimodal fusion features; As a preferred embodiment of step S6, it specifically includes: the forward propagation process of the multimodal spiking fusion neuron is described by the following leak-integral-fire model: ; ; in, , , These are intermediate variables of membrane potential, namely, time-point cloud mode, image mode, and multimodal weighted fusion. This indicates the number of point cloud points in a point cloud mode; and These represent the height and width of the multi-view depth map, respectively. and These are the learnable parameters for the point cloud mode and the image mode, respectively; Indicates will Intermediate variable of membrane potential at time t exist Dimensional execution scale is The segmentation operation, Indicates will and exist Perform a concatenation operation on the dimensions. This indicates that the operation is performed on the last dimension.

[0030] The forward propagation process of the aforementioned multimodal spiking fusion neurons can be referred to Figure 2 ,Depend on membrane potential at time 1 Accumulation of membrane potential generates Intermediate variable of membrane potential at time t And thus to Intermediate variables of temporal point cloud modal membrane and image modal membrane potential , By performing weighted fusion, we obtain Intermediate variable of membrane potential after time-mode multimodal weighted fusion Then, through pulse generation, membrane potential reset, and membrane potential decay operations, the following is obtained: membrane potential at time 1 .

[0031] In this embodiment, by designing a multimodal pulse fusion neuron, feature splicing and adaptive weighted fusion of point cloud modality and image modality are realized during pulse firing. This fully combines multimodal information with the energy consumption advantage of pulsed computation, enhances the consistency and discriminativeness of cross-modal features, and effectively improves the classification robustness in low-sample scenarios.

[0032] S7. Split the multimodal fusion features into support set features and query set features. Apply a cosine similarity-based attention mechanism to the support set features and query set features to generate enhanced support set features and enhanced query set features. Then calculate the support prototype vectors for each category. Obtain the category probability distribution of the query samples based on the Euclidean distance matrix between the enhanced query set features and the support prototype vectors to obtain the category prediction results of the few-sample point cloud.

[0033] As a preferred embodiment of step S7, the specific process includes the following steps: S71. Split the multimodal fusion features and use the batches with category labels as support set features. Unlabeled batches are used as query set features. ; S72, Regarding support set features Calculate its features with the query set. The cosine similarity matrix between the features is used to generate an attention weight matrix, which is then applied to the query set features. The process is performed to obtain attention-enhanced support features, which are then compared with the support set features. By concatenating the features and passing them through a linear transformation layer and skip connections, we obtain the enhanced support set features. ; S73, Regarding query set features Perform a process similar to step S71 to obtain enhanced query set features. The above process can be expressed as the formula: ; ; ; in This is the cosine similarity matrix; , These are linear transformation layers used in the construction processes of enhancing support set features and enhancing query set features, respectively. Indicates a splicing operation; S74. Average the enhanced support set features for each category to generate support prototype vectors for each category; S75. Calculate the Euclidean distance matrix between the enhanced query set features and the support prototype vectors of each category. Input the negative value of the distance matrix into the Softmax function to obtain the category probability distribution of the query sample belonging to each category.

[0034] In this embodiment, by introducing an attention mechanism based on cosine similarity, the feature interaction between the support set and the query set is enhanced, the category prototype generation process is optimized, and the generalization ability of the model in the case of few samples is further improved.

[0035] The framework diagram of the few-shot point cloud classification model constructed by the above algorithm can be found in the following figure. Figure 3 Based on the above algorithm, compared with traditional single-modal artificial neural network methods, the visualization results of the few-sample point cloud classification feature distribution of the ModelNet40 dataset are obtained: The feature distribution extracted from the PointNet model, a traditional single-modal artificial neural network, on the ModelNet40 dataset can be found by referring to... Figure 4 It can be seen that there is significant overlap and confusion between different categories of features, and the boundary discrimination is limited, indicating that the PointNet model is insufficient in feature discrimination.

[0036] The feature distribution obtained using the few-shot classification method proposed in this embodiment on the ModelNet40 dataset can be referenced. Figure 5 It can be seen that similar features exhibit high aggregation, while dissimilar features show significantly improved separation, forming clearer clusters.

[0037] Figure 4 and Figure 5 The comparison shows that this embodiment effectively enhances the discriminative ability of feature representation through pulse timing coding and multimodal fusion mechanism, significantly reduces cross-class feature confusion, and thus provides a more robust feature foundation for few-shot point cloud classification tasks, achieving accurate classification of point cloud data under few-shot classification tasks.

[0038] This application also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a few-sample point cloud classification method based on multimodal spiking fusion neurons.

[0039] This application also provides a computer-readable storage medium storing computer instructions that enable a processor to implement a few-sample point cloud classification method based on multimodal spiking fusion neurons.

[0040] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0041] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0042] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A few-shot point cloud classification method based on a multi-modal pulse fusion neuron, characterized in that, Includes the following steps: S1. Using 3D point cloud data as the point cloud modality, perform queue-style sampling temporal encoding on the 3D point cloud data to obtain the point set at each time moment and construct the point cloud temporal sequence. S2. For each point set in the point cloud time series, perform perspective projection from each preset observation viewpoint to generate a multi-view depth map corresponding to each point set, thus forming an image modality. S3. Construct a multilayer pulse sensor and a pulse residual block; S4. Construct a point cloud modal feature extraction module; Feature extraction of point cloud temporal sequences is performed based on a multilayer pulse perceptron, and local neighborhood construction and pooling fusion are combined with a sampling grouping module to obtain point cloud modal features; S5. Construct an image modal feature extraction module; Feature extraction is performed based on a multilayer pulse perceptron and pulse residual blocks, and image modal features are obtained by combining max pooling and downsampling. S6. Based on the multimodal spiking fusion neuron inherited from the LIF spiking neuron, the point cloud modal features and image modal features are spliced ​​and weighted to obtain multimodal fusion features; S7. Split the multimodal fusion features into support set features and query set features. Apply a cosine similarity-based attention mechanism to the support set features and query set features to generate enhanced support set features and enhanced query set features. Then calculate the support prototype vectors for each category. Obtain the category probability distribution of the query samples based on the Euclidean distance matrix between the enhanced query set features and the support prototype vectors to obtain the category prediction results of the few-sample point cloud.

2. The method for few-shot point cloud classification based on multimodal spiking fusion neurons according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11. Perform farthest point sampling on the 3D point cloud data, and select... The initial core point set is constructed from 10 core points, and a mask is created to mark the remaining unsampled points. The initial core point set is used as the... A point set of moments; S12. Perform farthest point sampling again on the remaining point set to obtain... The set of newly added sampling points at each moment; S13. Merge the newly added sampling point set with the point set from the previous time step, following the first-in, first-out principle. Remove the earliest sampled points from the previous time step and add the newly added points to form a single point count. The new set of points, which serves as the set of points at the current moment; S14. Remove the points sampled at this moment from the current set of remaining points, update the set of remaining points for the next iteration; S15, For each moment Iteratively execute steps S12-S14 to obtain each time step. The point set constitutes the point cloud temporal sequence.

3. The method for few-shot point cloud classification based on multimodal spiking fusion neurons according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21. Preset a set of virtual camera parameters and observation angles, and perform perspective projection on the point set at each moment in the point cloud time series from each preset observation angle to render and generate a two-dimensional multi-view depth map corresponding to each point set. S22. The multi-view depth maps corresponding to the point sets at all times are combined into a multi-view, multi-time depth map set, which constitutes an image modality.

4. The few-shot point cloud classification method based on multimodal spiking fusion neurons according to claim 1, characterized in that: Step S3 specifically includes: the multilayer spiking sensor and spiking residual block are constructed by replacing the activation functions of the multilayer spiking sensor and residual block with LIF spiking neurons. The forward propagation process of the LIF spiking neurons is described by the following leak-integral-fire model: ； ； ； in, for Input at any moment; for Intermediate variables of membrane potential at time t; for Membrane potential at time t; for The pulse signal output at any given time; To reset the voltage; The threshold for issuance; The membrane time constant; Let be a step function, defined as: ； The input parameters for the step function are, here, Intermediate variable of membrane potential at time t With distribution threshold The difference.

5. The few-shot point cloud classification method based on multimodal spiking fusion neurons according to claim 4, characterized in that: Step S4 specifically includes the following steps: S41. Use a multilayer pulse perceptron to perform initial feature transformation and pulse coding on the input point cloud time sequence to obtain pulse-coded point cloud features; S42. The pulse-coded point cloud features are fed into the sampling grouping module for downsampling and local neighborhood construction to obtain the local neighborhood of the point cloud. The sampling grouping module selects the center point by the farthest point sampling method and finds the neighborhood of each center point by the K-nearest neighbor algorithm, dividing the input pulse-coded point cloud features into several local regions, and finally obtaining the local neighborhood of the point cloud. S43. Feed the local neighborhood of the point cloud into a multilayer pulse perceptron again to extract the local neighborhood features of the point cloud mode. S44. The hybrid pooling process performs max pooling and average pooling on the local neighborhood features of the point cloud, and adds the two pooling results to generate local aggregated features of the point cloud. S45. Finally, a multilayer pulse perceptron is used to extract global features from the local aggregated features of the point cloud to generate point cloud modal features.

6. The few-shot point cloud classification method based on multimodal spiking fusion neurons according to claim 4, characterized in that: Step S5 specifically includes the following steps: S51. Use a multilayer pulse perceptron to perform initial feature transformation and pulse coding on the input multi-view depth map to obtain pulse-coded depth features; S52. Perform max pooling and downsampling on the pulse coding depth features to generate pooled depth features; S53. Pooling depth features are further extracted through two pulse residual blocks to obtain image modal features.

7. The few-shot point cloud classification method based on multimodal spiking fusion neurons according to claim 4, characterized in that: Step S6 specifically includes: the forward propagation process of the multimodal spiking fusion neuron is described by the leak integration-fire model: S61, Regarding the previous moment membrane potential Accumulate membrane potential to obtain the current Intermediate variable of membrane potential at time t ; S62. The intermediate variable of membrane potential at the current moment. Intermediate variables of membrane potential segmented into point cloud modes and image modes , A splicing-style weighted fusion is performed to obtain the intermediate variable of membrane potential after multimodal weighted fusion. ; S63, using step function Based on the issuance threshold right Perform pulse generation to obtain the pulse signal output at the current moment. ; S64, based on right Perform membrane potential decay and use reset voltage right Perform membrane potential reset to obtain the membrane potential at the current moment. .

8. The method for few-shot point cloud classification based on multimodal spiking fusion neurons according to claim 1, characterized in that: Step S7 specifically includes the following steps: S71. Split the multimodal fusion features and use the batches with category labels as support set features. Unlabeled batches are used as query set features. ; S72, Regarding support set features Calculate its features with the query set. The cosine similarity matrix between the features is used to generate an attention weight matrix, which is then applied to the query set features. The process is performed to obtain attention-enhanced support features, which are then compared with the support set features. By concatenating the features and passing them through a linear transformation layer and skip connections, we obtain the enhanced support set features. ; S73, Regarding query set features Perform a process similar to step S71 to obtain enhanced query set features. The above process can be expressed as the formula: ； ； ； in This is the cosine similarity matrix; , These are linear transformation layers used in the construction processes of enhancing support set features and enhancing query set features, respectively. Indicates a splicing operation; S74. Average the enhanced support set features for each category to generate support prototype vectors for each category; S75. Calculate the Euclidean distance matrix between the enhanced query set features and the support prototype vectors of each category. Input the negative value of the distance matrix into the Softmax function to obtain the category probability distribution of the query sample belonging to each category.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the few-sample point cloud classification method based on multimodal spiking fusion neurons as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the few-sample point cloud classification method based on multimodal spiking fusion neurons as described in any one of claims 1-8.

Images