Radar position identification method and system based on pulse neural network space-time fusion

Abstract

The invention discloses a radar position identification method based on spiking neural network space-time fusion, and the method comprises the steps: obtaining a preset number of point cloud data of a radar for a same to-be-detected region, and converting the point cloud data into a depth map; according to the spiking neural network space-time fusion-based radar position identification method and system, a plurality of point cloud data scanned by a radar in an area to be detected are used as identifiers of spatial positions, a special spiking neural network for radar position identification is constructed, and the spiking neural network is used for extracting and fusing space-time features; the local features are aggregated by means of a NetVLAD model, and finally a global unique feature descriptor containing spatial-temporal dynamic features is generated, so that efficient and accurate position matching and recognition are realized; on the basis of a spiking neural network architecture, the efficient and accurate position matching and recognition effect is achieved, and meanwhile, compared with a traditional artificial neural network architecture, the method has less energy consumption.

Description

Technical Field

[0001] The present invention belongs to the technical field of position recognition, and in particular relates to a radar position recognition method and system based on pulse neural network spatiotemporal fusion. Background Art

[0002] Location recognition, the process by which a robot or intelligent agent determines its position on a known map by observing its current environment, is a fundamental aspect of autonomous navigation for mobile robots. This capability is crucial not only for the simultaneous localization and mapping (SLAM) process but also for efficient global positioning. Location recognition plays a vital role in various fields, including mobile robotics, environmental perception, intelligent driving, and autonomous navigation. Currently, location recognition methods are primarily categorized into vision-based and lidar (LiDAR)-based methods. While vision-based methods are widely documented, their reliability in large outdoor environments is significantly compromised due to the camera's sensitivity to lighting changes and seasonal variations. In contrast, LiDAR-based location recognition methods exhibit greater robustness to variations in lighting, weather, and viewing angle, making them a focus of recent research.

[0003] However, real-world navigation environments are often extremely complex, with a large number of dynamic objects and unavoidable environmental similarities. In addition, the original point cloud usually contains significant noise and exhibits sparsity and inconsistency in appearance. Relying solely on single-frame data for location recognition is usually inaccurate and lacks robustness. Therefore, some studies have proposed sequence matching methods using continuous LiDAR scan frames and have achieved certain results. Although these methods have broad prospects, they are mainly based on artificial neural network (ANN) architectures, and therefore it is difficult to avoid their inherent limitations, especially in terms of computational efficiency. This directly leads to high energy consumption and insufficient real-time response, which seriously hinders the practical deployment and widespread application of these algorithms.

[0004] In contrast, spiking neural networks (SNNs), implemented on brain-inspired computing hardware, have shown great potential for improving computational efficiency, offering a more effective alternative to ANNs. As the next generation of neural networks, SNNs are renowned for their exceptional bio-fidelity, asynchronous event-driven nature, and low energy consumption. Compared to ANNs, which rely on floating-point computations, SNNs transmit information via 0-1 spike signals between neurons, exhibiting more complex spatiotemporal dynamics. In tasks such as image classification and gesture recognition, SNNs can achieve performance comparable to ANNs, paving the way for their application in radar-based location recognition (LPR). Furthermore, the inherent spatiotemporal properties of SNNs make them naturally suitable for sequence matching tasks with similar properties.

[0005] In the field of LiDAR-based location recognition, most existing research relies on feature extraction from point clouds or depth maps. Point cloud-based methods directly process raw point clouds or voxelized 3D data to extract global descriptors. For example, the early work PointNet-VLAD proposed a framework for extracting global descriptors from point clouds. Subsequent works such as LPD-Net combined with graph neural networks to improve global feature representation by aggregating multiple local features from point clouds. SegMap divides point clouds into segments and processes them via convolutional neural networks to generate global features. MinkLoc3D uses a sparse voxel representation combined with convolution operations to extract robust global descriptors from point clouds. While these methods perform well in feature extraction, their high computational requirements limit their application in real-time scenarios. Recent research has increasingly adopted image representations of LiDAR data to improve efficiency. For example, OverlapNet uses depth maps to estimate the overlap and relative yaw angle between query and reference scans. OverlapTransformer extends this approach by introducing a yaw-invariant Transformer architecture, enhancing feature robustness. However, these methods all rely on artificial neural networks, resulting in low computational efficiency. In summary, the existing location recognition technology has many shortcomings, and there is an urgent need for a new, efficient and accurate SNNs-based LiDAR location recognition solution to meet the needs of autonomous systems for location recognition tasks in complex environments. Summary of the Invention

[0006] The purpose of the present invention is to solve the problems raised in the background technology and to propose a radar position recognition method and system based on pulse neural network spatiotemporal fusion.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] The present invention proposes a radar position recognition method based on spatiotemporal fusion of a spiking neural network, comprising:

[0009] Obtain a preset number of point cloud data from the radar for the same area to be measured, and convert each point cloud data into a depth map;

[0010] Pack all depth maps into a preset shape and input them into the trained spiking neural network;

[0011] In the spiking neural network, the spatiotemporal features of each depth map are extracted, and the spatiotemporal features are fused for a second time to obtain a comprehensive feature without a time dimension.

[0012] Aggregate the local features corresponding to each spatiotemporal feature in the comprehensive features without time dimension to obtain the descriptor of the area to be measured, and use the descriptor as the output of the spiking neural network;

[0013] The descriptor is compared with the descriptors in the preset database for similarity, and the descriptor with the closest similarity in the database is used as the position representation of the area to be measured.

[0014] Preferably, a preset number of point cloud data of the same area to be measured is obtained by the radar through the sensor, wherein the point cloud data includes multiple three-dimensional points, and each three-dimensional point is projected onto the depth map, and the formula is as follows: u= 1 2 [1-arctan(y,x) π -1 ]w ; v=[1-(arcsin(z r -1 )+ f up ) f -1 ]h ;

[0015] in, ; ;

[0016] in, Indicates the value of the horizontal axis on the depth map, Indicates the value of the vertical coordinate on the depth map, and Represents the width and height of the depth map, 、 and They represent the horizontal, vertical and vertical coordinates of the three-dimensional point respectively. Represents the distance from the sensor to each 3D point in the point cloud data, Indicates the vertical field of view of the sensor upwards, Indicates the vertical field of view of the sensor downward, Indicates the vertical field of view of the sensor.

[0017] Preferably, all depth maps of the preset shape are The matrix of Indicates the number of point cloud data obtained by the radar for the same area to be measured. represents the batch dimension, 、 and Represent the channel dimension, height and width of the depth map respectively.

[0018] Preferably, the spiking neural network comprises a downsampling module, a feature extraction module, a time domain aggregation module and a descriptor generation module connected in sequence;

[0019] The downsampling module includes a pulse space compression unit and a pulse channel expansion unit connected in sequence, and the pulse space compression unit is used to compress the height dimension of the depth map, and the pulse channel expansion unit is used to expand the channel dimension of the depth map. All depth maps of the preset shape are used as inputs of the pulse space compression unit, and the output of the pulse space compression unit is first fused with the output of the pulse channel expansion unit to obtain a fusion feature;

[0020] The feature extraction module includes a pulse-based self-attention mechanism unit and a pulse-based multi-layer perceptron unit connected in sequence, the fused feature is used as the input of the pulse-based self-attention mechanism unit, and the output of the pulse-based self-attention mechanism unit is first fused with the output of the pulse-based multi-layer perceptron unit to obtain the spatiotemporal features of each depth map;

[0021] The spatiotemporal features of each depth map are used as the input of the time domain aggregation module to obtain comprehensive features without time dimension;

[0022] The descriptor generation module is a NetVLAD model, and the comprehensive features without time dimension are used as input of the descriptor generation module to obtain the descriptor of the area to be measured.

[0023] Preferably, the spatiotemporal features of each depth map are represented by a sequence: , Indicates the area to be tested The spatiotemporal features of the depth map corresponding to the point cloud data, that is, The spatiotemporal characteristics of dimensions;

[0024] The time domain aggregation module performs a second fusion of the spatiotemporal features of each depth map to obtain a comprehensive feature without a time dimension. The calculation formula is as follows: ;

[0025] Among them, for each : ;

[0026] in, Represents comprehensive features without time dimension, Indicates the number of custom learnable parameters, Indicates the learnable parameters, Indicates the The weight of the dimensional spatiotemporal features, Indicates the The weights of the dimensional spatiotemporal features are softmaxed to ensure that the sum of all time dimension weights is 1.

[0027] Preferably, during the training process, the spiking neural network converts all collected point cloud data into depth maps, uses each depth map as a query sample, and finds positive and negative samples of the query sample from the remaining depth maps, and the specific formula is as follows: ;

[0028] in, Indicates the current query sample With another depth map The overlap percentage is the ratio of pixels in one depth map that are successfully projected to another depth map without occlusion. represents the overlap percentage threshold, Represents a query sample In its own coordinate frame, Represents another depth map In the query sample Representation in coordinate frame, Indicates the number of valid pixels in the current query sample, Represents another depth map The number of effective pixels in When it is true, ,otherwise , To find the minimum value;

[0029] When the overlap percentage is greater than , then the current query sample and the other depth map are mutually positive samples, otherwise they are mutually negative samples.

[0030] Preferably, during the training process of the spiking neural network, all query samples and corresponding positive and negative samples are packaged in a preset shape and input into the spiking neural network for training, thereby obtaining a query sample descriptor corresponding to each query sample, as well as a corresponding positive sample descriptor and a negative sample descriptor;

[0031] The corresponding query sample descriptor, positive sample descriptor and negative sample descriptor are calculated through the triple loss function to train the pulse neural network, and the formula of the triple loss function is as follows: ;

[0032] in, ; ;

[0033] in, represents the triplet loss function, Indicates the current query sample descriptor, represents the set of positive sample descriptors of the current query sample descriptor, represents the set of negative sample descriptors of the current query sample descriptor, For the preset parameters, Indicates the number of positive samples corresponding to the current query sample, Indicates the number of negative samples corresponding to the current query sample, represents the mean of the squared Euclidean distances between the current query sample descriptor and each positive sample descriptor in the positive sample descriptor set, Represents the mean of the squared Euclidean distances between the current query sample descriptor and each negative sample descriptor in the set of negative sample descriptors.

[0034] A radar position recognition system based on spatiotemporal fusion of pulse neural networks includes a preprocessing module, a packaging module, and a comparison module, wherein:

[0035] The pre-processing module is used to obtain a preset number of point cloud data of the same area to be measured by the radar, and convert each point cloud data into a depth map;

[0036] The packaging module is used to package all depth maps into a preset shape and input them into the trained spiking neural network;

[0037] In the spiking neural network, the spatiotemporal features of each depth map are extracted, and the spatiotemporal features are fused for a second time to obtain a comprehensive feature without a time dimension.

[0038] Aggregate the local features corresponding to each spatiotemporal feature in the comprehensive features without time dimension to obtain the descriptor of the area to be measured, and use the descriptor as the output of the spiking neural network;

[0039] The comparison module compares the descriptor with each descriptor in a preset database for similarity, and uses the descriptor with the closest similarity in the database as the position representation of the area to be measured.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This radar location recognition method and system based on spatiotemporal fusion of spiking neural networks uses multiple point cloud data from radar scans of the area to be measured as spatial location identifiers. A spiking neural network specifically designed for radar location recognition is constructed. The spiking neural network extracts and fuses spatiotemporal features, aggregates local features with the help of the NetVLAD model, and ultimately generates a globally unique feature descriptor that incorporates spatiotemporal dynamic characteristics, achieving efficient and accurate location matching and recognition.

[0042] 2. This method introduces temporal dynamics, and the spiking neural network can effectively capture complex spatial relationships, significantly improving its adaptability and practical value in changing real-world scenarios. At the same time, this method abandons the limitation of relying solely on single-frame radar scan data for position matching, and solves the problem of environmental changes affecting position recognition accuracy in existing technologies.

[0043] 3. This method is based on the pulse neural network architecture, which achieves efficient and accurate position matching and recognition effects while consuming less energy than the traditional artificial neural network architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the process of radar position recognition method and system based on spatiotemporal fusion of spiking neural network according to the present invention;

[0045] Figure 2 Schematic diagram of the structure of the pulse neural network of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] In one embodiment, Figure 1-Figure 2 As shown, a radar position recognition method based on pulse neural network spatiotemporal fusion is provided.

[0048] Step 1: Obtain a preset number of point cloud data from the radar for the same area to be measured, and convert each point cloud data into a depth map. The specific process includes:

[0049] In this embodiment, a preset number of point cloud data of the same area to be measured is obtained by the LiDAR sensor (the radar scans the same area to be measured);

[0050] The point cloud data contains multiple 3D points. Each 3D point is projected onto the depth map, and the formula is as follows: u= 1 2 [1-arctan(y,x) π -1 ]w ; v=[1-(arcsin(z r -1 )+ f up ) f -1 ]h ;

[0051] in, ; ;

[0052] in, Indicates the value of the horizontal axis on the depth map, Indicates the value of the vertical coordinate on the depth map, and Respectively represent the width and height of the depth map (the width and height of the depth map are customized), 、 and They represent the horizontal, vertical and vertical coordinates of the three-dimensional point respectively. Represents the distance from the sensor to each 3D point in the point cloud data, Indicates the vertical field of view of the sensor upwards, Indicates the vertical field of view of the sensor downward, Indicates the vertical field of view of the sensor.

[0053] Step 2: Pack all depth maps into a preset shape and input them into the trained spiking neural network;

[0054] Among them, all depth maps of preset shapes are The matrix of Indicates the number of point cloud data obtained by the radar for the same area to be measured (the same as the time dimension of the pulse neural network coincide), represents the batch dimension (here 1), 、 and Represent the channel dimension, height and width of the depth map respectively.

[0055] The spiking neural network includes a downsampling module, a feature extraction module, a time-domain aggregation module, and a descriptor generation module connected in sequence;

[0056] The downsampling module includes a pulse space compression unit and a pulse channel expansion unit connected in sequence, and the pulse space compression unit is used to compress the height dimension of the depth map (to adapt to the input size requirements of the subsequent modules. This unit keeps the channel dimension unchanged while compressing the height dimension, ensuring that the spatial information is effectively concentrated and providing a unified input size for subsequent processing), and the pulse channel expansion unit is used to expand the channel dimension of the depth map (to compensate for the information that may be lost in the pulse space compression unit and further enhance the feature expression capability. Through the synergistic effect of the pulse space compression unit and the pulse channel expansion unit, the downsampling module fully retains and enhances the feature expression capability while reducing the spatial resolution of the feature map, providing high-quality input for subsequent modules), all depth maps of the preset shape are used as input to the pulse space compression unit, and the output of the pulse space compression unit is first fused with the output of the pulse channel expansion unit (using feature addition) to obtain a fused feature;

[0057] The feature extraction module includes a sequentially connected spike-based self-attention mechanism unit (the spike-based self-attention mechanism unit belongs to an existing structure, such as the existing literature Yao M, Hu J, Hu T, et al. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the designof next-generation neuromorphic chips[J]. arXiv preprint arXiv:2404.03663,2024.) and a spike-based multi-layer perceptron unit (the difference from the ordinary multi-layer perceptron is that the original artificial neurons are replaced by spike neurons). The fused features are used as the input of the spike-based self-attention mechanism unit, and the output of the spike-based self-attention mechanism unit is first fused with the output of the spike-based multi-layer perceptron unit (using feature addition) to obtain the spatiotemporal features of each depth map;

[0058] The spatiotemporal features of each depth map are used as the input of the time domain aggregation module to obtain comprehensive features without time dimension;

[0059] The descriptor generation module is a NetVLAD model, and the comprehensive features without time dimension are used as the input of the descriptor generation module to obtain the descriptor of the area to be tested.

[0060] Step 3: In the spiking neural network, the spatiotemporal features of each depth map are extracted (i.e., the output of the spiking self-attention mechanism unit is first fused with the output of the spiking multi-layer perceptron unit to obtain the spatiotemporal features of each depth map), and the spatiotemporal features are secondarily fused to obtain a comprehensive feature without a time dimension.

[0061] Among them, the spatiotemporal features of each depth map are expressed as a sequence: , Indicates the area to be tested The spatiotemporal features of the depth map corresponding to the point cloud data, that is, the spatiotemporal features of the th dimension;

[0062] The time domain aggregation module performs a second fusion of the spatiotemporal features of each depth map to obtain a comprehensive feature without a time dimension. The calculation formula is as follows: ;

[0063] Among them, for each (i.e., for each Weights of dimensional spatiotemporal features ): ;

[0064] in, Represents comprehensive features without time dimension, Indicates the number of custom learnable parameters (i.e. the number of learnable heads), Indicates the learnable parameters, Indicates the The weight of the dimensional spatiotemporal features, Indicates the The weights of the dimensional spatiotemporal features are softmaxed to ensure that the sum of all time dimension weights is 1.

[0065] It should be noted that during the second fusion process, first, initialize Each head captures the changes of features along the time dimension from a different perspective; next, a learnable weight is assigned to each time step (i.e. ), and through The operation automatically adjusts its importance to capture the dynamic feature changes in the time domain; then the features of each time step are weighted and accumulated to obtain the global temporal feature representation; finally, The outputs of each head are accumulated and fused to capture time series changes from multiple angles, enhancing the model's ability to capture complex spatiotemporal relationships.

[0066] Step 4: Aggregate the local features corresponding to each spatiotemporal feature in the time-free comprehensive features to obtain a descriptor of the area to be measured (i.e., the time-free comprehensive features are used as the input of the descriptor generation module. The descriptor generation module aggregates the local features corresponding to each spatiotemporal feature in the time-free comprehensive features to obtain a descriptor of the area to be measured. This descriptor, as the only representation of the position of the area to be measured, can accurately reflect the differences between different positions in space. The descriptor generation module efficiently encodes the local features corresponding to each spatiotemporal feature in the time-free comprehensive features and converts them into global features to generate a globally unique descriptor that includes spatiotemporal dynamic characteristics), and this descriptor is used as the output of the spiking neural network.

[0067] Step 5: Compare the descriptor with the descriptors in the preset database for similarity (e.g., using normalized Euclidean distance to calculate similarity), and use the descriptor with the closest similarity in the database as the position representation of the area to be measured;

[0068] It should be noted that, if the method is applied to a certain city, the database will contain descriptors of all locations in the city, wherein the descriptors in the database can be obtained by training a spiking neural network.

[0069] In this embodiment, during the training process, the spiking neural network converts all collected point cloud data into depth maps, uses each depth map as a query sample, and finds positive and negative samples of the query sample from the remaining depth maps. The specific formula is as follows: ;

[0070] in, Indicates the current query sample With another depth map The overlap percentage is the ratio of pixels in one depth map that are successfully projected to another depth map without occlusion. represents the overlap percentage threshold, Represents a query sample In its own coordinate frame, Represents another depth map In the query sample Representation in coordinate frame, Indicates the number of valid pixels in the current query sample, Represents another depth map The number of effective pixels in When it is true, ,otherwise , To find the minimum value;

[0071] When the overlap percentage is greater than (30% in this embodiment), the current query sample and another depth map are mutually positive samples, otherwise they are mutually negative samples, where the positive sample has the same spatial position as the query sample, and the negative sample has the same spatial position as the query sample, and the number of positive samples and negative samples of each query sample is more than one.

[0072] In this embodiment, during the training process of the spiking neural network, all query samples and corresponding positive and negative samples are packaged in a preset shape and input into the spiking neural network for training (for example, in this embodiment, the number of collected point cloud data is ', then all query samples and corresponding positive and negative samples are packaged according to the preset shape to be , and the batch dimension is The query sample descriptor, the positive sample descriptor and the negative sample descriptor corresponding to each query sample are obtained, and the query sample descriptor, the positive sample descriptor and the negative sample descriptor are stored in a database;

[0073] The corresponding query sample descriptor, positive sample descriptor and negative sample descriptor are calculated through the triple loss function to train the pulse neural network, and the formula of the triple loss function is as follows: ;

[0074] in, ; ;

[0075] in, represents the triplet loss function, Indicates the current query sample descriptor, represents the set of positive sample descriptors of the current query sample descriptor, represents the set of negative sample descriptors of the current query sample descriptor, is the preset parameter (interval hyperparameter), Indicates the number of positive samples corresponding to the current query sample, Indicates the number of negative samples corresponding to the current query sample, represents the mean of the squared Euclidean distances between the current query sample descriptor and each positive sample descriptor in the positive sample descriptor set, Represents the mean of the squared Euclidean distances between the current query sample descriptor and each negative sample descriptor in the set of negative sample descriptors.

[0076] The pulse neural network is supervised by the triplet loss function, which makes the feature representation of the query sample closer to that of the positive sample, while significantly distinguishing the feature representation of the query sample from that of the negative sample, thereby improving the recognition discrimination at different positions.

[0077] In another embodiment, based on a radar position recognition method based on spiking neural network spatiotemporal fusion, a radar position recognition system based on spiking neural network spatiotemporal fusion is proposed, including a preprocessing module, a packaging module, and a comparison module, wherein:

[0078] The pre-processing module is used to obtain a preset number of point cloud data of the radar for the same area to be measured and convert each point cloud data into a depth map;

[0079] The packaging module is used to package all depth maps into a preset shape and input them into the trained spiking neural network;

[0080] In the spiking neural network, the spatiotemporal features of each depth map are extracted, and the spatiotemporal features are fused for a second time to obtain a comprehensive feature without a time dimension.

[0081] Aggregate the local features corresponding to each spatiotemporal feature in the comprehensive features without time dimension to obtain the descriptor of the area to be measured, and use the descriptor as the output of the spiking neural network;

[0082] The comparison module compares the similarity between the descriptor and each descriptor in the preset database, and uses the descriptor with the closest similarity in the database as the position representation of the area to be tested.

[0083] Regarding the specific limitations of the radar position recognition system based on the spatiotemporal fusion of spiking neural networks, please refer to the limitations of the radar position recognition method based on the spatiotemporal fusion of spiking neural networks above, which will not be repeated here. The various modules in the above-mentioned radar position recognition system based on the spatiotemporal fusion of spiking neural networks can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0084] The following experimental data compares the radar location recognition method based on pulse neural network spatiotemporal fusion with the radar location recognition technology in the existing technology:

[0085] Compared with existing technologies, this method has significantly improved robustness and adaptability in complex environments, enabling it to achieve higher recall rates in dynamic and changing real-world scenarios. Table 1 shows the average TOP1 recall rate comparison results of this radar location recognition method based on spatiotemporal fusion of spiking neural networks and existing technologies on the NCLT dataset (location recognition dataset):

[0086] Table 1

[0087] Among them, Comparative Example 1: Uy M A, Lee G H. Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4470-4479.

[0088] Comparative Example 2: Komorowski J. Minkloc3d: Point cloud based large-scale place recognition[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2021: 1790-1799.

[0089] Comparative Example 3: Ma J, Zhang J, Xu J, et al. Overlap transformer: An efficient and yaw-angle-invariant transformer network for lidar-based place recognition[J]. IEEE Robotics and Automation Letters, 2022, 7(3): 6958-6965.

[0090] Comparative Example 4: Milford M J, Wyeth G F. SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights[C] / / 2012 IEEE international conference on robotics and automation. IEEE, 2012: 1643-1649.

[0091] Comparative Example 5: Vysotska O, Stachniss C. Relocalization under substantial appearance changes using hashing[C] / / Proceedings of the IROS Workshop on Planning, Perception and Navigation for Intelligent Vehicles, Vancouver, BC, Canada. 2017, 24.

[0092] Comparative Example 6: Facil J M, Olid D, Montesano L, et al. Condition-invariant multi-view place recognition[J]. arXiv preprint arXiv:1902.09516, 2019.

[0093] Comparative Example 7: Garg S, Milford M. Seqnet: Learning descriptors for sequence-based hierarchical place recognition[J]. IEEE Robotics and Automation Letters, 2021, 6(3): 4305-4312.

[0094] Comparative Example 8: Liu Z, Suo C, Zhou S, et al. Seqlpd: Sequence matching enhanced loop-closure detection based on large-scale point cloud description for self-driving vehicles[C] / / 2019 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019: 1218-1223.

[0095] This method outperforms existing methods in real-world loop closure detection tasks. By leveraging the spatiotemporal characteristics of spiking neural networks, this method can effectively capture key information in the environment and improve the accuracy of loop closure detection. Table 2 shows the comparison of AUC, F1max, TOP1 recall, and TOP1% recall between this method and existing technologies on the KITTI dataset (loop closure detection dataset).

[0096] Table 2

[0097] Among them, Comparative Example 9: Röhling T, Mack J, Schulz D. A fast histogram-based similarity measure for detecting loop closures in 3-d lidar data[C] / / 2015IEEE / RSJ international conference on intelligent robots and systems (IROS).IEEE, 2015: 736-741.

[0098] Comparative Example 10: Kim G, Kim A. Scan context: Egocentric spatial descriptor for place recognition within 3d point cloud map[C] / / 2018 IEEE / RSJInternational Conference on Intelligent Robots and Systems (IROS). IEEE,2018: 4802-4809.

[0099] Comparative Example 11: Wang Y, Sun Z, Xu CZ, et al. Lidar iris for loop-closure detection[C] / / 2020 IEEE / RSJ International Conference on Intelligent Robotsand Systems (IROS). IEEE, 2020: 5769-5775.

[0100] Comparative Example 12: Uy M A, Lee G H. Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4470-4479.

[0101] Comparative Example 13: Chen X, Läbe T, Milioto A, et al. OverlapNet: Loop closing for LiDAR-based SLAM[J]. arXiv preprint arXiv:2105.11344, 2021.

[0102] Comparative Example 14: Zhou Z, Zhao C, Adolfsson D, et al. Ndt-transformer: Large-scale 3d point cloud localisation using the normal distribution transform representation[C] / / 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 5654-5660. <�

[0103] Comparative Example 15: Komorowski J. Minkloc3d: Point cloud based large-scale place recognition[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2021: 1790-1799.

[0104] It should be noted that there seems to be a typo in the original text where <�

[0103] is likely meant to be

[0103] . This has been left as is in the translation to maintain consistency with the original text.This method demonstrates excellent generalization in real-world loop closure detection tasks: Spiking neural networks based on 0-1 spikes exhibit strong generalization due to their sparse activation, spike-driven mechanism, and nonlinear dynamics. We trained on the KITTI dataset (a loop closure detection dataset) and tested it on the Ford Campus dataset without fine-tuning. Table 3 shows the generalization performance of our method compared to existing techniques on the Ford Campus dataset.

[0105] Table 3

[0106] Among them, Comparative Example 16: Röhling T, Mack J, Schulz D. A fast histogram-based similarity measure for detecting loop closures in 3-d lidar data[C] / / 2015IEEE / RSJ international conference on intelligent robots and systems (IROS).IEEE, 2015: 736-741.

[0107] Comparative Example 17: Kim G, Kim A. Scan context: Egocentric spatial descriptor for place recognition within 3d point cloud map[C] / / 2018 IEEE / RSJInternational Conference on Intelligent Robots and Systems (IROS). IEEE,2018: 4802-4809.

[0108] Comparative Example 18: Wang Y, Sun Z, Xu CZ, et al. Lidar iris for loop-closure detection[C] / / 2020 IEEE / RSJ International Conference on Intelligent Robotsand Systems (IROS). IEEE, 2020: 5769-5775.

[0109] Comparative Example 19: Uy M A, Lee G H. Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4470-4479.

[0110] Comparative Example 20: Chen X, Läbe T, Milioto A, et al. OverlapNet: Loop closing for LiDAR-based SLAM[J]. arXiv preprint arXiv:2105.11344, 2021.

[0111] Comparative Example 21: Zhou Z, Zhao C, Adolfsson D, et al. Ndt-transformer: Large-scale 3d point cloud localisation using the normal distribution transform representation[C] / / 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 5654-5660.

[0112] Comparative Example 22: Komorowski J. Minkloc3d: Point cloud based large-scale place recognition[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2021: 1790-1799.

[0113] This method demonstrates excellent real-time processing capabilities in real-world location recognition and closed-loop detection tasks: Based on the efficient computing architecture of spiking neural networks, this method can simultaneously process multiple frames of radar scan point cloud data while maintaining a computing speed comparable to that of an artificial neural network processing a single frame of data. Furthermore, the feature descriptors generated by this method are highly compact, significantly reducing the complexity of the retrieval process, thereby further improving the efficiency of feature matching and retrieval, and providing strong support for real-time applications in real-world scenarios. Table 4 shows the time it takes to generate feature descriptors and retrieve databases for this method compared to existing technologies on the KITTI dataset:

[0114] Table 4

[0115] Among them, Comparative Example 23: Röhling T, Mack J, Schulz D. A fast histogram-based similarity measure for detecting loop closures in 3-d lidar data[C] / / 2015IEEE / RSJ international conference on intelligent robots and systems (IROS).IEEE, 2015: 736-741.

[0116] Comparative Example 24: Kim G, Kim A. Scan context: Egocentric spatial descriptor for place recognition within 3d point cloud map[C] / / 2018 IEEE / RSJInternational Conference on Intelligent Robots and Systems (IROS). IEEE,2018: 4802-4809.

[0117] Comparative Example 25: Wang Y, Sun Z, Xu CZ, et al. Lidar iris for loop-closure detection[C] / / 2020 IEEE / RSJ International Conference on Intelligent Robotsand Systems (IROS). IEEE, 2020: 5769-5775.

[0118] Comparative Example 26: Uy M A, Lee G H. Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4470-4479.

[0119] Comparative Example 27: Chen X, Läbe T, Milioto A, et al. OverlapNet: Loop closing for LiDAR-based SLAM[J]. arXiv preprint arXiv:2105.11344, 2021.

[0120] Comparative Example 28: Zhou Z, Zhao C, Adolfsson D, et al. Ndt-transformer: Large-scale 3d point cloud localisation using the normal distribution transform representation[C] / / 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 5654-5660.

[0121] Comparative Example 29: Komorowski J. Minkloc3d: Point cloud based large-scale place recognition[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2021: 1790-1799.

[0122] This method demonstrates excellent low-energy consumption in real-world location recognition and closed-loop detection tasks: Thanks to the event-driven and asynchronous computing architecture of the spiking neural network, the model achieves extremely low energy consumption while maintaining high accuracy and real-time performance. Compared to existing artificial neural network-based solutions, its energy consumption is significantly reduced, making it more environmentally adaptable and practically deployable. Table 5 shows the energy consumption of this method and existing technologies on the NCLT dataset:

[0123] Table 5

[0124] Among them, Comparative Example 30: Uy MA, Lee G H. Pointnetvlad: Deep point cloud basedretrieval for large-scale place recognition[C] / / Proceedings of the IEEEconference on computer vision and pattern recognition. 2018: 4470-4479.

[0125] Comparative Example 31: Zhou Z, Zhao C, Adolfsson D, et al. Ndt-transformer: Large-scale 3d point cloud localization using the normal distribution transformrepresentation[C] / / 2021 IEEE International Conference on Robotics andAutomation (ICRA). IEEE, 2021: 5654-5660.

[0126] Comparative Example 32: Ma J, Zhang J, Xu J, et al. Overlaptransformer: An efficientand yaw-angle-invariant transformer network for lidar-based recognition place recognition[J]. IEEE Robotics and Automation Letters, 2022, 7(3): 6958-6965.

[0127] This radar location recognition method and system based on spatiotemporal fusion of spiking neural networks uses multiple point cloud data from radar scans of the target area as spatial location identifiers to construct a specialized spiking neural network for radar location recognition. The spiking neural network extracts and fuses spatiotemporal features and aggregates local features with the help of the NetVLAD model, ultimately generating a globally unique feature descriptor that incorporates spatiotemporal dynamics. This method achieves efficient and accurate location matching and recognition. By incorporating temporal dynamics, the spiking neural network effectively captures complex spatial relationships, significantly improving its adaptability and practical value in changing real-world scenarios. Furthermore, this method eliminates the limitation of relying solely on single-frame radar scan data for location matching, addressing the impact of environmental changes on location recognition accuracy in existing technologies. Furthermore, based on the event-driven mechanism and discrete pulse transfer characteristics of spiking neural networks, this method achieves comparable or even superior performance to existing technologies while significantly reducing energy consumption. Existing radar location recognition models, primarily based on artificial neural networks, suffer from low computational efficiency, resulting in high energy consumption and insufficient real-time responsiveness, which severely restricts the deployment and promotion of such algorithms in real-world scenarios. This method significantly alleviates this issue and has significant application value.

[0128] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0129] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A radar position recognition method based on spatiotemporal fusion of pulse neural networks, characterized by: The radar position recognition method based on pulse neural network spatiotemporal fusion includes: Obtain a preset number of point cloud data from the radar for the same area to be measured, and convert each point cloud data into a depth map; Pack all depth maps into a preset shape and input them into the trained spiking neural network; In the spiking neural network, the spatiotemporal features of each depth map are extracted, and the spatiotemporal features are fused for a second time to obtain a comprehensive feature without a time dimension. Aggregate the local features corresponding to each spatiotemporal feature in the comprehensive features without time dimension to obtain the descriptor of the area to be measured, and use the descriptor as the output of the spiking neural network; The descriptor is compared with the descriptors in the preset database for similarity, and the descriptor with the closest similarity in the database is used as the position representation of the area to be measured.

2. The radar position recognition method based on spatiotemporal fusion of spiking neural networks according to claim 1, characterized in that: The sensor obtains a preset number of point cloud data of the radar for the same area to be measured. The point cloud data contains multiple three-dimensional points. Each three-dimensional point is projected onto the depth map, and the formula is as follows: ； ； in, ； ； in, Indicates the value of the horizontal axis on the depth map, Indicates the value of the vertical coordinate on the depth map, and Represents the width and height of the depth map respectively, 、 and They represent the horizontal, vertical and vertical coordinates of the three-dimensional point respectively. Represents the distance from the sensor to each 3D point in the point cloud data, Indicates the vertical field of view angle of the sensor upwards, Indicates the vertical field of view of the sensor downward, Indicates the vertical field of view of the sensor.

3. The radar position recognition method based on spatiotemporal fusion of spiking neural networks according to claim 1, characterized in that: All depth maps of the preset shape are The matrix of Indicates the number of point cloud data obtained by the radar for the same area to be measured. represents the batch dimension, 、 and Represent the channel dimension, height and width of the depth map respectively.

4. The radar position recognition method based on spatiotemporal fusion of spiking neural networks according to claim 1, characterized in that: The spiking neural network includes a downsampling module, a feature extraction module, a time domain aggregation module and a descriptor generation module connected in sequence; The downsampling module includes a pulse space compression unit and a pulse channel expansion unit connected in sequence, and the pulse space compression unit is used to compress the height dimension of the depth map, and the pulse channel expansion unit is used to expand the channel dimension of the depth map. All depth maps of the preset shape are used as inputs of the pulse space compression unit, and the output of the pulse space compression unit is first fused with the output of the pulse channel expansion unit to obtain a fusion feature; The feature extraction module includes a pulse-based self-attention mechanism unit and a pulse-based multi-layer perceptron unit connected in sequence, the fused feature is used as the input of the pulse-based self-attention mechanism unit, and the output of the pulse-based self-attention mechanism unit is first fused with the output of the pulse-based multi-layer perceptron unit to obtain the spatiotemporal features of each depth map; The spatiotemporal features of each depth map are used as the input of the time domain aggregation module to obtain comprehensive features without time dimension; The descriptor generation module is a NetVLAD model, and the comprehensive features without time dimension are used as input of the descriptor generation module to obtain the descriptor of the area to be measured.

5. The radar position recognition method based on spatiotemporal fusion of spiking neural networks according to claim 4, characterized in that: The spatiotemporal features of each depth map are expressed as a sequence: , Indicates the area to be tested The spatiotemporal features of the depth map corresponding to the point cloud data, that is, The spatiotemporal characteristics of dimensions; The time domain aggregation module performs a second fusion of the spatiotemporal features of each depth map to obtain a comprehensive feature without a time dimension. The calculation formula is as follows: ； Among them, for each : ； in, Represents comprehensive features without time dimension, Indicates the number of custom learnable parameters, Indicates the learnable parameters, Indicates the The weight of the dimensional spatiotemporal features, Indicates the The weights of the dimensional spatiotemporal features are subjected to softmax operation to ensure that the sum of the weights of all time dimensions is 1.

6. The radar position recognition method based on spatiotemporal fusion of spiking neural networks according to claim 1, characterized in that: During the training process, the spiking neural network converts all collected point cloud data into depth maps, uses each depth map as a query sample, and finds positive and negative samples of the query sample from the remaining depth maps. The specific formula is as follows: ； in, Indicates the current query sample With another depth map The overlap percentage is the ratio of pixels in one depth map that are successfully projected to another depth map without occlusion. represents the overlap percentage threshold, Represents a query sample Represented in its own coordinate frame, Represents another depth map In the query sample Representation in coordinate frame, Indicates the number of valid pixels in the current query sample, Represents another depth map The number of effective pixels in When it is true, ,otherwise , To find the minimum value; When the overlap percentage is greater than , then the current query sample and the other depth map are mutually positive samples, otherwise they are mutually negative samples.

7. The radar position recognition method based on spatiotemporal fusion of spiking neural networks according to claim 1, characterized in that: During the training process of the spiking neural network, all query samples and corresponding positive and negative samples are packaged in a preset shape and input into the spiking neural network for training, thereby obtaining a query sample descriptor corresponding to each query sample, as well as a corresponding positive sample descriptor and a negative sample descriptor; The corresponding query sample descriptor, positive sample descriptor and negative sample descriptor are calculated through the triple loss function to train the pulse neural network, and the formula of the triple loss function is as follows: ； in, ； ； in, represents the triplet loss function, Indicates the current query sample descriptor, represents the set of positive sample descriptors of the current query sample descriptor, represents the set of negative sample descriptors of the current query sample descriptor, For the preset parameters, Indicates the number of positive samples corresponding to the current query sample, Indicates the number of negative samples corresponding to the current query sample, represents the mean of the squared Euclidean distances between the current query sample descriptor and each positive sample descriptor in the positive sample descriptor set, Represents the mean of the squared Euclidean distances between the current query sample descriptor and each negative sample descriptor in the set of negative sample descriptors.

8. A radar position recognition system based on pulse neural network spatiotemporal fusion, characterized by: The radar position recognition system based on pulse neural network spatiotemporal fusion includes a preprocessing module, a packaging module and a comparison module, wherein: The pre-processing module is used to obtain a preset number of point cloud data of the same area to be measured by the radar, and convert each point cloud data into a depth map; The packaging module is used to package all depth maps into a preset shape and input them into the trained spiking neural network; In the spiking neural network, the spatiotemporal features of each depth map are extracted, and the spatiotemporal features are fused for a second time to obtain a comprehensive feature without a time dimension. Aggregate the local features corresponding to each spatiotemporal feature in the comprehensive features without time dimension to obtain the descriptor of the area to be measured, and use the descriptor as the output of the spiking neural network; The comparison module compares the descriptor with each descriptor in a preset database for similarity, and uses the descriptor with the closest similarity in the database as the position representation of the area to be measured.

Images