A robot low-power edge visual positioning system and method based on pulse neural network

Abstract

The application relates to a robot low-power edge visual positioning system and method based on a pulse neural network. The system comprises an event camera, a pulse coding module, an SNN feature extraction layer, a multi-sensor pulse fusion module, a pulse Kalman filtering module and a neuromorphic hardware; and an asynchronous control logic dynamically adjusts coding parameters through a feedback bus. The method comprises the following steps: event streams are coded into a spatiotemporal pulse sequence; visual features are extracted by the SNN; IMU and laser radar pulses are fused; and a 6-DOF pose is output through pulse Kalman filtering. The application adopts event-driven and sparse pulse calculation, combines a dynamic reconfiguration mechanism, realizes sub-watt power consumption, millisecond-level delay and centimeter-level positioning accuracy, and has adaptive energy efficiency and anti-congestion capability. Compared with a traditional visual SLAM, the power consumption is reduced by more than 30 times, the delay is reduced by 3-6 times, and the application is suitable for resource-limited edge autonomous navigation scenes such as micro unmanned aerial vehicles and long-term monitoring robots.

Description

Technical Field

[0001] This invention relates to the fields of robot autonomous navigation and computer vision technology, specifically to a low-power edge vision localization system and method for robots based on a spiking neural network (SNN), which is particularly suitable for resource-constrained mobile platforms such as micro unmanned aerial vehicles, long-term monitoring robots, and wearable auxiliary devices. Background Technology

[0002] Visual Simultaneous Localization and Mapping (VSLAM) is one of the core technologies for autonomous navigation of mobile robots. Traditional visual SLAM systems, such as ORB-SLAM, DSO, and SVO, typically employ frame-based image processing methods: CMOS or CCD image sensors output complete image frames at a fixed frame rate (e.g., 30Hz, 60Hz), and then localization and mapping are achieved through steps such as feature extraction (e.g., ORB, SIFT, SURF), feature matching, pose optimization, and loop closure detection. These methods can achieve high accuracy on desktop or embedded GPUs (such as the NVIDIA Jetson series), but they have the following inherent limitations: 1. High computational power requirements and high power consumption: Image frames contain a large amount of redundant information (such as static backgrounds and areas with constant lighting). Traditional algorithms need to process all pixels in each frame, resulting in a high computational load. To achieve real-time performance (≥30Hz), GPU acceleration or a high-performance CPU is usually required, with system power consumption typically between 10W and 50W. For resource-constrained platforms such as micro-drones, long-term monitoring robots, and wearable devices, such high power consumption makes it difficult to meet battery life requirements.

[0003] 2. High latency: Frame-based processing is limited by fixed exposure time and transmission bandwidth. The end-to-end latency from event occurrence to output pose is often more than 30ms, making it difficult to handle high-speed motion scenes or control tasks that require fast response.

[0004] 3. Redundant data processing: When the lighting changes slowly or the robot is stationary, the information overlap between consecutive frames is extremely high. Traditional methods still process the entire frame, resulting in unnecessary energy waste.

[0005] In recent years, event cameras, as a novel type of visual sensor, have attracted attention due to their high temporal resolution (microsecond level), low data redundancy, and high dynamic range. Event cameras output pixel-level brightness change events asynchronously, rather than complete frame images. However, most existing visual localization methods based on event cameras employ traditional artificial neural networks (ANNs) or hybrid architectures, still requiring the reconstruction of event streams into pseudo-frames or event voxel grids before feeding them into CNNs for processing. Essentially, they have not escaped the predicament of high computational overhead.

[0006] Spiking Neural Networks (SNNs), hailed as the third generation of neural networks, transmit information through discrete spikes (Spikes) in their neurons, possessing inherent event-driven characteristics, spatiotemporal information processing capabilities, and low-power potential. When deployed on neuromorphic hardware (such as asynchronous circuits implemented in Intel Loihi, IBM TrueNorth, SpiNNaker, or FPGAs), SNNs can achieve sub-milliwatt to milliwatt-level power consumption. However, current SNN research primarily focuses on pattern recognition tasks such as image classification and object detection. Applying SNNs to robot vision localization and mapping systems, especially achieving complete low-power edge vision localization, still faces the following technical challenges: How to efficiently encode asynchronous event streams from event cameras into spatiotemporal pulse patterns that can be processed by SNNs; how to design deep SNN structures to extract spatiotemporal features that can be used for pose estimation from sparse pulse sequences; how to achieve the fusion of multiple sensors (IMU, LiDAR) and visual pulses to improve localization robustness and accuracy; how to perform state estimation in SLAM (such as Kalman filtering) in an event-driven manner on neuromorphic hardware; and how to train localization networks containing pulse operations end-to-end to overcome the problem of non-differentiability of pulse firing functions.

[0007] Therefore, there is an urgent need for a robot edge vision localization system and method that can fully utilize the sparsity of event cameras, the low power consumption of SNNs, and the advantages of neuromorphic hardware event-driven systems to solve the problems of high power consumption, high latency, and high resource consumption in existing technologies. Summary of the Invention

[0008] This invention designs a low-power edge vision localization system and method for robots based on spiking neural networks. The technical problem it solves is that traditional visual SLAM relies on high-performance GPUs, resulting in high power consumption and large latency, making it difficult to run on resource-constrained mobile robots for extended periods. At the same time, it solves key technical challenges in combining event cameras with spiking neural networks, such as pulse coding, multi-sensor fusion, neuromorphic hardware adaptation, and network training.

[0009] To solve the aforementioned technical problems, the present invention adopts the following solution: A low-power edge vision localization system for robots based on spiking neural networks, comprising: An event camera sensor is used to output an asynchronous, pixel-level brightness change event stream; The pulse coding module, connected to the event camera, is used to encode the event stream into a spatiotemporal pulse pattern; The spiking neural network processing module, deployed on neuromorphic hardware, is used to receive the pulse sequence output by the pulse coding module and perform visual feature extraction and pose estimation. The multi-sensor pulse fusion module is electrically connected to the pulse neural network processing module, the pulse output terminal of the IMU, and the pulse output terminal of the lidar. It performs weighted superposition and fusion on the three pulse sequences and outputs a fused pulse sequence. The pulse Kalman filter module, connected to the multi-sensor pulse fusion module, is used to perform state estimation on the fused pulse sequence and output a 6-DOF pose. The asynchronous control logic, located within the neuromorphic hardware, has its signal acquisition terminal connected to the output terminal of the pulse code module or the status monitoring point of the spiking neural network processing module. Its output terminal is connected to the parameter configuration register of the pulse code module via a feedback control bus, which is used to dynamically adjust the encoding parameters of the pulse code module.

[0010] Preferably, the encoding parameters of the pulse coding module include at least one of the following: time window length, spatial aggregation scale, and pulse threshold; the encoding parameters are used to control the conversion accuracy and density of event flow to spatiotemporal pulse patterns.

[0011] Preferably, the asynchronous control logic includes: a pulse density statistics unit, a comparator for comparing with a preset threshold, and a finite state machine for outputting reconfiguration commands; the pulse density statistics unit collects pulse sequence density data and transmits it to the comparator; the comparator compares the measured data with the preset threshold and then sends a control command to the finite state machine; the finite state machine outputs a reconfiguration command.

[0012] Preferably, when the asynchronous control logic detects that the pulse density exceeds the first threshold, it prioritizes increasing the spatial aggregation scale and then decreases the time window length; when the pulse density is lower than the second threshold, it prioritizes decreasing the spatial aggregation scale and then increases the time window length; the first threshold is 1500-2000 events / ms, and the second threshold is 50-80 events / ms.

[0013] Preferably, the spiking neural network processing module includes: a spiking convolutional layer, a leaky integral ignition (LIF) neuron layer, a spiking pooling layer, and an output layer; the output layer contains 6 neuron populations, each population consisting of multiple LIF neurons, the 6 neuron populations corresponding one-to-one with the robot's three-dimensional position components and three-dimensional pose components, and the population firing rate encoding each component of the 6-DOF pose.

[0014] Preferably, the multi-sensor pulse fusion module adopts a weighted superposition fusion rule: , in, S v (t), S i (t), S l (t) These are pulse sequences for vision, IMU, and LiDAR, respectively. W v 、W i 、W l For the preset weighting coefficients, satisfy W v +W i +W l =1; The threshold function is defined as follows: if the independent variable is ≥1.5, output 1; otherwise, output 0.

[0015] Preferably, the pulse Kalman filter module is deployed in the neuronal core array of neuromorphic hardware; its prediction step is mapped to the neuronal core array in the following manner: the state neuron population encodes the posterior state of the previous time step with firing rate, the firing rate is linearly transformed and accumulated by the synaptic matrix array to form a prior estimated firing rate on the prediction neuron population, wherein the synaptic matrix array pre-stores the synaptic weights of the state transition matrix; each element of the state transition matrix is ​​stored as a 4-8 bit signed integer in an independent storage unit of the synaptic matrix array, the firing rate of each component of the state vector is used as the presynaptic input, and after being weighted and accumulated by the synaptic weights, it is accumulated on the membrane potential of the prediction neuron population to form a prior estimated firing rate; its update step is mapped to the neuronal core array in the following manner: fusion pulse sequence The sequence is converted into observation residuals by the decoding unit. The observation residuals are the difference between the pulse count of the fused pulse sequence within the time window and the physical quantity corresponding to the predicted firing rate. The residuals are converted into pulse current signals by the event generator. The event generator converts the difference into pulse current signals with a frequency proportional to the absolute value of the residuals. The pulse current signals are injected into the updated neuron population in the form of asynchronous event packets and superimposed with the predicted membrane potential to complete the state correction. Finally, the firing rate of the updated neuron population is converted into a 6-DOF pose by the pose decoding output layer. The prediction and update process of the pulse Kalman filter module is completed asynchronously in the neuron core array without a global clock synchronization signal. Each neuron core only performs membrane potential accumulation and threshold comparison when a pulse event arrives or is triggered, and remains dormant when idle.

[0016] Preferably, the neuromorphic hardware includes: an asynchronous event-driven circuit, an address-event representation routing network, a distributed neuronal core array, and a synaptic memory; the address-event representation routing network connects the asynchronous event-driven circuit and the distributed neuronal core array, and the synaptic memory is independently configured for each neuronal core, used to store 4-8 bit signed synaptic weights and call them in real time.

[0017] A low-power edge vision localization method for robots based on spiking neural networks includes the following steps: Step 1: Use the event camera to capture the event stream of scene brightness changes; Step 2: Encode the event stream into a spatiotemporal pulse sequence; Step 3: Input the pulse sequence into the spiking neural network deployed on neuromorphic hardware to extract visual spatiotemporal features; Step 4: Estimate the robot's 6-DOF pose using the output layer of the spiking neural network; Step 5: Using the event camera encoding time window as a unified timing reference, synchronously encode the sensor data of IMU and LiDAR into pulse sequences, and fuse them with the visual pulse sequences; Step 6: Perform pulse Kalman filtering on the fused pulse sequence to obtain a smoothed pose estimate; Step 7: The asynchronous control logic monitors the density or sparsity of the pulse sequence in real time at millisecond intervals and dynamically adjusts the encoding parameters in the encoding step.

[0018] Preferably, the training of the spiking neural network adopts a supervised learning algorithm based on surrogate gradient, the surrogate gradient function is the fast sigmoid function with a slope parameter of 1; the loss function includes pose estimation error, spiking rate sparsity constraint and energy consumption regularization term, wherein the target firing rate of the sparsity constraint is 5% and the energy regularization coefficient is 0.001; after training, the synaptic weights are quantized into 4-8 bit integers and loaded into the neuromorphic hardware for inference.

[0019] This low-power edge vision localization system for robots based on spiking neural networks has the following advantages: (1) This invention adopts a sparse pulse computing paradigm that combines an event camera with a spiking neural network. It is processed only by an asynchronous event-driven circuit when an event occurs, without a global clock. The static power consumption is only about 8mW, and the dynamic power consumption is about 0.3W at a positioning frequency of 100Hz. The total power consumption of the system can be as low as 0.5W. Compared with traditional GPU-based visual SLAM solutions, which typically consume 15W–50W, the power consumption is reduced by more than 30 times, which can significantly extend the battery life of micro drones, long-term monitoring robots, and wearable devices.

[0020] (2) In this invention, the event camera outputs an asynchronous event stream, and the SNN processes each event in units of pulses, without waiting for the accumulation of complete frames. The end-to-end positioning delay can be controlled within 8–10ms. Compared with frame-based methods, which are typically ≥30ms, the delay is reduced by 3–6 times, meeting the real-time requirements of high-speed motion scenarios.

[0021] (3) This invention extracts spatiotemporal pulse features through deep SNN, combines multi-sensor pulse fusion of IMU and lidar with pulse Kalman filtering, and the average position RMSE of this system can reach 0.045m and the angle RMSE can reach 0.02rad in an indoor 10m×10m environment, which is comparable to or even better than the mainstream traditional SLAM accuracy.

[0022] (4) The unique dynamic reconfiguration mechanism of this invention—asynchronous control logic—monitors pulse density and neuron load in real time and dynamically adjusts the time window, spatial aggregation scale, and pulse threshold of the pulse coding module. In low-dynamic scenarios, it automatically reduces pulse redundancy, further saving power consumption; in high-dynamic or strong light change scenarios, it instantly increases the aggregation scale to prevent hardware saturation caused by event congestion, expands the effective event rate of the system to 10–5000 events / ms, and improves the robustness and scene adaptability of the system. Attached Figure Description

[0023] Figure 1 The overall architecture diagram of the robot low-power edge vision localization system based on spiking neural network of this invention; Figure 2 : Flowchart of the neuromorphic hardware architecture and event-driven processing of this invention; Figure 3 : Schematic diagram of the computational mapping principle of the pulse Kalman filter module in the neuron core array in this invention. Detailed Implementation

[0024] The following is combined Figures 1 to 3 The present invention will be further described below.

[0025] Example 1

[0026] like Figure 1 As shown, the low-power edge vision localization system for robots based on spiking neural networks provided by this invention includes the following core modules: an event camera sensor that outputs an asynchronous pixel-level brightness change event stream; a pulse coding module that converts the event stream into a spatiotemporal pulse sequence; an SNN feature extraction layer, consisting of a pulse convolutional layer, a pulse pooling layer, an optional pulse LSTM layer, and an output layer, deployed on neuromorphic hardware; a multi-sensor pulse fusion module that receives visual pulse sequences, IMU pulse sequences, and LiDAR pulse sequences, and outputs a fused pulse sequence; and a pulse Kalman filter module that performs state estimation on the fused pulses and outputs a 6-DOF pose.

[0027] The system's data flow is as follows: event camera → pulse coding module → SNN feature extraction layer → multi-sensor pulse fusion module → pulse Kalman filter module → pose output. Simultaneously, data from the IMU and LiDAR are directly fed into the fusion module after being pulse-coded.

[0028] like Figure 2 As shown, the core computing platform of this invention is a neuromorphic hardware based on FPGA. In this embodiment, a Xilinx Zynq UltraScale and a ZU9EG FPGA are selected, and its internal structure includes: The AER interface receives address-event representation AER packets from the event camera, each packet containing pixel coordinates, timestamp, and polarity.

[0029] The AER routing network uses an 8×8 mesh topology and supports multicast and broadcast to route events to the corresponding neuron cores.

[0030] The neuronal core array consists of 64 cores, each containing 256 LIF neurons, for a total of 16,384 neurons. Each neuron has: a 16-bit fixed-point membrane potential register, ranging from 0 to 65,535, calibrated to 0–2.0V; an 8-bit threshold register, defaulting to 1.0; and an 8-bit leakage coefficient register, defaulting to 0.9.

[0031] Synaptic memory, each core of which is equipped with 64KB 8T SRAM, stores synaptic weights with 4-bit precision, is signed, ranges from -8 to 7, and supports read-modify-write operations.

[0032] The asynchronous control logic employs a four-phase handshake protocol and has no global clock. It only wakes up the corresponding core's synaptic computing unit when an event occurs, performs membrane potential accumulation and threshold comparison, and then enters sleep mode. The static power consumption is approximately 8mW, and the dynamic power consumption is approximately 0.3W at a 100Hz positioning frequency.

[0033] The output interface outputs the final 6DOF pose result via the AXI or SPI bus.

[0034] Figure 1 The specific implementation of the pulse code module is as follows: The event camera, model DVS128, with a resolution of 128×128, outputs an asynchronous event stream, where each event is represented as a quadruple (x, y, t, p). The pulse coding module performs the following steps: 1. Time window division: The continuous time axis is divided into windows of fixed length Δt=5ms. Each window corresponds to a discrete time step k, for a total of T=20 time steps, covering 100ms of history.

[0035] 2. Polarity separation: Within each window, positive polarity events (p=+1) and negative polarity events (p=-1) are mapped to two independent pulse channels respectively.

[0036] 3. Spatial Aggregation: Perform 3×3 neighborhood aggregation for each channel: if at least one event exists in the neighborhood, generate a pulse at the center pixel of that neighborhood; otherwise, do not generate one. This operation is equivalent to max pooling, and the output pulse tensor has a dimension of T×128×128×2.

[0037] Figure 1 The feature extraction layers of a mid-SNN include the following levels in sequence: Input layer: Receives pulse tensor S∈{0,1}20×128×128×2.

[0038] Pulsed convolutional layer 1: 3×3 convolutional kernel, stride 1, output channels 32, followed by LIF neurons. The LIF neuron membrane potential update equation is: ,

[0039] V i ( t ) is the first i At time step, one neuron t The membrane potential, the accumulated potential value; β The leakage coefficient (0 < β < 1) controls the rate at which the membrane potential decays over time; V i ( t- 1) is the first i The neurons at the previous time step t- 1 membrane potential; W ij It is the synaptic weight, connecting the first j The input from the first or previous layer neuron to the first... i The connection strength of each neuron; S j ( t ) is the first j An input at time step t The pulse is emitted, with a value of 0 or 1, where 1 indicates that a pulse has arrived; It is all input pulses at time step t For neurons i The total input current, and the postsynaptic potential accumulation; Where the leakage coefficient β = 0.9 and the threshold θ = 1.0. If V i ( t If )≥θ, then a pulse is emitted. S j ( t )=1 and will V i ( t Reset to 0.

[0040] Pulsed pooling layer: 2×2 max pooling, output dimension 20×64×64×32.

[0041] Pulsed convolutional layer 2: 3×3 convolutional kernel, stride 1, output channels 64, followed by LIF neurons (parameters as above), output dimension 20×64×64×64.

[0042] Global average pulse pooling: Calculate the average firing rate of each channel in the spatial dimension, with an output dimension of 20×1×1×64.

[0043] Output layer: Fully connected spiking layer, containing 6 neuron populations, each with 20 LIF neurons. After the time window ends, the average firing rate r of each population is calculated. k Then linearly mapped to the actual pose. pose k： ,

[0044] in, pose k It is the first k Pose estimates for a number of degrees of freedom or dimensions, such as x, y, z position coordinates or roll, pitch, and yaw angles; min k It is the first k The minimum value of each degree of freedom, for example, -5m for the position component and - for the angle component. π rad ; max k It is the first k The maximum value of each degree of freedom, for example, +5m for the position component and + for the angle component. πrad ; r k It is the first k The average firing rate of a group of neurons, with values ​​normalized to [0,1], is obtained by statistically analyzing the pulses of multiple neurons within the group; the position component ranges from [-5m,5m], and the angle component ranges from [-π,π].

[0045] like Figure 1 As shown, the specific implementation of the multi-sensor pulse fusion module and pulse Kalman filter is as follows: The multi-sensor pulse fusion module includes sensor pulse encoding, a pulse fusion layer, and pulse Kalman filtering.

[0046] The sensor pulse codes include IMUBMI160, TF-Luna lidar, and visual pulse sequences.

[0047] IMU (BMI160): Samples acceleration and angular velocity at 5ms intervals. Employs rate coding: Normalizes each axis physical quantity to the [0,1] interval, and then generates pulses with this value as a probability, i.e., the firing rate is proportional to the absolute value of the physical quantity.

[0048] TF-Luna LiDAR: The distance value d is converted to time encoding: pulse delay τ = (d / 8m) × 5ms. That is, within each 5ms window, the LiDAR emits a pulse at a delay of τ.

[0049] Visual pulse sequence: directly taken from the output of pulse convolution layer 2 in the SNN feature extraction layer, downsampled, and pulse counted in each 5ms window as visual feature pulses.

[0050] The pulse fusion layer aligns the pulse sequences of each stream in time, using a 5ms window from the IMU as a baseline. Within each window, the pulse sequence is simplified to pulse counting (visual, IMU) or fixed pulses from the LiDAR. Fusion is performed using a weighted superposition formula: , in, S f ( t ) is the fused pulse sequence, a binary output, with a value of 0 or 1, representing the pulse at time step . t Should a fusion pulse be output? W v These are the weighting coefficients of the visual pulse sequence; W i These are the weighting coefficients of the IMU pulse sequence; W l These are the weighting coefficients of the lidar pulse sequence; S v ( t () is a visual pulse sequence, which comes from the visual feature pulses output by the spiking neural network processing module, and the pulse count or pulse event at each time step; S i ( t ) is an IMU pulse sequence, which encodes physical quantities such as acceleration and angular velocity into a sequence of pulses; S l ( t () is a lidar pulse sequence, which encodes the distance value into a sequence of time-delayed pulses; W v =0.5、 W i =0.3、 W l =0.2, threshold function Θ(x)=1; if x≥1.5 otherwise 0.

[0051] In pulsed Kalman filtering, the prediction and update steps of classical Kalman filtering are rewritten as operations on the firing rate of the neuron population.

[0052] State vector: x=[x,y,z,vx,vy,vz,qw,qx,qy,qz] T , where (x,y,z) is the position, (vx,vy,vz) is the velocity, and the quaternion represents the attitude.

[0053] Prediction step: Implemented by multiplexing a synaptic array using a matrix multiplication engine in the FPGA: , Wherein, the state vector x contains the robot position (x,y,z), velocity (vx,vy,vz) and attitude quaternion (qw,qx,qy,qz); A is the identity matrix plus the angular velocity term, u is the control quantity obtained by integrating the IMU angular velocity in this embodiment, and B is the control input matrix, which maps the external control quantity into the state space. According to as of t- Information at time 1, for t The prior estimate of the state vector at time step; It is based on the date t- Information at moment 1, for t- The posterior estimate of the state vector at time 1, and the optimal estimate from the previous time step. This operation is mapped to a linear transformation of the firing rate of the neuron population.

[0054] Update steps: Fusion pulse S f (t) Decoded into observation location coordinates Z t The observed residuals are converted into a current-injected state neuron population. Kalman gain K t Pre-compute and store as fixed values ​​to simplify implementation and ignore online updates of covariance; final state estimation: , in, Based on all information up to time t, including predicted values ​​and new observations, for t The posterior estimate of the state vector at time step, i.e., the optimal estimate at the current time step; According to as of t- Information at moment 1, for t The prior estimate of the state vector at time step K, i.e., the output of the state prediction step. t The Kalman gain matrix measures the weight between prediction uncertainty and observation uncertainty, determining the degree to which new observations correct the state estimate; in this embodiment, it is pre-calculated and stored as a fixed value; y t The observation residual (news) is defined as the difference between the actual observed value and the predicted observed value. Z t The values ​​are actual observations, obtained by decoding the fused pulse sequence, where H is the observation matrix. The output is a 6-DOF pose.

[0055] like Figure 3 As shown, the pulsed Kalman filter module is deployed in the neuronal core array of neuromorphic hardware and is divided into two stages: prediction and update.

[0056] See Figure 3 In the left column, the prediction phase: the state neuron population encodes the posterior state of the previous time step using the firing rate. This firing rate is processed by a synaptic matrix array, in which the synaptic weights of the pre-stored state transition matrix are accumulated through a linear transformation to form a prior estimate of the firing rate on the prediction neuron population. Each element of the state transition matrix is ​​independently stored in the synaptic memory of each neuron core in the form of quantized weights. The firing rate of the state components serves as the presynaptic input, and the prediction result is obtained through weighted accumulation.

[0057] See Figure 3 The right column, update phase: fused pulse sequence S f (t) The signal is converted into observation residual by the decoding unit. This residual is the difference between the pulse count of the fused pulse within the time window and the physical quantity corresponding to the predicted firing rate. The residual signal is converted into a pulse current signal with a frequency proportional to the absolute value of the residual by the event generator. It is injected into the updated neuron population in the form of asynchronous event packets and superimposed with the predicted membrane potential to complete the state correction.

[0058] Finally, the firing rate of the updated neuron population is converted into a 6-DOF pose through the pose decoding output layer.

[0059] The entire prediction and update process is completed asynchronously in the neuron core array without the need for global clock synchronization; each neuron core only performs membrane potential accumulation and threshold comparison when a pulse event arrives, and remains dormant when idle, consistent with the event-driven mechanism of the event camera and spiking neural network processing module.

[0060] This embodiment employs a scalar Kalman filter, with each degree of freedom operating independently to reduce hardware complexity. The filter update frequency is 100Hz, consistent with the SNN inference frequency.

[0061] This invention employs a supervised learning algorithm based on surrogate gradients to train an SNN offline on a GPU NVIDIA RTX 3090.

[0062] Training data preparation used the V1_01_easy sequence from the public Euroc MAV dataset, generating event streams using the ESIM event simulator. The event streams were converted into pulse tensors using the pulse coding method described in Section 3. Each sample contained T=20 time steps, corresponding to a 100ms time window. The true pose was used as the label.

[0063] In the forward propagation, the SNN is simulated sequentially over time steps: at each time step, the membrane potential update of each LIF neuron in each layer is calculated, and a pulse is fired based on a threshold. The pulse firing function is a step function S=Θ(V-θ), which is non-differentiable.

[0064] In the loss function, the total loss is: , : Mean squared error of pose estimation. N is the number of samples; It is the first i Predicted pose of each sample (network output). It is the first i The true pose (label) of each sample.

[0065] Impulse sparsity constraint, target firing rate rtarget=0.05 (5%); L It refers to the number of neuron layers or the number of neuron populations; It is the first l Layer (or first) l The actual average distribution rate for each group; r The target is the target distribution rate, which is 5% (0.05) in this embodiment.

[0066] Energy consumption regularization encourages low-weight and low-pulse activities. L It refers to the number of floors; N l It is the first l The number of neurons in a layer; W ij It is the connection of the first j The input to the first i Synaptic weights of individual neurons; S j It is the first j A number of input pulses (usually 0 or 1, indicating whether a pulse has arrived). Take λ. spike =0.01, λ energy =0.001.

[0067] Backpropagation (surrogate gradient) uses the fast sigmoid function to approximate the derivative of the impulse firing function: , ,

[0068] It is the pulse firing function s with respect to the membrane potential V The derivative of , in practice s is a step function, which is not differentiable, therefore a surrogate function is used for approximation; s is the impulse output of the neuron, usually defined as a step function: s = Θ( V-θ ), θ The threshold value is s=1 when there is a pulse, otherwise s=0. V It is the membrane potential of neurons, the accumulated potential value of LIF neurons, etc.α It is the kurtosis parameter, which controls the shape of the surrogate gradient function; α The larger the value, the closer the function is to the true derivative of the step function (Dirac pulse). α The smaller the value, the smoother the function. In this embodiment, we take... α =5.0; e It is a natural constant, approximately 2.71828.

[0069] This surrogate gradient can be used to calculate the gradient of the loss with respect to the weights, leakage coefficient, and threshold. The optimizer uses Adam with an initial learning rate of 0.001 and is trained for 200 epochs.

[0070] After training, the synaptic weights are quantized from 32-bit floating-point to 4-bit integers using linear quantization, with the scaling factor determined based on the maximum weight value. The leakage coefficient and threshold of the LIF neuron are fixed to the trained values. The quantized weights and parameters are then loaded into... Figure 2 The neuromorphic hardware shown is used for online inference.

[0071] This system was deployed on a micro quadcopter drone with a wheelbase of 150mm and a weight of 250g. The actual test results are as follows: Power consumption: 0.5W (including DVS128 event camera, FPGA, BMI160 IMU, and TF-Luna LiDAR). Positioning frequency: 120Hz. End-to-end latency: 8ms from event occurrence to attitude output. Positioning accuracy: After flying for 2 minutes in a 10m×10m warehouse, the average position RMSE was 0.045m, and the angle RMSE was 0.02rad. Compared with traditional solutions: Under the same task, ORB-SLAM3 consumes 15W and has a latency of 50ms on a Jetson TX2. This invention reduces power consumption by 30 times and latency by 6 times.

[0072] Example 2: The system of the present invention includes an event camera, a pulse coding module, an SNN feature extraction layer, a multi-sensor pulse fusion module, and a pulse Kalman filter module. All of the above modules are deployed on a neuromorphic hardware platform.

[0073] This embodiment further implements a hardware-level dynamic reconfiguration mechanism, which involves asynchronous control logic monitoring the statistical characteristics of the pulse sequence in real time and dynamically adjusting the encoding parameters of the pulse encoding module accordingly to adapt to different scenarios and load conditions.

[0074] Monitoring point A: Output of the pulse code module or input of the SNN, counting the number of pulse events per unit time and calculating the pulse density ρ. event = Nspikes / ΔT, unit: events / ms, where Nspikes is the total number of pulse events counted within this time window.

[0075] Monitoring point B: Neuronal core array, statistically analyzing the average neuron firing rate and membrane potential saturation rate, i.e., the proportion of neurons whose membrane potential exceeds the threshold.

[0076] The asynchronous control logic incorporates a finite state machine, which outputs a reconfiguration command based on the comparison between the monitored value and a preset threshold. Congestion condition: If ρ event >Thr high For example, an event flow rate of 2000 events / ms or a membrane potential saturation rate >80% indicates an excessively dense event flow, which may lead to saturation of the subsequent neuron core or event loss. At this point, the asynchronous control logic sends a command to the pulse coding module: Prioritize increasing the spatial aggregation scale, for example, changing it from 3×3 to 4×4 or 5×5; secondly, appropriately increase the pulse threshold θ. spike For example, increasing the time window length Δt, for example, from 0.5 to 0.7; alternatively, increasing the time window length Δt, for example, from 5 ms to 8 ms, and reducing the number of windows. These adjustments can immediately reduce the output pulse density, allowing the system to return to stability. Let Thr high For high congestion thresholds, e.g., 1500–2000 events / ms; Thr low For low idle thresholds, such as 50–80 events / ms. When ρ event Thr high When ρ is determined to be in a congested state, event < Thr low When it is determined to be in an idle state

[0077] Idle condition: If ρ event <Thr low For example, 50 events / ms and an average firing rate practical < 2% indicates a static scene or sparse textures. In this case, the asynchronous control logic sends commands: first, reduce the spatial aggregation scale, for example, restore it to 3×3 or even 1×1, and second, reduce the pulse threshold; optionally, reduce the time window, for example, from 5ms to 2ms.

[0078] Normal state: If the monitored value is between the high and low thresholds, keep the current encoding parameters unchanged.

[0079] Reconfigurability of the pulse coding module: The coding module has an internal parameter register with a mappable address, supporting real-time write operations. Parameters include: time window length: 1–20ms, step size 1ms; spatial aggregation kernel size: 1×1, 2×2, 3×3, 4×4, 5×5; pulse threshold: 0.1–1.0, step size 0.05.

[0080] Asynchronous control logic: A finite state machine implemented using the pure hardware description language Verilog / VHDL, independent of CPU or software interrupts. Monitoring counters are coupled with an event routing network, enabling statistical analysis within nanoseconds. The feedback bus latency is less than 1μs, guaranteeing real-time closed-loop adjustment.

[0081] Parameter smoothing: To prevent parameters from oscillating near the critical point, the state machine introduces hysteresis comparisons. For example, congestion recovery requires three consecutive windows below the low threshold before backing down.

[0082] In low-dynamic scenarios, the system automatically reduces the spatial aggregation scale or window length to decrease redundant pulses, thereby reducing the number of synaptic operations and dynamic power consumption. In high-dynamic scenarios, pulse density is controlled through aggregation downsampling to avoid ineffective power consumption caused by hardware overload. Real-world testing shows that power consumption can be further reduced by 40% in static scenarios, and the power consumption increase does not exceed 15% in high-dynamic scenarios.

[0083] When a robot encounters rapid rotation or strong lighting changes, the event camera may generate an explosive stream of events. Dynamic reconfiguration can instantly (<10μs) increase the aggregation scale, keeping the pulse density within the tolerance range of the neuron core and ensuring that the localization system does not collapse.

[0084] In the FPGA implementation of this embodiment, let Thr high = 1500-2000 events / ms, Thr low =50-80 events / ms. Initial parameters: Δt=5ms, spatial aggregation=3×3, threshold=0.5. During the drone takeoff phase, the event density suddenly increases to 2000 events / ms. The asynchronous control logic switches the aggregation scale to 4×4 within 2ms, and the pulse density drops to 1200 events / ms, at which point the system stabilizes. Subsequently, when the drone hovers, the density drops to 60 events / ms, and the logic restores the aggregation scale to 3×3 after 5ms.

[0085] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the implementation of the present invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A low-power edge vision localization system for robots based on spiking neural networks, characterized in that, include: An event camera sensor is used to output an asynchronous, pixel-level brightness change event stream; The pulse coding module, connected to the event camera, is used to encode the event stream into a spatiotemporal pulse pattern; The spiking neural network processing module, deployed on neuromorphic hardware, is used to receive the pulse sequence output by the pulse coding module and perform visual feature extraction and pose estimation. The multi-sensor pulse fusion module is electrically connected to the pulse neural network processing module, the pulse output terminal of the IMU, and the pulse output terminal of the lidar. It performs weighted superposition and fusion on the three pulse sequences and outputs a fused pulse sequence. The pulse Kalman filter module, connected to the multi-sensor pulse fusion module, is used to perform state estimation on the fused pulse sequence and output a 6-DOF pose. The asynchronous control logic, located within the neuromorphic hardware, has its signal acquisition terminal connected to the output terminal of the pulse code module or the status monitoring point of the spiking neural network processing module. Its output terminal is connected to the parameter configuration register of the pulse code module via a feedback control bus, which is used to dynamically adjust the encoding parameters of the pulse code module.

2. The low-power edge vision localization system for robots based on spiking neural networks according to claim 1, characterized in that: The encoding parameters of the pulse coding module include at least one of the following: time window length, spatial aggregation scale, and pulse threshold; the encoding parameters are used to control the conversion accuracy and density of event flow to spatiotemporal pulse patterns.

3. The robot low-power edge vision localization system based on spiking neural network according to claim 1, characterized in that: The asynchronous control logic includes: a pulse density statistics unit, a comparator for comparing with a preset threshold, and a finite state machine for outputting reconfiguration commands; the pulse density statistics unit collects pulse sequence density data and transmits it to the comparator, the comparator compares the measured data with the preset threshold and then sends control commands to the finite state machine, and the finite state machine outputs reconfiguration commands.

4. The robot low-power edge vision localization system based on spiking neural network according to claim 3, characterized in that: When the asynchronous control logic detects that the pulse density exceeds the first threshold, it prioritizes increasing the spatial aggregation scale and then decreases the time window length; when the pulse density is lower than the second threshold, it prioritizes decreasing the spatial aggregation scale and then increases the time window length; the first threshold is 1500 to 2000 events / ms, and the second threshold is 50 to 80 events / ms.

5. The robot low-power edge vision localization system based on spiking neural network according to claim 1, characterized in that: The spiking neural network processing module includes: a spiking convolutional layer, a leaky integral ignition (LIF) neuron layer, a spiking pooling layer, and an output layer; the output layer contains 6 neuron populations, each population consisting of multiple LIF neurons, and the 6 neuron populations correspond one-to-one with the robot's three-dimensional position components and three-dimensional pose components, respectively, with the population firing rate encoding each component of the 6-DOF pose.

6. The robot low-power edge vision localization system based on spiking neural network according to claim 1, characterized in that: The multi-sensor pulse fusion module employs a weighted superposition fusion rule: ； in, S v (t), S i (t), S l (t) These are pulse sequences for vision, IMU, and LiDAR, respectively. W v 、W i 、W l For the preset weighting coefficients, satisfy W v +W i +W l =1; The threshold function is defined as follows: if the independent variable is ≥1.5, output 1; otherwise, output 0.

7. The robot low-power edge vision localization system based on spiking neural network according to claim 1, characterized in that: The pulsed Kalman filter module is deployed in the neuronal core array of the neuromorphic hardware; The prediction step is mapped to the neuronal core array in the following way: the state neuron population encodes the posterior state of the previous time step with the firing rate, the firing rate is linearly transformed and accumulated by the synaptic matrix array to form a prior estimated firing rate on the prediction neuron population, wherein the synaptic matrix array pre-stores the synaptic weights of the state transition matrix; each element of the state transition matrix is ​​stored in an independent storage unit of the synaptic matrix array in the form of a 4-8 bit signed integer, the firing rate of each component of the state vector is used as the presynaptic input, and after being weighted and accumulated by the synaptic weights, it is accumulated on the membrane potential of the prediction neuron population to form the prior estimated firing rate; The update process is mapped to the neuron core array as follows: the fused pulse sequence is converted into an observation residual by a decoding unit. The observation residual is the difference between the pulse count of the fused pulse sequence within the time window and the physical quantity corresponding to the predicted firing rate. The residual is converted into a pulse current signal by an event generator. The event generator converts the difference into a pulse current signal whose frequency is proportional to the absolute value of the residual. The pulse current signal is injected into the updating neuron population in the form of asynchronous event packets and superimposed with the predicted membrane potential to complete the state correction. Finally, the firing rate of the updating neuron population is converted into a 6-DOF pose by the pose decoding output layer. The prediction and update process of the pulse Kalman filter module is completed asynchronously in the neuron core array without a global clock synchronization signal. Each neuron core performs membrane potential accumulation and threshold comparison only when a pulse event arrives or is triggered; otherwise, it remains dormant.

8. The low-power edge vision localization system for robots based on spiking neural networks according to claim 1, characterized in that: The neuromorphic hardware includes: an asynchronous event-driven circuit, an address-event representation routing network, a distributed neuron core array, and a synaptic memory; the address-event representation routing network connects the asynchronous event-driven circuit and the distributed neuron core array, and the synaptic memory is independently configured for each neuron core, used to store 4-8 bit signed synaptic weights and call them in real time.

9. A low-power edge vision localization method for robots based on spiking neural networks, characterized in that, Includes the following steps: Step 1: Use the event camera to capture the event stream of scene brightness changes; Step 2: Encode the event stream into a spatiotemporal pulse sequence; Step 3: Input the pulse sequence into the spiking neural network deployed on neuromorphic hardware to extract visual spatiotemporal features; Step 4: Estimate the robot's 6-DOF pose using the output layer of the spiking neural network; Step 5: Using the event camera encoding time window as a unified timing reference, synchronously encode the sensor data of IMU and LiDAR into pulse sequences, and fuse them with the visual pulse sequences; Step 6: Perform pulse Kalman filtering on the fused pulse sequence to obtain a smoothed pose estimate; Step 7: The asynchronous control logic monitors the density or sparsity of the pulse sequence in real time at millisecond intervals and dynamically adjusts the encoding parameters in the encoding step.

10. The low-power edge vision localization method for robots based on spiking neural networks according to claim 9, characterized in that: The training of the spiking neural network adopts a supervised learning algorithm based on surrogate gradient, and the surrogate gradient function is the fast sigmoid function with a slope parameter of 1. The loss function includes pose estimation error, pulse firing rate sparsity constraint, and energy consumption regularization term, where the target firing rate for the sparsity constraint is 5% and the energy regularization coefficient is 0.

001. After training, the synaptic weights are quantized into 4-8 bit integers and loaded into the neuromorphic hardware for inference.

Images