Event vision based mobile robot hybrid ann-snn neurodynamic obstacle avoidance system

Abstract

This invention provides a hybrid ANN-SNN neuromorphic obstacle avoidance system for mobile robots based on event vision. It utilizes a monocular event camera instead of a traditional frame camera and constructs temporal image input using an exponentially decaying surface accumulation method. While achieving compatibility with convolutional networks, it highlights recent events and suppresses noise, significantly improving perception robustness in strong light, backlight, and low-light environments. The proposed neuromorphic perception and control integrated solution is compact, computationally inexpensive, and can be extended to different types of small mobile platforms and event sensors of varying resolutions, demonstrating good scalability and promising engineering applications.

Description

Technical Field

[0001] This invention belongs to the field of autonomous navigation and neuromorphic computing technology for mobile robots, and particularly relates to a hybrid ANN-SNN neuromorphic obstacle avoidance system for mobile robots based on event vision. Background Technology

[0002] With the widespread application of small mobile robots in long-endurance inspections, complex scene exploration, and hazardous environment operations, such platforms are usually limited by size, weight, and energy consumption, and can only carry edge computing devices with limited computing power, making it difficult to support traditional visual perception and control algorithms with high power consumption and high latency.

[0003] Traditional mobile robots often use frame-based RGB or depth cameras, combined with artificial neural networks (ANNs) such as convolutional neural networks, for environmental perception and obstacle avoidance control. However, frame-based cameras have limited frame rates and low dynamic ranges, and are prone to overexposure and underexposure in bright light, backlight, or low-light environments, leading to significant uncertainties in the perception phase. At the same time, deep neural networks based on high-resolution images have high computational demands, resulting in high power consumption and latency when running on edge GPUs or embedded platforms, which is not conducive to the long-term autonomous operation of small robots.

[0004] Event cameras and other dynamic vision sensors can asynchronously trigger pixel-level brightness changes, outputting sparse event stream data. They offer advantages such as high temporal resolution, high dynamic range, and natural sparsity, making them ideal for visual perception under rapid motion or extreme lighting conditions. On the other hand, spiking neural networks (SNNs) and neuromorphic chips, through event-driven and massively parallel computing, offer significant energy efficiency advantages in processing sparse spatiotemporal data. Existing research has proposed combining event vision with SNNs for tasks such as target recognition, optical flow estimation, and depth estimation; however, these studies have largely focused on the perception layer or run on traditional GPU platforms, and a complete neuromorphic perception-control closed loop for mobile robot control has not yet been formed.

[0005] In existing technologies, some works utilize event cameras combined with reinforcement learning or imitation learning to achieve obstacle avoidance for drones, but these mostly rely on GPUs for inference, making it difficult to fully leverage the energy efficiency advantages of neuromorphic hardware in sparse computing scenarios. Furthermore, existing hybrid ANN-SNN networks are mostly used for offline perception tasks, lacking system-level design and verification for end-to-end motion control, especially for deployment on actual neuromorphic chips.

[0006] Therefore, there is an urgent need for an end-to-end obstacle avoidance control method and system for mobile robots that can leverage the advantages of event vision, combine a hybrid ANN-SNN network structure, and run efficiently on neuromorphic hardware, so as to significantly reduce energy consumption and latency while ensuring obstacle avoidance performance. Summary of the Invention

[0007] To address the problems of insufficient perception robustness, large computational latency, and high energy consumption in obstacle avoidance for resource-constrained small mobile robots under complex lighting conditions, this invention provides a hybrid ANN-SNN neuromorphic obstacle avoidance system for mobile robots based on event vision. By constructing a nearly full-link neuromorphic processing pipeline from event input to motion command output, it achieves efficient and low-power end-to-end obstacle avoidance control.

[0008] A hybrid ANN-SNN neuromorphic obstacle avoidance system for mobile robots based on event vision includes a monocular event camera and a neuromorphic host; wherein, the neuromorphic host includes an event preprocessing module and a hybrid ANN. SNN network module, communication module;

[0009] The monocular event camera is used to acquire asynchronous event streams from the environment in front of the robot; The event preprocessing module uses the exponentially decaying surface accumulation method to preprocess the asynchronous event stream to obtain the current time step. Event cumulative image sequence ; The hybrid ANN The SNN network module is used to determine the current time step. Event cumulative image sequence Get current time step Two-dimensional velocity vector prediction; The communication module is used to convert the predicted two-dimensional velocity vector into a driving signal to control the robot's movement, enabling the robot to complete obstacle avoidance tasks.

[0010] Furthermore, the current time step Event cumulative image sequence The method for obtaining it is as follows: The asynchronous event stream is sampled at fixed time steps, and each sampled event is then represented as a quadruple. :

[0011] in, For the first The timestamp of each sampling event For the first The position of each sampling event in the pixel coordinate system No. The polarity of each sampled event, and When position The logarithmic brightness change of the pixel at that location exceeds the positive threshold. hour, , indicating the first Each sampling event is an ON event; when the position The logarithmic brightness change of the pixel at that location exceeds the negative threshold. hour, , indicating the first Each sampling event is an OFF event; Construct a two-dimensional accumulation surface with the same resolution as the event camera, where at any time step... pixel coordinates on the two-dimensional cumulative surface Cumulative value at The calculation formula is as follows:

[0012] in, coordinates in the pixel coordinate system One-to-one correspondence, and They are two different positive numbers. For time step pixel coordinates on the two-dimensional cumulative surface The cumulative decay value at that location; make According to cumulative value The calculation formula to obtain the first The cumulative value corresponding to each sampling event Then, the cumulative value corresponding to all sampled events is used as the current time step. Event cumulative image sequence .

[0013] Furthermore, the cumulative decay value The calculation method is as follows:

[0014] ∈[0,1) is the attenuation factor. For the set time interval, For time step pixel coordinates on the two-dimensional cumulative surface The cumulative value at that location.

[0015] Furthermore, training a hybrid ANN The loss function used in the SNN network module as follows:

[0016] in, The actual two-dimensional velocity vector acquired by the expert through remote operation. For hybrid ANN The predicted two-dimensional velocity vector output by the SNN network module.

[0017] Furthermore, the hybrid ANN The SNN network module includes an encoder, a decoder, and an output head; The decoder includes sequentially cascaded upsample1 module, concat1 module, up1_conv module, upsample2 module, concat2 module, up2_conv module, upsample3 module, concat3 module, up3_conv module, final_conv module, and Aggregation module; The output head includes a head_conv1 module, a head_lif1 module, a head_conv2 module, a head_lif2 module, a head_read module, a GAP module, and an Output layer, all cascaded in sequence. The encoder includes an input pulse sequence layer, an enc1 module, a pool1 module, an enc2 module, a pool2 module, an enc3 module, a pool3 module, and a bottleneck module connected in sequence. In the decoder, the output of the Aggregation module is connected to the input of the head_conv1 module in the output header. The Output layer in the output header is used to output the final two-dimensional velocity vector prediction value. The input pulse sequence layer in the encoder is used to receive the event accumulation image sequence. The output e1 of the enc1 module is connected to the input of the concat3 module, the output e2 of the enc2 module is connected to the input of the concat2 module, the output e3 of the enc3 module is connected to the input of the concat1 module, and the bottleneck module outputs the fused features. The input connected to the upsample1 module.

[0018] Furthermore, the up1_conv, up2_conv, up3_conv, and bottleneck modules are all leakage integral distribution models, specifically Conv2dLifIt. The data processing method for the Conv2dLifIt leakage integral distribution model is as follows:

[0019]

[0020] in, As a leakage factor, This is the synaptic weight matrix. For the current time step Input data, The threshold for issuance, For Heaviside step function, For the current time step ; output pulse; For the current time step The intermediate pulse, For the previous time step The intermediate pulse; Wherein, the Heaviside step function The derivative is expressed using the Sigmoid function instead of the gradient as follows:

[0021] in, For the Sigmoid function, Represents the Heaviside step function The independent variable.

[0022] Furthermore, the GAP module performs global average pooling on the output of the head_read module to obtain the predicted two-dimensional velocity vector. The method is as follows:

[0023] in, This is the output of the Aggregation module. For the set time step threshold, This represents the ANN convolution operation. To prevent the leakage of points distribution operations, This is a one-dimensional convolution operation. This is a global average pooling operation.

[0024] Furthermore, the predicted two-dimensional velocity vector obtained by the Output layer is as follows:

[0025] in, Indicates forward speed. Indicates lateral velocity. and The outputs of the head_read module are respectively , To average the height and width, ⊙ represents element-wise multiplication. For size Horizontal coordinate weighted graph , They are respectively or Height and width.

[0026] Furthermore, the fusion features output by the bottleneck module The method for obtaining it is as follows:

[0027] in, For learnable residual scaling factors, The function is a 1×1 convolutional projection operator, where X is the output of the pool3 module, and features are fused. As input to the upsample1 module in the decoder, For the set time step, ~ These are the components in the bottleneck module. Pulse characteristics of the bottleneck output of a layer SNN.

[0028] Beneficial effects: 1. This invention provides a hybrid ANN-SNN neuromorphic obstacle avoidance system for mobile robots based on event vision. It utilizes a monocular event camera instead of a traditional frame camera and constructs temporal image input using an exponentially decaying surface accumulation method. While achieving compatibility with convolutional networks, it highlights recent events and suppresses noise, significantly improving perception robustness in strong light, backlight, and low-light environments. The neuromorphic perception proposed in this invention... The integrated control solution has a compact structure and low computational complexity. It can be expanded to different types of small mobile platforms and event sensors of different resolutions as needed, and has good scalability and engineering application prospects.

[0029] 2. This invention provides a hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots based on event vision, and proposes an HCLU. Net Hybrid ANN The SNN network adopts a fully convolutional leak integral distribution backbone structure and a hybrid regression head to uniformly encode the temporal and spatial structural information of the event vision, realizing end-to-end control from event input to planar velocity output. Compared with the pure ANN model, it has higher computational sparsity and energy efficiency while maintaining similar obstacle avoidance performance.

[0030] 3. This invention provides a hybrid ANN-SNN neuromorphic obstacle avoidance system for mobile robots based on event vision. By training the model on a GPU and deploying it to a neuromorphic chip using a neuromorphic compilation toolchain, it achieves a higher frame rate per unit power consumption compared to edge GPU platforms under the same task and network structure, significantly reducing the energy consumption of mobile robots in long-term autonomous obstacle avoidance tasks. Experiments show that the inference power consumption of the event vision-based strategy on the GPU platform is lower than that of the depth vision strategy, further improving energy efficiency on the neuromorphic platform.

[0031] 4. This invention provides a hybrid ANN-SNN neuromorphic obstacle avoidance system for mobile robots based on event vision. It adopts a behavior cloning method to directly collect expert data on a real robot platform for training, avoiding the inter-domain differences that may arise from the migration from simulation to reality. This allows the trained neuromorphic obstacle avoidance strategy to be directly deployed and run on a real robot, simplifying the engineering implementation process. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the obstacle avoidance control process for an end-to-end GPU-based benchmark method. Figure 2 For hybrid ANN SNN network HCLU Net structure diagram; Figure 3 This is a schematic diagram of the end-to-end neural mimicry obstacle avoidance control process; Figure 4 This is a schematic diagram of the obstacle avoidance trajectory of the mobile robot in a random obstacle environment in the embodiment. Figure 5 This is a schematic diagram comparing the energy efficiency of the neuromorphic platform and the GPU platform in the embodiments. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0034] This invention provides a neuromorphic obstacle avoidance system for mobile robots based on a monocular event camera and a hybrid artificial neural network spiking neural network structure. In particular, it is an integrated neuromorphic perception and control scheme for resource-constrained small mobile robots that achieves high-efficiency end-to-end obstacle avoidance control in complex lighting environments.

[0035] The mobile robot itself can be a small omnidirectional mobile chassis such as the Wheeltec R550, with a maximum speed of approximately 1 m / s. The front end is equipped with a monocular event camera such as the iniVation DAVIS346, with a preferred spatial resolution of 346×260, a temporal resolution better than 20µs, and a dynamic range greater than 120dB. The onboard computing unit uses the Lingxi HS110 edge neuromorphic host with an integrated brain-inspired processing module HM100 as the main inference platform, and is equipped with an NVIDIA Jetson Orin NX for comparative experiments. The neuromorphic host connects to the event camera via USB 3.0 and to the mobile robot chassis controller via Ethernet. Under the Ubuntu 20.04 + ROS framework, the CPU is responsible for sensor data access and preprocessing, while the neuromorphic APU is responsible for hybrid ANN-SNN network inference. Finally, the planar velocity command is sent to the chassis motor controller via ROS messages, forming an end-to-end perception-control closed loop.

[0036] Specifically, such as Figure 1 As shown, a hybrid ANN-SNN neuromorphic obstacle avoidance system for mobile robots based on event vision includes a monocular event camera and a neuromorphic host; wherein, the neuromorphic host includes an event preprocessing module and a hybrid ANN... SNN network module, communication module; The monocular event camera is used to acquire asynchronous event streams from the environment in front of the robot; The event preprocessing module uses the exponentially decaying surface accumulation method to preprocess the asynchronous event stream to obtain the current time step. Event cumulative image sequence ; The hybrid ANN The SNN network module is used to determine the current time step. Event cumulative image sequence Get current time step Two-dimensional velocity vector prediction; The communication module is used to convert the predicted two-dimensional velocity vector into a driving signal to control the robot's movement, enabling the robot to complete obstacle avoidance tasks.

[0037] Furthermore, the current time step Event cumulative image sequence The method for obtaining it is as follows: The asynchronous event stream is sampled at fixed time steps, and each sampled event is then represented as a quadruple. :

[0038] in, For the first The timestamp of each sampling event For the first The position of each sampling event in the pixel coordinate system No. The polarity of each sampled event, and When position The logarithmic brightness change of the pixel at that location exceeds the positive threshold. hour, , indicating the first Each sampling event is an ON event; when the position The logarithmic brightness change of the pixel at that location exceeds the negative threshold. hour, , indicating the first Each sampling event is an OFF event; To adapt to convolutional networks, this invention proposes the Exponential Decay Surface Accumulation (EDSA) method, which transforms asynchronous event streams into dense frame-like representations. A two-dimensional accumulation surface with the same resolution as the event camera is constructed, where at any time step... pixel coordinates on the two-dimensional cumulative surface Cumulative value at The calculation formula is as follows:

[0039] in, coordinates in the pixel coordinate system One-to-one correspondence, and These are two different positive constants; in this embodiment, it is preferred to take... and , For time step pixel coordinates on the two-dimensional cumulative surface The cumulative decay value at that location; simultaneously, the cumulative decay value The calculation method is as follows:

[0040] in, ∈[0,1) is the decay factor, used to control the "memory" length of historical events. For the set time interval, For time step pixel coordinates on the two-dimensional cumulative surface The cumulative value at the location; the decay factor of the event accumulation surface. Preferably at 0.9 Within the range of 0.95, it is used to smooth out noisy events while highlighting recent motion information. In this embodiment, it is preferred to select... The reset values ​​corresponding to the ON and OFF events. and There is a fixed proportional relationship between them, which is used to represent the difference between the increase and decrease of brightness in the network input.

[0041] make According to cumulative value The calculation formula to obtain the first The cumulative value corresponding to each sampling event Then, the cumulative value corresponding to all sampled events is used as the current time step. Event cumulative image sequence .

[0042] Therefore, this invention samples events at a fixed time step (e.g., 20ms) and updates the surface according to the above formula. This yields an event accumulation image sequence with a frequency of approximately 50 Hz. EDSA serves as the input to subsequent hybrid ANN-SNN networks. Essentially, it acts as a time-leaking filter, highlighting recent motion events and suppressing stale and noisy events, thus making the network input smoother and more stable.

[0043] The following details the hybrid ANN used in this invention. The structure and training methods of SNN network modules.

[0044] First, train the hybrid ANN. The loss function used in the SNN network module as follows:

[0045] in, The actual two-dimensional velocity vector acquired by the expert through remote operation. For hybrid ANN The SNN network module outputs a predicted two-dimensional velocity vector. Based on this, a behavior cloning approach is used to learn the mapping from the accumulated event image to planar velocity commands. The goal is to regress the two-dimensional velocity vector given an input accumulated event image I. ,in Indicates forward speed. This indicates lateral velocity.

[0046] To achieve end-to-end regression, this invention employs a U-Net encoder-decoder structure as the backbone network. The encoder downsamples layer by layer to expand the receptive field, while the decoder upsamples layer by layer and concatenates the features with the corresponding encoder features to fuse multi-scale spatial information. In the HCLU Net, this invention replaces the ordinary convolutional activation modules in the U-Net with convolutional leak integral distribution (Conv2dLifItConv2dLifIt) units, enabling the entire backbone to have explicit temporal modeling capabilities.

[0047] Specifically, such as Figure 2 As shown, the hybrid ANN The SNN network module includes an encoder, a decoder, and an output head; the encoder The decoder adopts a symmetrical structure, with the number of channels in the encoder doubling layer by layer and the number of channels in the decoder halving layer by layer. Multi-scale features are fused through cross-layer skip connections. The bottleneck of the SNN is to perform pulse integration on low-resolution features at multiple time steps to generate pulse rate features, which are then weighted and fused with the features of the ANN branch in a certain proportion to balance temporal modeling capability and network stability.

[0048] The decoder includes sequentially cascaded upsample1 module, concat1 module, up1_conv module, upsample2 module, concat2 module, up2_conv module, upsample3 module, concat3 module, up3_conv module, final_conv module, and Aggregation module; The output head includes a head_conv1 module, a head_lif1 module, a head_conv2 module, a head_lif2 module, a head_read module, a GAP module, and an Output layer, all cascaded in sequence. The encoder includes an input pulse sequence layer, an enc1 module, a pool1 module, an enc2 module, a pool2 module, an enc3 module, a pool3 module, and a bottleneck module connected in sequence. In the decoder, the output of the Aggregation module is connected to the input of the head_conv1 module in the output header. The Output layer in the output header is used to output the final two-dimensional velocity vector prediction value. The input pulse sequence layer in the encoder is used to receive the event accumulation image sequence. The output e1 of the enc1 module is connected to the input of the concat3 module, the output e2 of the enc2 module is connected to the input of the concat2 module, the output e3 of the enc3 module is connected to the input of the concat1 module, and the bottleneck module outputs the fused features. The input connected to the upsample1 module.

[0049] It should be noted that, in order to enhance the sensitivity to the distribution of obstacles on the left and right sides, the output head adopts an anisotropic readout method, and the output feature map is weighted and averaged along the height and width directions. The lateral velocity channel introduces linear coordinate weights in the horizontal direction, making the network more sensitive to the differences between the left and right regions of the image.

[0050] Furthermore, the up1_conv, up2_conv, up3_conv, and bottleneck modules are all leakage integral distribution models, specifically Conv2dLifIt. The data processing method for the Conv2dLifIt leakage integral distribution model is as follows:

[0051]

[0052] in, As a leakage factor, This is the synaptic weight matrix. For the current time step Input data, The threshold for issuance, For Heaviside step function, For the current time step ; output pulse; For the current time step The intermediate pulse, For the previous time step The intermediate pulse; Among them, due to The Heaviside step function is not differentiable, so that backpropagation can be used for training. The derivative is expressed using the Sigmoid function instead of the gradient as follows:

[0053] in, For the Sigmoid function, Represents the Heaviside step function The independent variable is determined by Conv2dLifIt modeling and the alternative gradient function method described above. This allows for optimization of network parameters using gradient descent while maintaining the discrete nature of pulse delivery.

[0054] It should be noted that the Conv2dLifIt module, as the core computing unit, integrates a two-dimensional convolution operator, a spiking neuron dynamics model, and a time iteration mechanism. It extracts spatial features from the input signal through the Conv2d layer and uses the extracted results as the input current of the spiking neuron. Then, it performs iterative loops in the time dimension through the It wrapper to process pulse sequence signals containing the time dimension.

[0055] Meanwhile, the Conv2dLifIt module simulates the accumulation, firing, reset, and leakage behavior of biological neurons. Its dynamic process includes: accumulating the input signal over time using the membrane potential; firing a pulse signal when the membrane potential exceeds a preset threshold; resetting the membrane potential to the initial potential according to the firing state; and simulating the natural leakage process of the membrane potential using an attenuation coefficient. The cmode parameter in the function determines whether the module outputs a discrete pulse signal or a continuous simulated membrane potential signal.

[0056] It should be noted that in some models (such as Hybrid A used for comparison), an SNN regression head is used after the U-Net backbone. Let the feature maps output by the SNN head at T time steps be Y1, ..., Y_T, then the time average is calculated first. ,right Global average pooling is used to obtain the speed prediction:

[0057] The above structure illustrates the integration method of the "pulse head" in the time dimension in this invention.

[0058] Specifically, such as Figure 3 As shown, the final regression head of HCLU-Net adopts a hybrid ANN-SNN structure. Let the high-resolution spatiotemporal features output by the decoder be... First, spatial features are refined using several ANN convolutional layers (ConvANN), then... At each time step, temporal integration is performed using Conv2dLifIt_head to obtain a temporal feature sequence. This sequence is then mapped to two-channel features via 1×1 convolution and averaged in both temporal and spatial dimensions. Based on this, the GAP module performs global average pooling on the output of the head_read module to obtain the predicted two-dimensional velocity vector. The method is as follows:

[0059] in, This is the output of the Aggregation module. For the set time step threshold, This represents the ANN convolution operation. To prevent the leakage of points distribution operations, This is a one-dimensional convolution operation. This is a global average pooling operation.

[0060] The above formula fully describes the hybrid ANN-SNN regression process of HCLU-Net described in this invention, from high-resolution spatiotemporal features to the final velocity command.

[0061] In other words, the Aggregation module and the GAP module are responsible for integrating the temporal features of the signal and reducing its spatial dimension, respectively: the Aggregation module reduces the spatial dimension by integrating the temporal features of the signal and reducing its spatial dimension. The mean calculation is performed to aggregate the pulse sequence information that changes over time into a static feature map; the GAP module then performs mean calculation in the spatial dimension. Global average pooling is performed on the top layer, mapping the feature map to a prediction vector of a specific dimension, with the final output shape being... Obstacle avoidance speed command .

[0062] To enhance the sensitivity of lateral velocity to differences between the left and right regions of the image, this invention employs anisotropic readout during the readout stage: Let the time-averaged features of the two channels be respectively... and Introducing a size of Horizontal coordinate weighted graph , Linear variation in the horizontal direction and normalized to [ If 1,1], then the predicted two-dimensional velocity vector obtained by the Output layer is as follows:

[0063] in, Indicates forward speed. Indicates lateral velocity. and The outputs of the head_read module are respectively , To average the height and width, ⊙ represents element-wise multiplication. For size Horizontal coordinate weighted graph , They are respectively or The height and width of the image. This design enables the network to automatically learn the relationship between the left and right regions of the image and the lateral velocity.

[0064] To perform temporal modeling at low spatial resolution, this invention introduces an SNN bottleneck into HCLU-Net. Let X be the output of the deepest ANN feature in the encoder, i.e., the pool3 module. Expanding the SNN bottleneck at each time step yields pulse features. ,…, The fused features output by the bottleneck module The method for obtaining it is as follows:

[0065] in, For learnable residual scaling factors, The function is a 1×1 convolutional projection operator, where X is the output of the pool3 module, and features are fused. As input to the upsample1 module in the decoder, it is restored into a high-resolution spatiotemporal feature map through layer-by-layer upsampling and skip connections. For the set time step, ~ These are the components in the bottleneck module. Pulse characteristics of the bottleneck output of a layer SNN.

[0066] During the training phase, both the depth map and the event accumulation map were first scaled to the [0,1] interval using 99th percentile normalization, and then uniformly adjusted to a resolution of 256×256 using bilinear interpolation. Data was read using a multi-file HDF5 dataset and lazy-loaded in DataLoader to reduce I / O overhead. To enhance generalization ability, the input image was randomly horizontally flipped, and the lateral component in the velocity label was removed. Change to - To maintain geometric consistency, the optimizer uses AdamW with a learning rate of [missing information]. Weight decay And it employs mixed precision training to improve training efficiency. Finally, to verify the effectiveness and energy efficiency of the proposed event vision-based neuromorphic obstacle avoidance control method and system, multiple real-world experiments were conducted on the Wheeltec R550 omnidirectional mobile platform. The experimental environment was an indoor laboratory, where multiple sets of static obstacles, such as boxes and cylinders of varying sizes, were randomly placed within an area of ​​approximately 4m × 6m. Before each experiment, the robot's initial pose and obstacle layout were randomly changed to simulate a "random static obstacle scenario." The event camera was mounted at a height of approximately 0.4m, with a slightly downward pitch angle, and its field of view covered the main obstacle area in front. Events were processed through exponentially decaying surface accumulation to obtain a 50Hz event accumulation image, which was then input into the network.

[0067] Based on this, the workflow of the event vision-based hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots of the present invention can be summarized as follows: Step 1: Construction of the Neuromorphic Hardware Platform for the Mobile Robot. A monocular event camera, a neuromorphic processing unit, and a motion control unit are installed on the chassis of a small omnidirectional mobile robot. The event camera is used to acquire asynchronous event streams from the environment in front of the robot; the neuromorphic processing unit integrates a brain-inspired processing module and an edge CPU to perform event data preprocessing and hybrid ANN. The system includes SNN network inference and communication with the robot operating system ROS; the motion control unit connects to the chassis drive mechanism to convert planar velocity commands into drive signals. Event cameras, neuromorphic processing units, and motion control units are integrated into a unified software framework via ROS or equivalent middleware.

[0068] The neuromorphic processing unit is preferably an edge computing host integrating a brain-inspired processing module HM100, which supports parallel simulation of at least 250,000 neurons and 25 million synapses, with a typical power consumption of about 14W and a peak power consumption of no more than 30W, and can operate stably for a long time under the limited power supply conditions of mobile robots.

[0069] Step 2: Event Data Preprocessing. Using the event stream output from the event camera, each event is represented as a quadruple containing a timestamp, pixel coordinates, and polarity. To adapt to the convolutional network structure, this invention employs an exponentially decaying surface accumulation method to construct the event accumulation image: a two-dimensional event accumulation surface is maintained at discrete time steps, and at each time step, the entire surface is multiplied by a decay factor. This achieves exponential forgetting of historical events, and then resets the surface value to [value] for pixels where ON or OFF events occur. and The event image that emphasizes recent motion information is generated; the event stream is sampled by a fixed output frequency to obtain an event cumulative image sequence suitable for convolution processing.

[0070] Step 3: End-to-end policy data acquisition based on behavior cloning. In an indoor random static obstacle environment, an expert remotely controls a mobile robot to perform obstacle avoidance tasks. During operation, accumulated event images and the robot's actual forward and lateral velocities are recorded simultaneously, forming a labeled dataset with accumulated event images as input and planar velocity vectors as supervision signals. To improve data diversity, data is collected by repeatedly changing the robot's initial pose and obstacle layout.

[0071] Step 4: Mix ANN SNN network HCLU Net training. Construct a U-type network containing convolutional leak integral distribution units. The Net structure serves as the backbone network, extracting multi-scale spatiotemporal features from the accumulated image sequence of events. An SNN bottleneck layer is introduced at the deepest layer to perform multi-step pulse integration on low-resolution features. After the output is decoded, a hybrid regression head consisting of convolutional layers and Conv2dLifIt layers maps the high-resolution spatiotemporal features into planar velocity predictions.

[0072] During training, the mean squared error between the predicted speed and the expert speed is used as the loss function. AdamW and other optimization algorithms are used for end-to-end supervised learning. Mixed precision training and data augmentation are combined to improve training efficiency and generalization performance.

[0073] Step 5: Neuromorphic Compilation and Deployment. After completing network training on the GPU platform, HCLU... The Net model is input into a brain-inspired deep learning library and converted into a unified intermediate representation. Then, a neuromorphic compiler performs graph fusion, operator rearrangement, and parameter quantization on the network structure, quantizing the 32-bit floating-point weights into 8-bit or other low-bit integer formats. The network operators are then mapped to the parallel computing core and on-chip storage of the neuromorphic chip, generating an executable file for the neuromorphic processing unit. This executable file is deployed to the neuromorphic processing unit on a mobile robot, enabling local inference of the network on the neuromorphic hardware.

[0074] Step 6: Online neuromorphic obstacle avoidance control.

[0075] During the operation phase, the event camera continuously outputs an event stream, which, after undergoing exponential decay surface accumulation processing in step 2, forms an event accumulation image sequence that is input to the HCLU deployed on the neuromorphic processing unit. The Net network receives real-time planar velocity commands. The motion control unit generates chassis drive signals based on the planar velocity commands, driving the mobile robot to perform real-time obstacle avoidance in an environment with random static obstacles, thus forming a neuromorphic closed-loop control from event input to motion output.

[0076] Furthermore, to comprehensively evaluate the performance of this invention under different sensing modalities and illumination conditions, the following sets of comparative experiments were set up: 1) Depth-GPU Baseline: Using a RealSense D435i depth camera as input, the U-Net network was run on OrinNX. Navigation experiments were conducted under daytime (well-lit) and off-light (low-light) conditions to serve as a baseline for traditional dense geometry perception schemes.

[0077] 2) Event-GPU Platform: Using a DAVIS346 event camera, only the event stream was retained and converted into an event accumulation image via EDSA. Four network architectures (Baseline ANN, Hybrid A, Hybrid B, and HCLU-Net) were deployed on Orin NX, and experiments were conducted in daytime and dark environments. Experiments showed that all networks were able to learn effective obstacle avoidance strategies, allowing the robot to reach the target area from its starting point in a random obstacle environment. HCLU-Net maintained a more stable trajectory and smoother speed output under different lighting conditions.

[0078] 3) Robustness to Multiple Initial Pose: Under the condition of using only event camera input and deploying HCLU-Net, the robot successfully reached the destination from four different initial positions (A, B, C, and D), demonstrating the generalization ability of the learned strategy to initial pose and local obstacle layout. Statistical results show that in multiple experiments, the HCLU-Net-based strategy achieved a task success rate of over 70% in real-world environments.

[0079] Regarding energy efficiency, this invention focuses on comparing inference power consumption and frame rate on GPU and neuromorphic platforms. For ease of quantification, a frame rate per unit power consumption metric is defined.

[0080] in For inference frame rate, This represents the average power consumption during the inference process. First, all networks are deployed on the Orin NX platform, and the average power consumption during real-time obstacle avoidance tasks is measured using the onboard power monitoring interface. For example... Figure 5 As shown, the results indicate that, under the same network structure, the HCLU-Net based on event vision consumes approximately 2.1W of power when processing event data on a GPU, while consuming approximately 2.6W when processing depth data. This demonstrates that even on traditional GPUs, the sparse computation of the event-hybrid network can bring certain energy consumption advantages.

[0081] Subsequently, the trained HCLU-Net was compiled using the BIDL and Lyngor toolchains and deployed to the Lynxi HS110 neuromorphic host, where it was tested under the same event input and robot task conditions. Experiments showed that the energy efficiency of HCLU-Net on the HS110 was approximately 4.46 FPS / W, while on Orin NX it was 2.15 FPS / W, the former being approximately twice that of the latter. In experiments verifying the separate hybrid network SNN prediction head module, the energy efficiency ratio of network inference deployed on the Lynxi HS110 reached 88.86 FPS / W, while on Orin NX it was 12.90 FPS / W, representing a more than seven-fold improvement in the energy efficiency ratio of the velocity prediction head. This indicates that, while maintaining the same end-to-end obstacle avoidance performance, the neuromorphic deployment scheme proposed in this invention significantly improves the effective frame rate per unit of energy consumption.

[0082] From the perspective of navigation effectiveness, such as Figure 4 As shown, the robot's trajectory in a random obstacle environment can smoothly avoid obstacles and eventually reach the target area. In low-light environments such as when the lights are off, traditional depth camera perception links are prone to noise and gaps, while the neuromorphic scheme relying solely on event cameras can still generate reasonable obstacle avoidance trajectories. By combining event modeling and exponentially decaying surface accumulation, a hybrid ANN-SNN network structure, and the aforementioned neuromorphic compilation and deployment process, the system of this invention maintains stable closed-loop control performance during the operating period and achieves high energy efficiency. It has a clear advantage over the GPU platform.

[0083] In summary, through multi-condition and multi-platform comparative experiments on real robot platforms, the event vision-based hybrid ANN-SNN neuromorphic obstacle avoidance control method and system proposed in this invention not only has good task success rate and trajectory robustness under complex lighting conditions and multiple initial poses, but also significantly outperforms traditional GPU deployment schemes in terms of energy efficiency indicators such as unit power consumption and frame rate. It can achieve long-term and efficient autonomous obstacle avoidance on small mobile robots with limited resources.

[0084] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots based on event vision, characterized in that, It includes a monocular event camera and a neuromorphic host; the neuromorphic host includes an event preprocessing module and a hybrid ANN. SNN network module, communication module; The monocular event camera is used to acquire asynchronous event streams from the environment in front of the robot; The event preprocessing module uses the exponentially decaying surface accumulation method to preprocess the asynchronous event stream to obtain the current time step. Event cumulative image sequence ; The hybrid ANN The SNN network module is used to determine the current time step. Event cumulative image sequence Get current time step Two-dimensional velocity vector prediction; The communication module is used to convert the predicted two-dimensional velocity vector into a driving signal to control the robot's movement, enabling the robot to complete obstacle avoidance tasks.

2. The event-vision-based hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots as described in claim 1, characterized in that, Current time step Event cumulative image sequence The method for obtaining it is as follows: The asynchronous event stream is sampled at fixed time steps, and each sampled event is then represented as a quadruple. : in, For the first The timestamp of each sampling event For the first The position of each sampling event in the pixel coordinate system No. The polarity of each sampled event, and When position The logarithmic brightness change of the pixel at that location exceeds the positive threshold. hour, , indicating the first Each sampling event is an ON event; when the position The logarithmic brightness change of the pixel at that location exceeds the negative threshold. hour, , indicating the first Each sampling event is an OFF event; Construct a two-dimensional accumulation surface with the same resolution as the event camera, where at any time step... pixel coordinates on the two-dimensional cumulative surface Cumulative value at The calculation formula is as follows: in, coordinates in the pixel coordinate system One-to-one correspondence, and They are two different positive numbers. For time steps pixel coordinates on the two-dimensional cumulative surface The cumulative decay value at that location; make According to cumulative value The calculation formula to obtain the first The cumulative value corresponding to each sampling event Then, the cumulative value corresponding to all sampled events is used as the current time step. Event cumulative image sequence .

3. The event-vision-based hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots as described in claim 2, characterized in that, Decaying cumulative value The calculation method is as follows: ∈[0,1) is the attenuation factor. For the set time interval, For time steps pixel coordinates on the two-dimensional cumulative surface The cumulative value at that location.

4. The event-vision-based hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots as described in claim 1, characterized in that, Training Hybrid ANN The loss function used in the SNN network module as follows: in, The actual two-dimensional velocity vector acquired by the expert through remote operation. For hybrid ANN The predicted two-dimensional velocity vector output by the SNN network module.

5. The event-vision-based hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots as described in claim 1, characterized in that, The hybrid ANN The SNN network module includes an encoder, a decoder, and an output head; The decoder includes sequentially cascaded upsample1 module, concat1 module, up1_conv module, upsample2 module, concat2 module, up2_conv module, upsample3 module, concat3 module, up3_conv module, final_conv module, and Aggregation module; The output head includes a head_conv1 module, a head_lif1 module, a head_conv2 module, a head_lif2 module, a head_read module, a GAP module, and an Output layer, all cascaded in sequence. The encoder includes an input pulse sequence layer, an enc1 module, a pool1 module, an enc2 module, a pool2 module, an enc3 module, a pool3 module, and a bottleneck module connected in sequence. In the decoder, the output of the Aggregation module is connected to the input of the head_conv1 module in the output header. The Output layer in the output header is used to output the final two-dimensional velocity vector prediction value. The input pulse sequence layer in the encoder is used to receive the event accumulation image sequence. The output e1 of the enc1 module is connected to the input of the concat3 module, the output e2 of the enc2 module is connected to the input of the concat2 module, the output e3 of the enc3 module is connected to the input of the concat1 module, and the bottleneck module outputs the fused features. The input connected to the upsample1 module.

6. The event-vision-based hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots as described in claim 5, characterized in that, The up1_conv, up2_conv, up3_conv, and bottleneck modules are all leakage integral distribution models, specifically Conv2dLifIt. The data processing method for the Conv2dLifIt leakage integral distribution model is as follows: in, As a leakage factor, This is the synaptic weight matrix. For the current time step Input data, The threshold for issuance, For Heaviside step function, For the current time step ; output pulse; For the current time step The intermediate pulse, For the previous time step The intermediate pulse; Wherein, the Heaviside step function The derivative is expressed using the Sigmoid function instead of the gradient as follows: in, For the Sigmoid function, Represents the Heaviside step function The independent variable.

7. The event-vision-based hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots as described in claim 5, characterized in that, The GAP module performs global average pooling on the output of the head_read module to obtain the predicted two-dimensional velocity vector value. The method is as follows: in, This is the output of the Aggregation module. For the set time step threshold, This represents the ANN convolution operation. To prevent the leakage of points distribution operations, This is a one-dimensional convolution operation. This is a global average pooling operation.

8. The event vision-based hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots as described in claim 5, characterized in that, The predicted two-dimensional velocity vector obtained by the Output layer is as follows: in, Indicates forward speed. Indicates lateral velocity. and The outputs of the head_read module are respectively , To average the height and width, ⊙ represents element-wise multiplication. For size Horizontal coordinate weighted graph , They are respectively or Height and width.

9. The event-vision-based hybrid ANN-SNN neural mimicry obstacle avoidance system for mobile robots as described in claim 5, characterized in that, The fusion features output by the bottleneck module The method for obtaining it is as follows: in, For learnable residual scaling factors, The function is a 1×1 convolutional projection operator, where X is the output of the pool3 module, and features are fused. As input to the upsample1 module in the decoder, For the set time step, ~ These are the components in the bottleneck module. Pulse characteristics of the bottleneck output of a layer SNN.

Images