Visual perception method and device based on spiking neural network, equipment and medium

Abstract

The invention relates to the technical field of neural networks, in particular to a visual perception method, device and equipment based on a pulse neural network and a medium, and adopts a visual perception system based on the pulse neural network, the method comprises the following steps: acquiring multi-source image data by using an input module, and preprocessing the multi-source image data to obtain a pre-processed image; obtaining an image tensor containing a time dimension; performing feature extraction on the image tensor by using an SNN-ResNet visual backbone network to obtain a target time sequence feature; and based on a preset sparse attention mechanism, a Transform perception output head is utilized to perform visual perception according to the target time sequence features, and a target detection and tracking result is obtained. Therefore, the problems that a traditional visual perception system is high in power consumption, insufficient in time sequence modeling capacity, low in reasoning speed and the like are solved, and efficient perception with low power consumption, high time dynamic modeling capacity and quick response is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, and particularly relates to a visual perception method and device based on a spiking neural network, equipment and a medium. BACKGROUND

[0002] In recent years, with the increasing demand for environmental perception capability in application scenarios such as automatic driving, unmanned delivery and intelligent security, a multi-camera vision system has gradually become one of the mainstream solutions to replace laser radar. The visual perception system relies on a deep neural network to perform high-precision three-dimensional detection and target tracking on multi-view image information. Among them, the Bird's Eye View (BEV, bird's eye view) modeling method based on dense representation, such as BEVFormer (Bird's Eye View Former, bird's eye view transformer), PETR (Position Embedding Transformation, perspective embedding transformer for three-dimensional target detection), and the Sparse4D series algorithm based on sparse query modeling have achieved leading performance on multiple public datasets (such as nuScenes, Waymo), representing the advanced level of the current multi-camera 3D perception field. However, the existing system generally uses a traditional convolutional neural network (CNN, Convolutional Neural Network) as the image feature extraction backbone, such as ResNet (Residual Network), ResNeXt (Residual Next), Swin Transformer (Shifted Window Transformer), etc. Although these networks have strong expression ability, they still have obvious deficiencies in low-power deployment, timing modeling capability and neurobiological inspiration, and are difficult to meet the current trend of “high precision + low latency + low power consumption” three requirements.

[0003] Although the existing visual perception system has made significant progress in accuracy, the CNN backbone network used by the existing visual perception system has the following problems: first, the CNN model needs to perform pixel-by-pixel convolution operation on the input image, and even in sparse structure, frequent calculation of the entire feature map cannot be avoided, resulting in high calculation density and memory occupation. This is particularly difficult when deploying in edge devices such as vehicle-mounted SOC and low-power embedded platforms, often requiring a significant compression of the model, thereby causing performance degradation. Second, although some methods introduce a Transformer structure to process multi-frame information, the CNN backbone network itself does not have the ability to model time dynamics, and all time information needs to be fused in the Transformer head. This structure may cause the feature information to fail to fully retain the time sequence pattern, affecting the tracking stability. In addition, in the process of performing multi-camera input and multi-stage convolution extraction, the existing CNN architecture has multiple sequential processing steps, which is low in GPU execution efficiency and high in response delay, and is difficult to meet the timeliness requirements of real-time feedback scenarios such as autonomous driving and robot obstacle avoidance. Finally, the current mainstream visual perception network has not fully utilized the inspiration of the biological nervous system in time modeling, event-driven, power control, etc. In contrast, as a brain-inspired computing model that has emerged in recent years, the spiking neural network (SNN) naturally has the characteristics of sparsity and low power consumption by simulating the firing mechanism of neurons and only calculating when information changes. However, SNN is usually used in simple tasks such as gesture recognition and speech coding, and has not been fully integrated into complex perception tasks. Although some work has tried to introduce SNN structure into traditional neural networks, due to the essential differences between SNN and standard CNN in time domain propagation mechanism, gradient calculation method (such as surrogate gradient), tensor structure (such as time dimension T), etc., the existing deep learning framework (such as MMCV, MMDetection) cannot directly support it, thereby limiting the application of SNN in mainstream perception systems, which needs to be solved urgently. SUMMARY

[0004] The present application provides a visual perception method, device, equipment and medium based on a spiking neural network, to solve the problems of high power consumption, insufficient time modeling capability and slow inference speed of traditional visual perception systems, and to realize efficient perception with low power consumption, high time dynamic modeling capability and fast response.

[0005] The first aspect embodiment of the present application provides a visual perception method based on a spiking neural network, which adopts a visual perception system based on a spiking neural network, the system comprising an input module, an SNN-ResNet visual backbone network and a Transformer perception output head, comprising the following steps: The input module is used to obtain multi-source image data and pre-process the multi-source image data to obtain an image tensor containing a time dimension; The SNN-ResNet visual backbone network is used to extract features from the image tensor to obtain target time sequence features; Based on a preset sparse attention mechanism, the Transformer perception output head is used to perform visual perception according to the target time sequence features to obtain target detection and tracking results.

[0006] According to an embodiment of the present application, the pre-processing of the multi-source image data to obtain an image tensor containing a time dimension comprises: Synchronization and alignment operations are performed on each frame of multi-source image to obtain aligned multi-source images; Each frame of aligned multi-source image is expanded to a preset number of time steps to obtain the image tensor containing the time dimension.

[0007] According to an embodiment of the present application, the SNN-ResNet visual backbone network is used to extract features from the image tensor to obtain target time sequence features, comprising: The convolutional layer of the SNN-ResNet visual backbone network is used to extract features from the image tensor of each time step to obtain spatial features of each time step; The LIF pulse neuron activation module of the SNN-ResNet visual backbone network is used to simulate neuron membrane potential accumulation and / or discharge process according to the spatial features of each time step to obtain sparse features; The normalization layer of the SNN-ResNet visual backbone network is used to perform normalization operation on the sparse features to obtain normalized features; All time step normalized features are fused to obtain the target time sequence features.

[0008] According to an embodiment of the present application, before fusing all time step normalized features, further comprising: The residual connection module of the SNN-ResNet visual backbone network is used to connect multiple pulse neural networks.

[0009] According to an embodiment of the present application, based on a preset sparse attention mechanism, the Transformer perception output head is used to perform visual perception according to the target time sequence features to obtain target detection and tracking results, comprising: Initialize query vectors, wherein each query vector corresponds to a 3D target instance; Based on the preset sparse attention mechanism, the each query vector and the target time sequence feature are subjected to multi-round attention interaction to obtain the target detection and tracking result.

[0010] According to an embodiment of the present application, the target detection and tracking result includes at least one of a 3D bounding box of a target perception object, an object category, and a cross-frame target association ID.

[0011] According to the pulse neural network-based visual perception method, the input module is used to obtain multi-source image data, and the multi-source image data is preprocessed to obtain an image tensor containing a time dimension; the SNN-ResNet visual backbone network is used to extract features of the image tensor to obtain target time sequence features; and the Transformer perception output head is used to perform visual perception based on the target time sequence features based on the preset sparse attention mechanism to obtain a target detection and tracking result. Thus, the problems of high power consumption, insufficient time sequence modeling capability, and slow inference speed of a traditional visual perception system are solved, and efficient perception with low power consumption, high time dynamic modeling capability, and fast response is achieved.

[0012] The second aspect embodiment of the present application provides a pulse neural network-based visual perception device, comprising: The device adopts a pulse neural network-based visual perception system, and the system comprises an input module, an SNN-ResNet visual backbone network, and a Transformer perception output head, comprising: The acquisition module is configured to acquire multi-source image data by using the input module, and pre-process the multi-source image data to obtain an image tensor containing a time dimension. The feature extraction module is configured to extract features of the image tensor by using the SNN-ResNet visual backbone network to obtain target time sequence features. The perception module is configured to perform visual perception based on the target time sequence features by using the Transformer perception output head based on a preset sparse attention mechanism to obtain a target detection and tracking result.

[0013] According to an embodiment of the present application, the acquisition module comprises: The multi-source image of each frame is subjected to synchronization and alignment operations to obtain an aligned multi-source image. The aligned multi-source image of each frame is expanded to a preset number of time steps to obtain the image tensor containing the time dimension.

[0014] According to an embodiment of the present application, the feature extraction module is configured to: extract features of the image tensor of each time step by using a convolution layer of the SNN-ResNet visual backbone network, to obtain spatial features of each time step; simulate a neuron membrane potential accumulation and / or a firing process according to the spatial features of each time step by using a LIF pulse neuron activation module of the SNN-ResNet visual backbone network, to obtain sparse features; perform a normalization operation on the sparse features by using a normalization layer of the SNN-ResNet visual backbone network, to obtain normalized features; fuse the normalized features of all time steps, to obtain the target time sequence features.

[0015] According to an embodiment of the present application, before fusing the normalized features of all time steps, the feature extraction module is further configured to: connect a plurality of pulse neural networks by using a residual connection module of the SNN-ResNet visual backbone network.

[0016] According to an embodiment of the present application, the perception module is configured to: initialize a query vector, wherein each query vector corresponds to a 3D target instance; perform a plurality of rounds of attention interaction between the query vector and the target time sequence features based on a preset sparse attention mechanism, to obtain the target detection and tracking result.

[0017] According to an embodiment of the present application, the target detection and tracking result includes at least one of a 3D bounding box of a target perception object, an object category, and a cross-frame target association ID.

[0018] According to an embodiment of the present application, the visual perception device based on a pulse neural network acquires multi-source image data by using an input module, and pre-processes the multi-source image data to obtain an image tensor containing a time dimension; extracts features of the image tensor by using an SNN-ResNet visual backbone network, to obtain target time sequence features; and performs visual perception according to the target time sequence features by using a Transformer perception output head based on a preset sparse attention mechanism, to obtain a target detection and tracking result. In this way, the problems of high power consumption, insufficient time sequence modeling capability, and slow inference speed of a traditional visual perception system are solved, and efficient perception with low power consumption, high time dynamic modeling capability, and fast response is achieved.

[0019] An electronic device is provided in a third aspect of the present application, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the visual perception method based on a pulse neural network as described in the above embodiments.

[0020] A computer readable storage medium is provided in the fourth aspect of the present application, and the computer program is stored in the computer readable storage medium. The computer program is executed by a processor to implement the visual perception method based on the spiking neural network according to the above-mentioned embodiments.

[0021] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, wherein: Figure 1 A structural flowchart of a visual perception system based on a spiking neural network according to an embodiment of the present application; Figure 2 A reasoning flowchart of a visual perception system based on a spiking neural network according to an embodiment of the present application; Figure 3 A flowchart of a visual perception method based on a spiking neural network according to an embodiment of the present application; Figure 4 A block schematic diagram of a visual perception apparatus based on a spiking neural network according to an embodiment of the present application; Figure 5 A structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The embodiments of the present application are described in detail below with reference to the accompanying drawings. Examples of the embodiments are shown in the drawings, wherein the same or similar notations denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and are not to be understood as limiting the present application.

[0024] The visual perception method, apparatus, device and medium based on a spiking neural network according to the embodiments of the present application are described below with reference to the accompanying drawings.

[0025] Before introducing the visual perception method based on a spiking neural network of the embodiments of the present application, first, the visual perception system based on a spiking neural network involved in the visual perception method based on a spiking neural network proposed by the present application is introduced.

[0026] Specifically, the application provides a visual perception system architecture based on a spiking neural network, specifically: replacing the backbone network in a traditional multi-camera 3D target detection and tracking system with an SNN-ResNet module, processing image features through spiking neurons (LIF, Leaky Integrate-and-Fire) for multiple time steps, and fusing with a sparse Transformer head structure to form an end-to-end visual perception system with low power consumption and high time modeling capability.

[0027] It can be understood that the Transformer is a neural network structure for information interaction, and the sparse Transformer optimizes the attention mechanism on the basis of the Transformer, and only performs information calculation on a limited area or specific query, thereby reducing the computational complexity.

[0028] Further, the visual perception system based on the spiking neural network according to the embodiment of the application is composed of three parts: an input module, an SNN-ResNet visual backbone network and a Transformer perception output head.

[0029] Specifically, the input module is used to receive synchronous image frames from multiple cameras, and expand the static image into multiple time step representations to form an image tensor with a time dimension. Each image is expanded into T frames when input, representing the multiple time step responses of neurons to input stimuli.

[0030] Further, the SNN-ResNet visual backbone network is used to extract time sequence features, including a convolutional layer, a LIF spiking neuron activation module, a normalization layer and a residual connection module.

[0031] The convolutional layer extracts image features in parallel in time steps. The LIF spiking neuron module is used to simulate the process of neuron membrane potential accumulation and discharge, and only generates a pulse output when the input stimulus exceeds the threshold, thereby realizing event-driven sparse reasoning. The normalization module is responsible for maintaining the stability of the neuron state at each time step, and improving the stability of training and reasoning. The residual connection module is used for information preservation and reverse gradient flow, which helps to build a deep network.

[0032] Thus, the structure of the SNN-ResNet backbone network in the embodiment of the present application is reformed and the time window mechanism is introduced, so that the traditional ResNet network is reconfigured into an SNN-ResNet backbone supporting multi-time step input, and the image input drives the neuron firing behavior in a time series manner through the time window mechanism, so as to realize input timing modeling. In addition, the LIF pulse neuron is integrated into the activation link in the backbone module, and the LIF (Leaky Integrate-and-Fire) neuron is integrated as an activation function after all convolution layers, replacing the traditional ReLU or Swish activation mode, so as to simulate the response behavior of biological neurons under timing information, thereby realizing event-driven and energy-sparse calculation.

[0033] Further, the sparse Transformer perception head uses the multi-time step fused image features from the backbone network to complete the 3D target detection and multi-frame target association tracking tasks through the sparse attention mechanism. As shown in the example, Figure 1 As shown, first, the image input is obtained through six cameras, and the image data is input into the SNN-ResNet backbone network to extract timing features, and then the sparse Transformer head is used for further feature analysis and processing, and finally the target detection and tracking results containing 3D bounding boxes and target identifiers are output.

[0034] Further, the pulse coding mechanism in the visual perception system based on the pulse neural network in the embodiment of the present application is introduced.

[0035] Specifically, the repeat(T, 1, 1, 1) operation is performed on each frame of image, so that the static image is expanded into T time steps to form a five-dimensional tensor [T, B, C, H, W]. Wherein, each time step represents a neuron integration calculation, and the LIF neuron is used to determine whether to fire.

[0036] Further, the LIF neuron structure in the embodiment of the present application is implemented with reference to the LIF class in activation.py: for t in range(self.T): mem = mem * self.tau + x[t,...]# Integration process spike = self.act(mem - self.thresh, self.gama)# Trigger function to determine whether to fire mem = (1 - spike) * mem# Reset membrane potential spike_pot.append(spike) Wherein, the LIF neuron simulates the voltage change of the neuron in the time dimension, and has the following working principle: when the accumulated potential is greater than the threshold thresh, the neuron fires (output is 1), and the membrane potential is cleared after firing, and the non-firing continues to accumulate (tau decay coefficient). This mechanism allows the network to run in a time-sparse manner, that is, only when a "perception event" occurs, the system is activated, thereby improving the energy efficiency of the system.

[0037] The convolution operation and parallel processing mechanism in the visual perception system based on the spiking neural network are introduced below.

[0038] Specifically, Conv2ds is defined in conv.py, and its structure is as follows, supporting parallel convolution processing of image features at each time step in the time dimension T.

[0039] def forward(self, inputs): list = [] for input in inputs: output = self._conv_forward(input, self.weight, self.bias) list.append(output) return torch.stack(list, dim=0) The pulse BN structure in the visual perception system based on the spiking neural network is introduced below.

[0040] Specifically, Batch_norm_2ds in norm.py is implemented: three-dimensional normalization processing is performed on the [T, B, C, H, W] tensor, BatchNorm3d is used to realize statistical balance across time steps, and the BN weight is initialized to the firing threshold 0.5, which is more consistent with the propagation characteristics of SNN.

[0041] In addition, the BatchNorm3D of the embodiment of the application can also be equivalently replaced by GroupNorm, LayerNorm or the voltage normalization mechanism specific to SNN, which is not specifically limited here.

[0042] The SNN-ResNet module integration in the visual perception system based on the spiking neural network is introduced below.

[0043] Specifically, the forward method of the ResNets class in resnets.py is as follows: x = x.repeat(self.time_step, 1, 1, 1) # input image expanded to time steps x = self.stem(x) or self.conv1(x) # input convolution for x_i in x: x_i = self.maxpool(x_i) # pooling at each time step for i, layer_name in enumerate(self.res_layers): x = res_layer(x) # multiple ResNet residual layers outs = torch.stack(outs,dim=0).mean(0) # temporal dimension fusion Thus, the traditional ResNet module is guaranteed to be specialized in the time step dimension of SNN, while maintaining a consistent interface with sparse Transformer.

[0044] The inference process of the visual perception system based on the pulse neural network will be described in detail below. Figure 2

[0045] Specifically, the inference process of the visual perception system based on the pulse neural network can include the following steps: First, image preprocessing: synchronize and align images from multiple cameras, unify the resolution, and expand them into multiple time steps to form an input structure of [T, B, C, H, W].

[0046] Further, time step expansion: each image is duplicated T times as the input of neurons in T time steps, representing the process of accepting continuous stimuli.

[0047] Further, feature extraction and encoding: in the SNN-ResNet backbone, each time step input is sequentially passed through convolutional layers to extract spatial features, and then through LIF neurons to determine whether to fire. Pulse neurons decide whether to activate based on whether the accumulated voltage exceeds the firing threshold. Firing neurons empty the membrane potential, and non-firing neurons continue to accumulate, simulating the excitation-recovery mechanism in the biological nervous system.

[0048] Further, normalization processing: three-dimensional normalization operation across time steps is used in the pulse network to ensure the consistency of output scales between different time steps, facilitating subsequent fusion.

[0049] ​Further, feature fusion and down-sampling are performed: multiple SNN layers are connected through a pulse residual module to retain deep abstract features, and maximum pooling is used to complete spatial scale compression and retain key structural information.

[0050] Further, time dimension fusion output: the outputs of all time steps are averaged (or other fusion strategies) in the time dimension to synthesize a single frame of feature representation, which is input to a sparse Transformer for detection and tracking prediction.

[0051] Finally, output decoding: the Transformer module outputs 3D bounding boxes, object categories, and cross-frame target association IDs based on the fused features to achieve the final output of perception results.

[0052] Thus, according to the visual perception system architecture based on the pulse neural network proposed in the present application, the traditional convolutional neural network structure used for image feature extraction is replaced by an SNN-ResNet module with time dynamic modeling capability. This structure simulates the discharge behavior of biological neurons through a time window mechanism and a pulse activation function, thereby achieving time series modeling of continuous frame images at the visual backbone level and completing three-dimensional target detection and tracking by fusion with a sparse Transformer module.

[0053] The following describes in detail the visual perception method based on the pulse neural network applied to the visual perception system based on the pulse neural network described above.

[0054] Specifically, Figure 3 A flowchart of the visual perception method based on the pulse neural network provided by an embodiment of the present application is shown in FIG. 3.

[0055] As Figure 3 shown, the visual perception method based on the pulse neural network includes the following steps: In step S301, a plurality of source image data is obtained by using an input module, and the plurality of source image data is preprocessed to obtain an image tensor containing a time dimension.

[0056] Further, in some embodiments, the preprocessing of the plurality of source image data to obtain the image tensor containing the time dimension includes: performing synchronization and alignment operations on each frame of the plurality of source images to obtain aligned source images; and expanding each frame of the aligned source images into a preset number of time steps to obtain the image tensor containing the time dimension.

[0057] Specifically, since the multi-source image data comes from cameras at different positions and angles, there are problems of time synchronization and spatial alignment. Therefore, first of all, the images need to be synchronized to ensure that the multi-source image data is consistent in time, and the alignment operation of the embodiments of the present application can be image cropping, scaling, and translation operations to ensure that the images captured by different cameras correspond in space.

[0058] After completing the synchronization and alignment operations, each frame of image is expanded to a preset number of time steps by copying each frame of image multiple times, and each copied image represents a time step. The image tensor obtained after expansion has a five-dimensional structure, namely [T, B, C, H, W], where T is the number of time steps, B is the batch size, C is the number of color channels, and H and W represent the height and width of the image, respectively.

[0059] Thus, the conventional 2D convolution and BN module are upgraded to support the structure of time step [T, B, C, H, W] through the adaptation of the time dimension convolution, normalization, and residual propagation mechanism, ensuring the continuity of the features in the time domain and maintaining the long-range time dependence through the timing residual structure.

[0060] In step S302, the SNN-ResNet visual backbone network is used to extract features from the image tensor to obtain target timing features.

[0061] Further, in some embodiments, the SNN-ResNet visual backbone network is used to extract features from the image tensor to obtain target timing features, including: using the convolution layer of the SNN-ResNet visual backbone network to extract features from the image tensor of each time step to obtain spatial features of each time step; using the LIF pulse neuron activation module of the SNN-ResNet visual backbone network to simulate the neuron membrane potential accumulation and / or discharge process according to the spatial features of each time step to obtain sparse features; using the normalization layer of the SNN-ResNet visual backbone network to perform normalization operation on the sparse features to obtain normalized features; and fusing the normalized features of all time steps to obtain target timing features.

[0062] Specifically, the convolution layer of the SNN-ResNet visual backbone network processes the image tensor of each time step to extract the spatial features of the image. These convolution layers can capture local patterns and structures in the image and provide a basis for subsequent timing feature extraction. In each time step, the convolution operation is performed in parallel to independently process each time step of the image tensor, thereby obtaining the spatial features corresponding to each time step.

[0063] Further, the LIF pulse neuron activation module in the SNN-ResNet visual backbone network simulates the membrane potential accumulation and discharge process of biological neurons. The LIF pulse neuron activation module only generates pulse output when the input stimulus exceeds the threshold, thereby realizing event-driven sparse reasoning.

[0064] Further, the normalization layer of the SNN-ResNet visual backbone network performs normalization operation on the sparse features generated by the LIF neuron module. The normalization module is responsible for adjusting the distribution of features, so that the network can converge faster during training, and at the same time improve the robustness of the model to input changes.

[0065] Finally, the normalized features of all time steps are fused to obtain the time sequence features of the target. The fused target time sequence features contain the spatial information and temporal information of the image, providing rich feature representation for subsequent 3D target detection and tracking.

[0066] Further, in some embodiments, before fusing the normalized features of all time steps, the method further includes: connecting the multiple pulse neural networks by using the residual connection module of the SNN-ResNet visual backbone network.

[0067] The residual connection is a network structure design that allows signals in the network to bypass one or more layers and propagate directly, thereby alleviating the gradient vanishing and gradient explosion problems in deep network training, making the training of deep networks more stable.

[0068] Specifically, in the SNN-ResNet visual backbone network, the residual connection is used to connect multiple pulse neural network layers. The output of each SNN layer will not only be transmitted to the next SNN layer, but also be added or fused with the output of the previous layer to form a residual signal. The residual connection helps to maintain information in the network and facilitate the flow of backpropagation gradients. By adding the input directly to the output, the residual connection ensures that information can be effectively transmitted from the input layer to the output layer even in deep networks, while the gradient can also flow smoothly during backpropagation.

[0069] In addition, the embodiments of the present application can also replace LIF with IF (Integrate-and-Fire), PLIF (Parametric LIF), or SRM (Spike Response Model) and other neuron forms with pulse output characteristics, which are not limited here.

[0070] Therefore, in view of the problems of the existing CNN backbone network being a dense structure, frequent convolution calculation, high power consumption, and large memory occupation, the application introduces an SNN structure and utilizes the time-domain sparse activation characteristics of pulse coding to effectively reduce the calculation complexity and storage cost per sample.

[0071] In step S303, based on the preset sparse attention mechanism, visual perception is performed on the target time sequence feature by using a Transformer perception output head to obtain target detection and tracking results.

[0072] In some embodiments, the target detection and tracking results include at least one of a 3D bounding box of a target perception object, an object category, and a cross-frame target association ID.

[0073] Further, in some embodiments, based on the preset sparse attention mechanism, visual perception is performed on the target time sequence feature by using a Transformer perception output head to obtain target detection and tracking results, including: initializing a query vector, wherein each query vector corresponds to a 3D target instance; based on the preset sparse attention mechanism, performing multiple rounds of attention interaction between each query vector and the target time sequence feature to obtain the target detection and tracking results.

[0074] In the Transformer model, the query vector (Query) is a vector used to actively search for features, and each query corresponds to a candidate target instance.

[0075] Specifically, by using the preset sparse attention mechanism, the model performs multiple rounds of attention interaction between each query vector and the target time sequence feature extracted from the SNN-ResNet visual backbone network. In each round of interaction, the query vector updates itself according to the information in the target time sequence feature to better match the actual target instance.

[0076] After multiple rounds of attention interaction, the model outputs target detection and tracking results, which include the 3D bounding box and unique identifier (ID) of each target. The 3D bounding box provides the position and size information of the target in three-dimensional space, and the unique identifier is used to track the same target in consecutive frames, even if the target temporarily disappears from the field of view. The sparse attention mechanism ensures that the model can concentrate computing resources on the area most likely to contain the target, thereby significantly reducing the calculation complexity while maintaining high accuracy.

[0077] Thus, the sparse Transformer structure has been proved to have high precision in three-dimensional perception tasks, and the event-driven mechanism of SNN has stronger energy efficiency ratio, the present application combines the two to model the "double sparse mechanism" of sparse space sampling + sparse time response, realizes the efficient perception of the space-time joint compression of the input image. Through the fusion with the sparse Transformer structure, the unified space-time modeling method is used to fuse the output into the sparse Transformer structure after completing the pulse perception feature extraction, to optimize the attention query strategy with the space sparsity and time continuity, and realize the unified modeling of the 3D target detection and multi-target cross-frame tracking tasks.

[0078] In order to make the person skilled in the art more clearly and intuitively understand the visual perception method based on the pulse neural network provided in the present application, the following will be described in detail in combination with specific embodiments.

[0079] Firstly, the hardware and environment preparation of the embodiments of the present application is described. The hardware devices include a camera and a PC host. The PC host needs to have a GPU with high computing performance, support a deep learning framework such as TensorFlow or PyTorch, and the device of the embodiments of the present application can be an industrial computer with double 3090 graphics cards (single card 12GB).

[0080] Further, the software and tools include: ROS (Robot Operating System) as a data stream transmission and processing framework, a deep learning framework (TensorFlow or PyTorch), CUDA & TensorRT, and Opencv.

[0081] Further, the sparseend2end (snn) implementation code of the embodiments of the present application, the training time is 12 days, and the final snn tracking index AMOTA is 0.241, the vehicle detection AMOTA is 0.34, the detection index mAP is 0.202, and the AP of the vehicle is 0.453.

[0082] Further, the system of the embodiments of the present application is divided into three main parts: an input data expansion module, an SNN backbone network module, and a Transformer perception decoding module. Specifically as follows: (1) Time window expansion mechanism (T-step input modeling) The system replicates the static image input at time steps to form a T-dimensional time window tensor. For example, the input image dimension is [B, C, H, W], which is expanded to [T, B, C, H, W] after expansion, aiming to simulate the continuous transmission of information in the real nervous system over time. Each input at a time step is regarded as a "stimulus" received by a neuron, and the network needs to learn how to respond to the discharge under continuous stimulation.

[0083] It should be noted that the time series input of the embodiments of the present application is not limited to the form of replicating static images, but can also support the pulse input format generated by the DVS camera output or external encoder.

[0084] (2) LIF pulse neuron model The core activation unit adopts the LIF (Leaky Integrate-and-Fire) neuron model, which has the principle that the neuron has a "membrane potential" that continuously accumulates when continuously receiving input. If the potential exceeds a certain set threshold (called the discharge threshold), the neuron "fires" and outputs a pulse signal, and the membrane potential is reset to zero. If the threshold is not reached, the membrane potential will continue to accumulate, and the membrane potential will be exponentially decayed due to the influence of the time factor, which embodies the information forgetting mechanism. This activation mechanism can realize sparsity and time sensitivity, and has power consumption advantages on hardware, and is the basis of the SNN architecture.

[0085] Therefore, in the prior art, the time series information mainly depends on the fusion of the Transformer, and the perception backbone lacks time perception capability. The present application introduces a time window mechanism and a LIF neuron to construct a time series pulse response channel at the backbone level, so that the model has natural dynamic memory and pulse response capability.

[0086] It should be noted that in addition to using LIF activation during the training process, the embodiments of the present application can also use methods such as gated pulse neurons, analog pulse integration units, etc., to adjust the discharge frequency and energy consumption control strategy while maintaining sparsity and time sensitivity.

[0087] (3) Convolution and normalization module adaptation In the traditional CNN, the convolution layer extracts spatial features, and the BatchNorm layer normalizes the activation range to speed up the convergence. In the present application, the convolution operation needs to adapt to the time step dimension, that is, the convolution extraction is independently performed on T time steps. In order to maintain time consistency, the normalization layer is designed as a batch normalization (3D BN) that shares parameters across time steps to ensure stable neuron discharge.

[0088] (4) SNN-ResNet backbone structure The backbone network of the application still adopts the residual connection structure of ResNet to maintain the stability of deep features in the propagation process. The difference is that LIF spiking neurons are embedded as activation functions after all convolutional layers. The network as a whole processes the input in the time dimension T, and performs time average or maximum fusion operation in the last stage to change the feature of [T, B, C, H, W] back to [B, C, H, W], and finally output into the Transformer module.

[0089] The perception output module and system process are described in detail below.

[0090] The perception output part uses sparse Transformer (such as Sparse4Dv3 structure), and its core is sparse query mechanism and multi-frame information fusion capability. Its working process is as follows: receiving spatial feature information from the SNN-ResNet backbone network. Initialize the "query vector", each vector corresponds to a possible 3D target instance, and the query vector performs multi-round attention interaction with the image features through the Transformer module, so as to accurately locate the target position and category. If it is multi-frame input, the historical query information of the previous frame is fused to complete the time sequence association of the target, and realize 3D multi-target tracking. Among them, the time sequence fusion refers to the continuous frame information exchange between the continuous frames, realizing the continuous identification of the cross-frame target, and reducing the ID switching phenomenon.

[0091] As can be seen, the entire system process of the embodiment of the application is: multi-view image input→time step replication (T)→construction of pulse input sequence; SNN-ResNet module: multi-time step convolution+LIF neuron activation→feature sparse extraction; Time dimension feature fusion→output spatial feature tensor; Sparse Transformer module: generate target detection and tracking results according to the current features and historical instance library; The output is a 3D bounding box with ID and classification information for use by the upper layer system.

[0092] Thus, the ReLU activation function in the traditional convolutional neural network is replaced by the LIF neuron, and the time window encoding mechanism is used, and the neuron fires only when the membrane potential exceeds the threshold, to realize event-driven computing. Since most neurons remain in a silent state for most time steps, invalid calculations are significantly reduced, and the measured inference energy consumption is reduced by about 30%-60% under the condition of the same accuracy, and the energy saving effect is obvious. In addition, the LIF neuron has the ability of time integration and pulse output, and a natural time dynamic modeling mechanism is constructed in the visual perception backbone. Compared with the static features extracted by the traditional CNN, the structure can better preserve the temporal consistency and historical features of the target, and improve the accuracy of 3D target cross-frame matching and ID retention rate.

[0093] Further, the application adopts a parallel time window expansion mechanism, which replicates the static input into multiple time steps and processes it uniformly in the feature extraction stage, and cooperates with a structural sparse computing module to optimize the pipeline inference process. Experimental results show that the structure can complete the processing of multiple camera images with smaller delay on an embedded platform (such as NVIDIA Orin), effectively improving the system response speed. And the application retains the input and output tensor interface format of the standard CNN, and through customized pulse convolution, batch normalization and other modules, it ensures that the model can be completely exported to the ONNX format and supports running in the TensorRT engine. Cooperating with the optimized core operators (such as DeformableAttentionAggr) of CUDA, the end-to-end model can be efficiently deployed in the actual vehicle-mounted environment.

[0094] Thus, in view of the problems of high inference delay and operator incompatibility of traditional CNN models in embedded platform deployment, the application ensures that the input and output formats of the model are consistent with mainstream frameworks (such as ONNX, TensorRT) through deep fusion design of the SNN structure, and avoids complex nested operations in the critical path, improving the deployment efficiency and platform universality of the model.

[0095] At the same time, the pulse encoding backbone module proposed by the application can seamlessly replace various standard visual backbones (such as ResNet, ResNeXt), and can be directly integrated into existing sparse perception architectures (such as Sparse4Dv3, SparseEnd2End, etc.), and has good scalability and migration adaptability.

[0096] According to the spiking neural network-based visual perception method of the present application, an input module is used to acquire and preprocess multi-source image data to obtain an image tensor containing a time dimension. The SNN-ResNet visual backbone network is used to extract features from the image tensor to obtain target temporal features. Based on a preset sparse attention mechanism, the Transformer perception output head is used to perform visual perception based on the target temporal features to obtain target detection and tracking results. This solves the problems of high power consumption, insufficient temporal modeling capabilities, and slow inference speed of traditional visual perception systems, achieving efficient perception with low power consumption, high temporal dynamic modeling capabilities, and fast response.

[0097] Next, a visual perception device based on a pulse neural network according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0098] In this embodiment, the visual perception device based on the pulse neural network adopts a visual perception system based on the pulse neural network, and the system includes an input module, an SNN-ResNet visual backbone network, and a Transformer perception output head.

[0099] Figure 4 It is a block diagram of a visual perception device based on a pulse neural network according to an embodiment of the present application.

[0100] like Figure 4 As shown, the visual perception device 10 based on the pulse neural network includes: an acquisition module 100, a feature extraction module 200 and a perception module 300.

[0101] Among them, the acquisition module 100 is used to use the input module to acquire multi-source image data, and preprocess the multi-source image data to obtain an image tensor containing a time dimension; the feature extraction module 200 is used to use the SNN-ResNet visual backbone network to extract features from the image tensor to obtain target temporal features; the perception module 300 is used to perform visual perception based on the target temporal features using the Transformer perception output head based on the preset sparse attention mechanism to obtain target detection and tracking results.

[0102] Furthermore, in some embodiments, the acquisition module 100 includes: performing synchronization and alignment operations on each frame of the multi-source image to obtain an aligned multi-source image; and expanding each frame of the aligned multi-source image to a preset number of time steps to obtain an image tensor containing a time dimension.

[0103] Further, in some embodiments, the feature extraction module 200 is configured to: perform feature extraction on the image tensor of each time step by using a convolutional layer of the SNN-ResNet visual backbone network to obtain spatial features of each time step; simulate a neuron membrane potential accumulation and / or a firing process according to the spatial features of each time step by using a LIF spiking neuron activation module of the SNN-ResNet visual backbone network to obtain sparse features; perform normalization operation on the sparse features by using a normalization layer of the SNN-ResNet visual backbone network to obtain normalized features; and fuse the normalized features of all time steps to obtain the target temporal features.

[0104] Further, in some embodiments, before fusing the normalized features of all time steps, the feature extraction module 200 is further configured to: connect the plurality of spiking neural networks by using a residual connection module of the SNN-ResNet visual backbone network.

[0105] Further, in some embodiments, the perception module 300 is configured to: initialize query vectors, wherein each query vector corresponds to one 3D target instance; and perform multi-round attention interaction between each query vector and the target temporal features based on a preset sparse attention mechanism to obtain the target detection and tracking result.

[0106] Further, in some embodiments, the target detection and tracking result includes at least one of a 3D bounding box of a target perceived object, an object category, and a cross-frame target association ID.

[0107] It should be noted that the foregoing explanation of the embodiment of the method for visual perception based on a spiking neural network also applies to the embodiment of the device for visual perception based on a spiking neural network, which will not be described here again.

[0108] The device for visual perception based on a spiking neural network according to the embodiment of the present application acquires multi-source image data by using an input module and pre-processes the multi-source image data to obtain an image tensor containing a time dimension; extracts features from the image tensor by using an SNN-ResNet visual backbone network to obtain target temporal features; and performs visual perception according to the target temporal features by using a Transformer perception output head based on a preset sparse attention mechanism to obtain a target detection and tracking result. Thus, the problems of high power consumption, insufficient time modeling capability, and slow inference speed of a traditional visual perception system are solved, and efficient perception with low power consumption, high time dynamic modeling capability, and fast response is achieved.

[0109] Figure 5 A structural schematic diagram of an electronic device provided by the embodiment of the present application is shown. The electronic device can include: The memory 501, the processor 502 and the computer program stored in the memory 501 and executable on the processor 502.

[0110] The processor 502 implements the method of visual perception based on the pulse neural network provided in the above embodiments when executing the program.

[0111] Further, the electronic device further comprises: The communication interface 503 is configured to communicate between the memory 501 and the processor 502.

[0112] The memory 501 is configured to store the computer program executable on the processor 502.

[0113] The memory 501 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0114] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected through a bus and complete the communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 5 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0115] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can complete the communication between each other through an internal interface.

[0116] The processor 502 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0117] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the above-mentioned visual perception method based on a pulse neural network.

[0118] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0119] In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0120] Although the embodiments of the present application have been shown and described above, it can be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and the person skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A visual perception method based on a spiking neural network, characterized in that: The method uses a visual perception system based on a spiking neural network, which includes an input module, an SNN-ResNet visual backbone network, and a Transformer perception output head, and includes the following steps: Acquiring multi-source image data using the input module, and preprocessing the multi-source image data to obtain an image tensor including a time dimension; Using the SNN-ResNet visual backbone network to extract features from the image tensor to obtain target temporal features; Based on the preset sparse attention mechanism, the Transformer perception output head is used to perform visual perception according to the target temporal characteristics to obtain target detection and tracking results.

2. The method according to claim 1, characterized in that The preprocessing of the multi-source image data to obtain an image tensor including a time dimension includes: Perform synchronization and alignment operations on each frame of multi-source image to obtain an aligned multi-source image; Each frame of the aligned multi-source image is expanded to a preset number of time steps to obtain the image tensor containing the time dimension.

3. The method according to claim 1, characterized in that The method of extracting features from the image tensor using the SNN-ResNet visual backbone network to obtain target temporal features includes: Using the convolutional layer of the SNN-ResNet visual backbone network to extract features from the image tensor at each time step, obtaining spatial features at each time step; Utilizing the LIF spike neuron activation module of the SNN-ResNet visual backbone network, simulating the neuron membrane potential accumulation and / or discharge process according to the spatial features of each time step to obtain sparse features; Performing a normalization operation on the sparse features using the normalization layer of the SNN-ResNet visual backbone network to obtain normalized features; The normalized features of all time steps are fused to obtain the target time series features.

4. The method according to claim 3, characterized in that Before fusing the normalized features of all time steps, it also includes: The residual connection module of the SNN-ResNet visual backbone network is used to connect multiple spiking neural networks.

5. The method according to claim 1, wherein The preset sparse attention mechanism is based on the Transformer perception output head to perform visual perception according to the target temporal characteristics to obtain target detection and tracking results, including: Initialize query vectors, where each query vector corresponds to a 3D object instance; Based on a preset sparse attention mechanism, multiple rounds of attention interaction are performed between each query vector and the target temporal features to obtain the target detection and tracking results.

6. The method according to claim 1, characterized in that The target detection and tracking result includes at least one of a 3D bounding box of a target-perceived object, an object category, and a cross-frame target association ID.

7. A visual perception device based on a spiking neural network, characterized in that: The device adopts a visual perception system based on a spiking neural network. The system includes an input module, an SNN-ResNet visual backbone network, and a Transformer perception output head, including: an acquisition module, configured to acquire multi-source image data using the input module, and preprocess the multi-source image data to obtain an image tensor including a time dimension; A feature extraction module is used to extract features from the image tensor using the SNN-ResNet visual backbone network to obtain target temporal features; The perception module is used to perform visual perception based on the target timing characteristics using the Transformer perception output head based on a preset sparse attention mechanism to obtain target detection and tracking results.

8. The device according to claim 7, characterized in that The acquisition module includes: Perform synchronization and alignment operations on each frame of multi-source image to obtain an aligned multi-source image; Each frame of the aligned multi-source image is expanded to a preset number of time steps to obtain the image tensor containing the time dimension.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visual perception method based on a pulse neural network as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the visual perception method based on a pulse neural network as described in any one of claims 1 to 6.

Images