A semantic segmentation method based on event and spiking neural network
Through the method based on event and pulse neural network, the problem of semantic segmentation method relying on additional visual information and high energy consumption is solved, and efficient and accurate semantic segmentation is achieved, which is suitable for platforms with limited resources.
Patent Information
- Application Number
- CN202410145315.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-31
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-01-31
AI Technical Summary
The existing semantic segmentation method needs to rely on additional visual information, discard the time dimension information of the event, and consume high energy, limiting its deployment on platforms with limited resources and energy.
Using an event and pulse neural network method, high-dimensional semantic features are extracted and calculated in two dimensions of time and space through pre-processing of asynchronous event data, boundary-aware pulse encoder and pulse decoder, to realize semantic prediction. Reliance on additional visual information is avoided and computational efficiency is improved through pulsed neurons.
Semantic segmentation without relying on additional visual information is achieved, with high computing efficiency, accuracy and low energy consumption, and is better than traditional methods on multiple evaluation indicators.
Smart Images

Figure CN117934850B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of computer vision and provides a semantic segmentation method based on event and pulse neural network. Background Art
[0002] Semantic segmentation aims to assign a semantic label to each pixel in an image, which helps to understand and analyze the environment and is of great significance in the field of computer vision. It has wide applications in autonomous driving, robot perception, and video surveillance. The introduction of fully convolutional networks has laid a basic paradigm for the task of semantic segmentation. Based on this paradigm, many papers have proposed higher quality methods. However, these studies mainly focus on images captured by frame-based cameras. Although frame-based cameras are sufficient for daily use, they have practical difficulties in coping with real-time requirements, poor lighting conditions, and accurate recognition of moving objects.
[0003] In recent years, event cameras have attracted increasing attention due to their unique biomimetic characteristics. Unlike traditional cameras that capture images at a fixed rate, event cameras asynchronously measure the brightness change of each pixel and generate events when the brightness change exceeds a predefined threshold. Event cameras have four advantages over traditional cameras: higher temporal resolution, lower time delay, higher dynamic range, and lower energy consumption. Therefore, event-based semantic segmentation tasks have practical research significance and application prospects. In response to this research, some works have made initial progress. EV-SegNet is the first work to use event data for semantic segmentation. It proposes an innovative event representation method 6-channel and introduces how to automatically generate semantic labels for events. EvDistill learns from a teacher network trained on large-scale labeled image data (source modality) through knowledge distillation to handle unlabeled and unpaired event data (target modality). The recently proposed ESS directly transfers the semantic segmentation task from existing labeled image datasets to unlabeled event data through unsupervised domain adaptation. However, these works still have some shortcomings: the need for additional image information assistance, the characteristics of event data are not fully utilized, the network complexity is high, and the energy consumption is high. Therefore, it is necessary to propose an event-based semantic segmentation method that does not rely on additional visual information, fully exploits the characteristics of event data, and is accurate, efficient, and energy-efficient.
[0004] Existing semantic segmentation methods have one or more of the following problems: 1. They need to obtain texture details from additional visual information (grayscale images or RGB images) through techniques such as knowledge distillation and transfer learning to compensate for the sparse features of event data. 2. They rely on traditional artificial neural networks, which can only transfer features in the spatial dimension and discard the temporal dimension of events. 3. They have high energy consumption, which limits their deployment on platforms with limited resources and energy. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a semantic segmentation method based on event and pulse neural network, aiming to solve the problems that the existing semantic segmentation method needs to rely on additional visual information, discards the time dimension information of events, and has high energy consumption.
[0006] To achieve the above object, the present invention provides the following technical solution: a semantic segmentation method based on event and pulse neural network, comprising the following steps:
[0007] Step S1: preprocessing the asynchronous event data;
[0008] Step S2: input the preprocessed events into a boundary-aware pulse encoder;
[0009] Step S3: Input the high-dimensional semantic features extracted by the encoder into the pulse decoder for final semantic prediction.
[0010] Further, step S1 includes: in the event preprocessing stage, the asynchronous event stream e i ={(x i ,y i ,t i ,p i )} i∈W First, voxel grid encoding is performed, followed by adjustment to the time dimension, where (x i ,y i ) is the pixel coordinate of the event, t i is the timestamp of the event, p i is the polarity of the event, p i = ±1;
[0011] The final representation is a sequence of events Where W is the preset time window, T is the time step, E∈R X ×Y×C , X×Y and C are the spatial resolution and the number of channels, respectively.
[0012] Furthermore, step S2 includes: inputting the preprocessed event sequence E into the pulse encoder for multi-dimensional feature extraction, wherein the pulse encoder includes three branches: a semantic detail branch, a context aggregation branch, and a boundary warning branch, and integrating the context aggregation branch into the semantic detail branch through addition to guide low-dimensional feature learning from a high-dimensional perspective; at the same time, the pulse encoder is combined with pulse neurons to convert the basic computing operation from energy-consuming floating-point matrix multiplication to addition, and calculate and transmit event data in two dimensions of time and space.
[0013] Further, step S3 includes: inputting the high-dimensional semantic features extracted by the encoder into the pulse decoder; the pulse decoder first fuses the features of the three branches through an addition operation, then restores them to the original size through upsampling, and finally performs semantic prediction through a prediction head; the pulse decoder combines pulse neurons.
[0014] The remarkable effects of the present invention are:
[0015] The semantic segmentation method based on events and pulse neural networks of the present invention does not rely on additional visual information; it calculates and transmits event data in two dimensions of time and space; and it has the characteristics of high computational efficiency, high accuracy and low energy consumption of pulse neural networks.
[0016] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0018] Figure 1 It is the overall structure diagram of the present invention;
[0019] Figure 2 It is a schematic diagram of the event preprocessing method in the present invention;
[0020] Figure 3 This is a visualization diagram of the segmentation results of some samples on the DDD17 dataset of the present invention;
[0021] Figure 4 The figure is a flow chart of the overall method of the present invention. DETAILED DESCRIPTION
[0022] The present invention is further described below in conjunction with specific implementation methods. The accompanying drawings are only used for exemplary descriptions and are only schematic diagrams, not actual drawings, and cannot be understood as limiting this patent; in order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the accompanying drawings may be omitted.
[0023] like Figure 1-Figure 4 As shown, the present invention provides an innovative event-based, boundary region-guided semantic segmentation network, and combines the efficient computational efficiency of pulse neural networks to achieve an accurate, efficient and low-power semantic segmentation method. This method has three benefits: 1. It can predict semantic labels only from event data without the assistance of additional visual information. 2. The designed boundary region-guided segmentation network extracts event features from three aspects: semantic details, contextual information and boundaries, respectively, to fully mine event information. 3. Combined with pulse neural networks, energy savings and efficient computing for time-sensitive applications are achieved.
[0024] Figure 4 The flowchart of the overall method of the present invention is a semantic segmentation method based on event and pulse neural network, comprising the following steps:
[0025] Step S1: pre-process the asynchronous event data.
[0026] Step S2: Input the preprocessed events into the boundary-aware pulse encoder.
[0027] Step S3: Input the high-dimensional semantic features extracted by the encoder into the pulse decoder for final semantic prediction.
[0028] Furthermore, in the event preprocessing phase, the asynchronous event stream e i ={(x i ,y i ,t i ,p i )} i∈W First, voxel grid encoding is performed, then it is adjusted to adapt to the time dimension, and finally represented as a time series Where W is the preset time window, T is the time step, E∈R X×Y×C , X×Y and C are the spatial resolution and the number of channels, respectively.
[0029] Furthermore, the preprocessed event sequence E is input into the pulse encoder for multi-dimensional feature extraction. Specifically, the pulse encoder contains three branches: semantic detail branch, context aggregation branch and boundary alert branch, and these three branches are not completely independent. Since the context aggregation branch contains high-level semantic information of event data, it can effectively guide the extraction of semantic detail information. Therefore, the context aggregation branch is integrated into the semantic detail branch through simple addition to guide low-dimensional feature learning from a high-dimensional perspective. At the same time, the encoder combines pulse neurons to convert the basic computing operation from energy-consuming floating-point matrix multiplication to addition, and calculates and transmits event data in two dimensions of time and space.
[0030] Furthermore, the high-dimensional semantic features extracted by the encoder are input into the pulse decoder. The decoder first fuses the features of the three branches through a simple addition operation, then restores them to their original size through upsampling, and finally performs semantic prediction through the prediction head. Similarly, the decoder also combines pulse neurons.
[0031] The present invention is a semantic segmentation method based on event and pulse neural networks. It does not rely on additional visual information (such as images), can cope well with changes in ambient lighting conditions, and has the characteristics of high computational efficiency, high accuracy, and low energy consumption. Here, the proposed method is evaluated by the public event camera autonomous driving data sets DDD17 and DSEC-Semantic. The following comparisons are made during the evaluation process: the segmentation accuracy and parameter amount of the method proposed in the present invention are compared with other advanced ANN or SNN-based event semantic segmentation methods on the data sets DDD17 and DSEC-Semantic. The comparison results are shown in Tables 1 and 2, and the present invention is superior to all SNN-based methods. Specifically, with a parameter amount of 7.68M, the present invention achieves 54.72% and 54.76% mIoU in DDD17 and DSEC-Semantic, respectively, exceeding the most advanced SNN methods by 1.57% and 1.72%, respectively.
[0032] The energy consumption of the method proposed in the present invention is compared with other advanced event semantic methods based on ANN or SNN on the data set DDD17. The comparison results are shown in Table 3. Compared with the ANN model, the present invention can achieve up to 17.76 times less energy consumption.
[0033] Table 1 Comparison of the accuracy and parameter amount of the event semantic segmentation model of the present invention and different ANN / SNN models on the DDD17 dataset:
[0034]
[0035] Table 2 Comparison of the accuracy and parameter amount of the event semantic segmentation model of the present invention and different ANN / SNN models on the DSEC-Semantic dataset:
[0036]
[0037] Table 3 Comparison of energy consumption of the present invention and different ANN / SNN event semantic segmentation models on the DDD17 dataset:
[0038]
[0039] Table 4 Ablation experiment for the event characterization method of the present invention:
[0040]
[0041] Table 5 shows the ablation experiment of the loss function designed by the present invention:
[0042]
[0043] Figure 1 It is a schematic diagram of the overall structure of the present invention; Figure 2 Schematic diagram of the event preprocessing method in the present invention; Figure 3 A schematic diagram showing the visualization of the segmentation results of some samples in the DDD17 dataset.
[0044] The following will refer to the attached Figure 1-Figure 3 , the preferred embodiments of the present invention are described in detail. It should be understood that the preferred embodiments are only for the purpose of using the present invention, rather than for limiting the protection scope of the present invention.
[0045] The overall structure of the present invention is as follows Figure 1 As shown. It uses an asynchronous event stream i ={(x i ,y i ,t i ,p i )} i∈W As input, it is converted into an event sequence after preprocessing Input the event sequence and calculate the predicted probability of each pixel Among them C class Indicates the number of categories; Figure 2 The event preprocessing method of the present invention is presented in detail. In order to meet the characteristics of the spiking neural network, we transform the asynchronous event stream e i ={(x i ,y i ,t i ,p i )} i∈WEncoded into a voxel grid E(x,y,t). Specifically, we first set the time window W according to the requirements to divide the event stream. Second, we proportionally assign each event to adjacent time bins based on normalized time, thereby scaling each event in W to B time bins. This process can be formalized as:
[0046]
[0047]
[0048] Wherein, f(x)=max(0,1-|x|). In the present invention, we set B=5, W=50ms.
[0049] The preprocessed events are used as the input of the semantic segmentation model to complete semantic prediction. Figure 1 As shown, the present invention follows the classic semantic segmentation paradigm and includes two modules: feature encoder and decoder. In the encoding stage, first, the preprocessed event sequence The feature extraction is preprocessed through three downsampling convolutional layers. Then, the encoder extracts event features through three branches: 1. Semantic detail branch: maintain high resolution to protect spatial and texture information; 2. Context aggregation branch: capture rich global dependencies with downsampling strategy; 3. Boundary alert branch: focus on identifying the boundary information of objects. Each branch is stacked by residual blocks proposed by MS-ResNet. Considering the rich semantic information contained in the context aggregation branch, we use the addition operation to supplement the context features to the semantic detail branch, so as to achieve the use of high-dimensional semantic information to assist low-dimensional detail analysis. In the decoder module, the features extracted by the three branches are fused through a simple addition operation. To improve computational efficiency, we use bilinear interpolation to restore the fused features to the original resolution. In the prediction head, we use the average of the membrane potential of the last spike neuron in all time dimensions to predict the category of the pixel.
[0050] In order to make full use of the event-driven nature of spiking neural networks, the present invention incorporates the spiking paradigm into the segmentation model by using spiking neurons after the convolutional layer. This method converts floating-point values transmitted in the spatial dimension into binary spiking signals transmitted in the spatiotemporal dimension. We use the LIF model, which is a popular model for describing the dynamics of spiking neurons and has a good balance between biological interpretability and computational efficiency. After integrating the LIF model, the forward propagation process of the overall model can be expressed as:
[0051]
[0052]
[0053]
[0054] in, Indicates that at timestamp t, th The membrane potential of the neurons in the layer, is the spatial feature obtained by calculation, W l Yes th The synaptic weights of the layer, is the pulse output of the current layer at time t, τ is the leakage constant, U th is the ignition threshold, H(·) is the Heaviside step function, in which H(x)=1 when x≥0, otherwise H(x)=0.
[0055] In order to train the network proposed in the present invention, we designed a loss function specifically according to the pulse characteristics of the network and the input data. The cross entropy loss l0 is used to measure the degree of difference between the predicted probability of the output layer and the one-hot true value label. In order to promote the rapid convergence of the semantic detail branch in the network, we used the early semantic loss l1. Unlike the method proposed by J.Xu et al., due to the ambiguity of event boundaries, we do not use binary cross entropy to implement hard boundary constraints. Instead, we use the boundary-aware cross entropy loss l2 to achieve soft constraints on the boundary by semantically optimizing the boundary and its neighborhood. Specifically, if the confidence that a pixel is predicted to be a boundary exceeds the preset threshold θ, then we will include it in the subsequent model optimization steps. The complete loss is:
[0056] Loss=γ0l0+γ1l1+γ2l2#(6)
[0057] In the present invention, the best settings are γ0=1, γ1=0.4, γ2=1 and θ=0.6.
[0058] Due to the non-differentiability of the pulse generating function, we cannot directly use the error back propagation method. To illustrate this problem, we give a formula for the gradient calculation in back propagation:
[0059]
[0060] Among them, L represents the loss function, W t represents the weight matrix, S t and U t They represent the pulse output and membrane potential of spiking neurons respectively. Represents the gradient of the pulse activation function, which is in U th This leads to a training dilemma: the weight update is either stagnant or updated in an infinite series.
[0061] To solve this problem, we use an evolution asymptotic function EvAF to approximate the derivative of the impulse function, namely:
[0062]
[0063] Where K(i) is a coefficient and will change dynamically during the training process:
[0064]
[0065] Among them, K min and K max denote the lower and upper bounds of K respectively, and i represents the number of training iterations from 0 to N-1. In the present invention, the best setting is K min =1, K max =10.
[0066] In this example, we use the SpikingJelly framework to implement the model and set the reset parameter V of the LIF model. reset =0, membrane potential leakage constant τ = 0.25, pulse emission threshold V th = 0.5. We use the publicly available event-camera-based autonomous driving semantic segmentation datasets DDD17 and DSEC-Semantic to evaluate the proposed method.
[0067] For the DDD17 dataset: In the event representation part, we transform the event into the model input X∈R 5×1×200×352 , where T = 5, C = 1. In training, we use Adam to optimize the model on the training dataset, with the initial learning rate set to 0.001, the weight decay parameter set to 0.0001, the number of training iterations set to 60, and the batch size set to 32. In addition, we use the StepLR learning rate update strategy and set step_size = 5. R is the reshape operation.
[0068] For the DSEC-Semantic dataset: In the event representation part, we transform the event into the model input X∈R 5 ×1×440×640 , where T = 5, C = 1. In training, we use Adam to optimize the model on the training dataset, with the initial learning rate set to 0.05, the weight decay parameter set to 0.001, the number of training iterations set to 150, and the batch size set to 20. In addition, we use the StepLR learning rate update strategy and set step_size = 10.
[0069] The following comparisons and analyses were performed during the evaluation of the two datasets:
[0070] The segmentation effectiveness of the proposed method and different ANN / SNN methods on the datasets DDD17 and DSEC-Semantic is compared, and the evaluation index is MIoU. The comparison results of the present invention on the DDD17 dataset are shown in Table 1. When compared with other SNN-based methods, the present invention achieved the best result with 54.72% MIoU. Specifically, the present invention exceeds Spiking-DeepLab and Spiking-FCN by 51.02% and 20.50%, respectively. It is worth noting that the present invention is 1.57% MIoU higher than the recently proposed SSAM-SNN, while having fewer parameters. Next, we analyze the comparison results in the ANN field. The present invention and EV-SegNet can achieve comparable performance. Although the results of the present invention are worse than the ANN-based ESS and EVdistill models, both of which use transfer learning techniques to learn rich features from the image field, while the present invention only needs to learn from events without relying on other visual information. In addition, compared with the SNN-based methods, the computational overhead of the ANN-based methods is very expensive, and the specific comparison will be described later. In addition, we Figure 3 The segmentation results on individual samples of the DDD17 dataset are visualized and compared with the best model ESS. It can be seen that the segmentation effect of the present invention is comparable to that of ESS. The comparison results of the present invention on the DSEC-Semantic dataset are shown in Table 2. The present invention achieves the best segmentation result among all methods with 54.76% MIoU. Specifically, the present invention exceeds Ev-SegNet, ESS and SSAM-SNN by 3%, 3.19% and 1.72% MIoU respectively. These results prove that the pulse boundary alert branch can effectively guide the segmentation head to improve the object boundaries.
[0071] The energy consumption of the proposed method and different ANN / SNN methods on the dataset DDD17 is compared. In ANN, we usually use floating-point operands to quantify the computational overhead and adopt floating-point multiplication and addition (MAC) operations to evaluate the energy consumption. Due to the pulse drive and bionic characteristics of SNN, the spiking neurons will output sparse binary pulses, which will convert the MAC in ANN into the accumulation (ACC) operation in SNN. Therefore, we combine the pulse firing rate γ and the time step T to evaluate the energy consumption of SNN, which is expressed as follows:
[0072] E=E MAC ×FL C +E ACC ×FL C ×T×γ#(10)
[0073] Among them, in the convolutional layer, FL C =k 2 ×Cin ×C out ×H out ×W out .
[0074] To be consistent with previous work, we assume that the MAC of 32-bit floating point numbers in ANN consumes 4.6pJ of energy and the ACC of 32-bit floating point numbers in SNN consumes 0.9pJ of energy. Table 3 shows the energy consumption comparison results. Compared with the ANN-based model, the energy consumption overhead of the present invention is much smaller. Specifically, Ev-SegNet, Evdistill and ESS consume 5.57×, 17.76× and 6.99× times more energy than the present invention, respectively. Although SSAM is also an SNN-based architecture, its continuous four input streams lead to an increase of 3.54× in energy cost.
[0075] The impact of event preprocessing methods on the performance of the present invention is analyzed. Different event representation methods encode information in different dimensions and densities, which will greatly affect the segmentation performance. Table 4 shows the performance comparison of different event representation methods on the DDD17 dataset. The compared representation methods include: time-based stacking (SBT), event-based stacking (SBE), event frame, six-channel, voxel grid with fixed number of events (VoxelGrid(E)) and the method adopted by the present invention. Using SBT, SBE and Voxel Grid(E), the preprocessed event is expressed as E∈R H×W×C As with the method used in this invention, we retain its time dimension and set the number of channels C = 1 at each time step. The event frame accumulates events in a specific time interval in a pixel-level manner to generate a two-dimensional histogram h + (x,y) and h - (x, y). The 6-channel representation is based on the event frame and additionally stores the mean and variance of the normalized timestamp. As the input of the model, the above two representation methods also need to perform broadcast operations in the time dimension and retain the original number of channels. It can be seen from the table that when processed with the same model, the representation method used in the present invention exceeds other methods by at least 1.07% MIoU.
[0076] The effects of different loss function settings on the performance of the present invention are analyzed. As shown in Table 5, we explored the effects of early semantic loss l1 and boundary-aware cross entropy loss l2 on network performance. As can be seen from Table 5, removing l1 results in a significant drop in performance (-1.96% MIoU) because it can effectively help the network learn abstract semantic features at an early stage. It is worth noting that the optimization effect of l2 only becomes apparent in the presence of l1 (+0.82% MIoU) because l2 relies on effective spatial information for boundary optimization. This is reasonable because l2 aims to selectively enhance features around boundary regions. If the semantic information is weak, enhancing semantics only in boundary regions does not effectively enhance boundary recognition.
[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A semantic segmentation method based on event and pulse neural network, characterized in that: The method comprises the following steps: step S1: preprocessing the asynchronous event data; step S2: inputting the preprocessed events into a boundary-aware pulse encoder; step S3: inputting the high-dimensional semantic features extracted by the encoder into a pulse decoder, and finally performing semantic prediction; Step S2 includes: inputting the preprocessed event sequence E into a pulse encoder for multi-dimensional feature extraction, wherein the pulse encoder includes three branches: a semantic detail branch, a context aggregation branch, and a boundary alert branch, and integrating the context aggregation branch into the semantic detail branch through addition to guide low-dimensional feature learning from a high-dimensional perspective; at the same time, the pulse encoder combines with pulse neurons to convert the basic computing operation from energy-consuming floating-point matrix multiplication to addition, and calculates and transmits event data in two dimensions of time and space; The preprocessed events are used as the input of the semantic segmentation model to complete semantic prediction; following the classic semantic segmentation paradigm, it contains two modules: feature encoder and decoder; in the encoding stage, first, the preprocessed event sequence The feature extraction is preprocessed through three downsampling convolutional layers; then, the encoder extracts event features through three branches: 1). Semantic detail branch: maintain high resolution to protect spatial and texture information; 2). Context aggregation branch: Use downsampling strategy to capture rich global dependencies; 3) Boundary alert branch: focuses on identifying the boundary information of objects; Each branch is stacked by the residual blocks proposed by MS-ResNet; considering the rich semantic information contained in the context aggregation branch, the context features are added to the semantic detail branch using the addition operation to achieve the use of high-dimensional semantic information to assist low-dimensional detail analysis; the context features are added to the boundary alert branch using the addition operation; in the decoder, the features extracted by the three branches are fused through the addition operation; to improve computational efficiency, bilinear interpolation is used to restore the fused features to the original resolution; in the prediction head, the average value of the membrane potential of the last spike neuron in all time dimensions is used to predict the category of the pixel; A loss function is designed specifically according to the pulse characteristics of the network and the input data. The cross entropy loss l0 is used to measure the difference between the predicted probability of the output layer and the one-hot true value label. In order to promote the rapid convergence of the semantic detail branch in the network, the early semantic loss l1 is used. The boundary-aware cross entropy loss l2 is used to achieve soft constraints on the boundary by semantically optimizing the boundary and its neighborhood. The specific operation of the loss l2 is that if the confidence of predicting a pixel as a boundary exceeds the preset threshold θ, it will be included in the subsequent model optimization steps. The complete loss is: Loss=γ0l0+γ1l1+γ2l2#(6) Among them, γ0=1, γ1=0.4, γ2=1 and θ=0.
6.
2. The semantic segmentation method based on event and pulse neural network according to claim 1, characterized in that: Step S1 includes: in the event preprocessing stage, the asynchronous event stream e i ={(x i ,y i ,t i ,p i )} i∈W First, voxel grid encoding is performed, followed by adjustment to the time dimension, where (x i ,y i ) is the pixel coordinate of the event, t i is the timestamp of the event, p i is the polarity of the event, p i = ±1; The final representation is a sequence of events Where W is the preset time window, T is the time step, E∈R X×Y×C , X×Y and C are the spatial resolution and the number of channels, respectively.
3. The semantic segmentation method based on event and pulse neural network according to claim 2 is characterized in that: Step S3 includes: inputting the high-dimensional semantic features extracted by the encoder into the pulse decoder; the pulse decoder first fuses the features of the three branches through an addition operation, then restores them to the original size through upsampling, and finally performs semantic prediction through the prediction head; the pulse decoder combines pulse neurons.
4. The semantic segmentation method based on event and pulse neural network according to claim 3 is characterized in that: In step S1, the asynchronous event stream e i ={(x i ,y i ,t i ,p i )} i∈W As input, it is converted into an event sequence after preprocessing Input the event sequence and calculate the predicted probability of each pixel Among them C class Indicates the number of categories, and converts the asynchronous event stream e i ={(x i ,y i ,t i ,p i )} i∈W Encode into a voxel grid E(x,y,t); set the time window W according to the requirements to divide the event stream; secondly, assign each event to adjacent time bins based on normalized time in proportion, so as to scale each event in W to B time bins; this process can be formulated as: Where f(x)=max(0,1-|x|).
5. The semantic segmentation method based on event and pulse neural network according to claim 4, characterized in that: In order to make full use of the event-driven nature of spiking neural networks, the spiking paradigm is incorporated into the segmentation model by using spiking neurons after the convolutional layer; the floating-point values transmitted in the spatial dimension are converted into binary spiking signals transmitted in the spatiotemporal dimension; Using the LIF model, the LIF model is a model that describes the dynamics of spiking neurons. After integrating the LIF model, the forward propagation process of the overall model can be expressed as: in, Indicates that at timestamp t, th The membrane potential of the neurons in the layer, is the spatial feature obtained by calculation, W l Yes th The synaptic weights of the layer, is the pulse output of the current layer at time t, τ is the leakage constant, U th is the ignition threshold, H(·) is the Heaviside step function, in which H(x)=1 when x≥0, otherwise H(x)=0.
6. The semantic segmentation method based on event and pulse neural network according to claim 5, characterized in that: Due to the non-differentiability of the pulse generating function, a formula for gradient calculation in back propagation is given: Among them, L represents the loss function, W t represents the weight matrix, S t and U t represent the pulse output and membrane potential of spiking neurons, respectively; Represents the gradient of the pulse activation function, which is in U th It is infinite when , and 0 at other times; An evolution asymptotic function EvAF is used to approximate the derivative of the impulse function, namely: Where K(i) is a coefficient and will change dynamically during the training process: Among them, K min and K max They represent the lower and upper bounds of K respectively, and i represents the number of training iterations from 0 to N-1.
Citation Information
Patent Citations
Neuromorphic visual target classification method based on improved spiking neural network
CN112699956A
Neuromorphic visual target classification system based on input weighted spiking neural network
CN112712170A
Three-dimensional indoor scene semantic segmentation method based on residual pulse neural network
CN116958557A
Target segmentation method and system based on spiking neural network
CN117253039A
