A target segmentation method and system based on pulse neural network

Abstract

The application discloses a target segmentation method and system based on a pulse neural network, and belongs to the field of target segmentation. The application converts continuous event pulse streams into input representations suitable for the pulse neural network through an event pulse conversion relationship; improves the feature extraction capability of a target segmentation model for continuous event pulse inputs and reduces energy consumption by extracting information and memorizing past features through pulse neurons; improves the utilization rate of the target segmentation model for features by fusing multi-time step features through a space-time feature aggregation module; improves the fusion capability of the target segmentation model for coarse-grained and fine-grained semantic information through a multi-scale feature fusion module, and uses multi-time step loss to constrain the convergence of the target segmentation model, reduce the dependence of the target segmentation model on time steps, and improve the robustness of the target segmentation model. The application can not only process continuous space-time event pulse streams with low power consumption, but also obtain accurate segmentation results in low-light and high-speed scenes.

Description

TECHNICAL FIELD

[0001] The application relates to a target segmentation method and system based on a pulse neural network, in particular to a target segmentation method with low power consumption and high speed in an environment, and belongs to the field of target segmentation. BACKGROUND

[0002] Target segmentation is a basic task of computer vision and has been widely used in many fields such as target detection and tracking, automatic driving and human-computer interaction. In recent years, the use of deep learning and neural networks (ANN) to solve the target segmentation problem has made a lot of progress and has gradually become the mainstream solution. However, the existing neural network methods with good performance often face the dilemma of high power consumption and have become more and more unable to meet the existing application requirements; the main reason is that the network internally uses floating-point numbers for calculation, although there is a single-precision model, but the excessive consumption of energy cannot be avoided. On the other hand, the traditional camera based on visual sensor to obtain picture modal data cannot obtain good segmentation results when facing low light, high dynamic and high speed scenes; these sensors usually sample the scene according to the preset frequency, and this sampling method cannot adapt to the dynamic changes of the scene, resulting in that in the low light scene, not enough information is captured; at the same time, the pictures collected are not continuous in time, and the inter-frame motion information is not recorded, which often causes serious dynamic blur in high-speed scenes, which has an adverse effect on target segmentation.

[0003] According to the principle of the visual sampling mechanism of the biological retina, a series of new cameras for collecting pulse array signals have gradually entered the field of vision, including sensors that emit pulse signals according to changes in light intensity, such as dynamic vision sensors (DVS), asynchronous time-based image sensors (ATIS), dynamic and active pixel vision sensors (DAVIS), and the like; sensors that emit signals based on the cumulative intensity of light intensity, such as pulse cameras (Vidar) and the like. Unlike the sampling method of traditional cameras, the sensor of such a camera collects information of light signals in a certain time and area, has the advantages of low motion blur, high dynamic range, high time resolution, and the like, and is more suitable for target segmentation tasks in high-speed scenes. At the same time, for commonly used ANN methods, processing continuous high-speed pulse event streams will bring great energy consumption, and the pulse neurons in the spiking neural network simulate the biological signal emission process of the human brain, release pulse signals through the way of membrane potential accumulation and threshold emission. With the research and development of the spiking neural network, the SNN has gradually achieved results comparable to the ANN. Unlike the ANN, which uses continuous floating-point numbers to represent features, the SNN has a lower energy consumption and a faster running speed due to the discrete pulse feature representation, and is suitable for processing high-speed and continuous data streams.

[0004] In recent years, spatiotemporal event pulse streams and spiking neural networks have been applied to a variety of computer vision tasks, such as object classification, target detection, semantic segmentation, depth estimation, and the like, and have achieved certain results; the low-power feature of the SNN has been gradually applied to basic equipment, and can solve a series of problems such as storage and energy. The data modality of spatiotemporal event pulse streams and the SNN spiking neural network still have great potential and are expected to solve scenes that cannot be adapted by traditional picture modalities and ANNs. SUMMARY

[0005] The existing ANN target segmentation method cannot well adapt to high-speed and continuous pulse event streams, and the main purpose of the present application is to provide a target segmentation method and system based on a pulse neural network, which is based on the data characteristics of the space-time event pulse stream, converts the continuous event pulse stream into a data format suitable for the input of the SNN pulse neural network, extracts information through the pulse neuron, uses the membrane potential to remember the past characteristics, improves the feature extraction capability of the target segmentation model for continuous event pulse input, models the space-time relationship through the space-time aggregation module, improves the matching capability of the target segmentation model for the target, aggregates the features through the multi-size feature fusion module, improves the feature fusion capability of the target segmentation model, and further improves the accuracy and robustness of the predicted target area. The present application can not only process continuous space-time event pulse streams with low power consumption, but also obtain accurate segmentation results in low-light and high-speed scenes.

[0006] The object of the present application is achieved by the following technical solutions.

[0007] The target segmentation method based on the pulse neural network disclosed by the present application comprises the following steps:

[0008] Step 101: Collecting a space-time event pulse stream E through an event pulse camera, in order to adapt to the input form of the pulse neural network, using the characteristic that the pulse interval can represent the brightness difference of the object, converting the continuous space-time event pulse stream E into an input representation I suitable for the input of the pulse neural network according to the event pulse interval conversion relationship.

[0009] The continuous space-time event pulse stream E includes event data generated by an event camera and pulse data generated by a pulse camera.

[0010] The space-time event pulse stream E is an N x H x W matrix, and E belongs to {0, 1} N×H×W , wherein N is the number of times of recording pulse data with T as the time interval, and H x W is the spatial resolution of the camera. In order to adapt to the input form of the neural network, according to the characteristic that the pulse data records the cumulative light intensity, the continuous space-time event pulse stream E is converted into an input representation I belonging to R H×W using the pulse interval conversion relationship shown in formula (1).

[0011]

[0012] , wherein C is the maximum gray value set, x and y are the spatial positions of the pulse data, and Δt (x,y) is the interval time of the pulse firing at the current position.

[0013] Step 102: Using a pulse neuron as an activation function to simulate pulse firing, reducing the energy consumption of the target segmentation model and improving the running efficiency.

[0014] As preferred, the pulse neuron is IF, LIF or QLIF.

[0015] The light intensity of the scene affects the density and sparsity of the event pulse data. The light intensity of the scene is measured using a light intensity scale (LIS), and the specific formula of LIS is:

[0016]

[0017] where M is the number of pulse firings in each frame of event pulse data, and HxW is the spatial resolution of the camera. For scenes with low light intensity, the camera cannot obtain sufficient event pulse data, resulting in sparse spatial distribution of event pulse data and insufficient information in the input representation I. Based on the characteristics of event pulse data and the process of simulating biological signal firing in the human brain by pulse neural networks, a pulse neuron is used as an activation function, and the membrane potential of the pulse neuron is used as a hidden state to remember pulse features. The pulse neuron has three steps of charging, discharging and state resetting, and the specific formulas are as follows:

[0018] H[t]=f(V[t-1],X[t]) (3)

[0019] S[t]=Θ(H[t]-V threshold ) (4)

[0020] V[t]=H[t]-V threshold ·S[t] (5)

[0021] where V[t-1] is the membrane potential of the neuron at time t-1, X[t] is the input of the neuron at time t, H[t] is the membrane potential after charging of the neuron, S[t] is the current output of the neuron, V threshold is the pulse firing threshold set for the neuron, Θ is a step function, which outputs 1 when the input is greater than 0, and outputs 0 in other cases, and V[t] is the membrane potential of the neuron after discharging.

[0022] Step 103: using an N-layer pulse neural network to perform multi-size feature extraction on the input representation I t of time t, and obtaining multi-step features F i,t while updating the membrane potential mem i,t of the pulse neuron, thereby improving the feature extraction capability of the target segmentation model for continuous event pulse input.

[0023] Spiking neural networks extract features by combining the membrane potential of the spiking neurons themselves. The accumulation of membrane potential states is beneficial for processing continuous high-speed event pulse data. When the input data is sparse and the spiking neurons cannot extract effective features, the input data is iterated Step times to update the membrane potential, ensuring that the output has sufficient feature information. The specific formula for feature extraction at each step is as follows:

[0024]

[0025] Layer i For each layer of the spiking neural network, s is the iteration step number of the current neuron. When s = 1 and t = 1, It is initialized to a matrix of all zeros; This represents the neuron membrane potential at time t-1 when iterating to step s-1. Let H′ and W′ be the output features of each iteration in the i-th layer, and H′ and W′ be the resolutions of the multi-size features. The features from each step are aggregated and concatenated to obtain the multi-step feature F. i,t ∈{0,1} Step×B×C×H′×W′ .

[0026] Step 104: The multi-step features F from the last layer in step 103... N,t E, as the input of the spatiotemporal feature aggregation module t The target features are aggregated simultaneously in the time and space dimensions to obtain the aggregated output A. t This improves the target segmentation model's ability to match targets and its utilization rate of features.

[0027] The spatiotemporal feature aggregation module is based on a spiking neural network. Preferably, a Recurrent Spiking Transformer (RST) is used. Since the features extracted by the spiking neural network contain rich information with each iteration of the time step, RST strengthens the features from earlier time steps by using features from later time steps to compensate for insufficient information in earlier time steps. The target segmentation model first generates features for each time step, and then aggregates these multi-step features using the spatiotemporal feature aggregation module RST. RST consists of multiple linear layers, spiking neurons, and an MLP, as shown in the following formula:

[0028]

[0029]

[0030]

[0031] in W Q W K W V ∈R d×C W Q W K W V W W

[0032]

[0033] W W W W W ′ t ∈{0,1} B×H′W′×d′ W W W

[0034]

[0035]

[0036] W W t ∈{0,1} Step×B×d×H′×W′ .

[0037] F N-1,t F N-2,t A t O t

[0038] ​The upsampling module (Decoder) includes the UNet Decoder-like ANN upsampling module and the SNN upsampling module. Multi-size features possess rich semantic information. Using the pyramid upsampling module to fuse the semantic information of features of different sizes improves the target segmentation model's ability to utilize features, thereby enhancing the overall accuracy of the prediction results. The formula is as follows:

[0039] O t =Decoder(A t ,F N-1,t ,F N-2,t (13)

[0040] in The target features after fusion.

[0041] Step 106: Use the output O from step 105 t As input to the segmentation head, the pulse size is recovered through interpolation, and the multi-step prediction result Pred at time t is obtained using the Sigmoid function. t .

[0042] Features of multi-size fusion modules O t ∈{0,1} Step×B×d×H′×W′ For pulse features, the segmentation head uses convolution mapping to convert pulse features into floating-point features, and then obtains the target saliency matrix through the Sigmoid function, as shown in the following formula:

[0043] Pred t =Sigmoid(upsample(Conv) head (O t (14)

[0044] Among them Pred t ∈R Step×B×1×H×W For each step, `unsample` represents the target saliency matrix predicted; `Conv` represents the interpolation upsampling operation. head This is the mapping convolution kernel for the segmentation head.

[0045] Step 107: Analyze the multi-step prediction results from Step 106 using Pred. t Multi-Step Loss is used to constrain the early time step features generated by the target segmentation model, thereby reducing the target segmentation model's dependence on the time step and improving its robustness.

[0046] The feature aggregation module enhances features from later time steps to features from earlier time steps. Based on this, it further constrains the efficiency of feature utilization by the target segmentation model by increasing the weight of earlier time step results during training. This aims to achieve accurate results at lower time steps and improve the robustness of the target segmentation model. The loss function is as follows:

[0047]

[0048] Among them GT t Let α be the label of the target object at time t. s The weights of the loss at step s.

[0049] Step 108: Repeat steps 101-107 to process the continuous event pulse stream. Use spiking neurons as activation functions to improve the computational speed of the target segmentation model and reduce its energy consumption. Use the membrane potential V of spiking neurons as hidden states to transmit spatiotemporal information and improve the feature extraction capability of the target segmentation model for continuous event pulse inputs. Fuse feature information between multiple time steps through the spatiotemporal feature aggregation module to improve the utilization rate of features by the target segmentation model, thereby improving the accuracy and robustness of the target segmentation results.

[0050] This invention discloses a target segmentation system based on a spiking neural network, implemented according to a target segmentation method based on a spiking neural network. The target segmentation system based on a spiking neural network includes a spatiotemporal event stream acquisition module, a spiking neural network feature extraction module, a spatiotemporal feature aggregation module, and a target region generation module.

[0051] The spatiotemporal event stream acquisition module acquires a spatiotemporal event stream E through an event pulse camera, and converts the continuous spatiotemporal event pulse stream E into an input representation I adapted to a spiking neural network according to the event pulse interval conversion relationship. The module then converts the current time input representation I... t Input to the feature extraction module.

[0052] The spiking neural network feature extraction module is used to extract continuous event pulse features in the time and space dimensions. By memorizing long-term information through spiking neurons, it outputs multi-scale features F containing target information at the current moment while updating the internal membrane potential. i,t , feature F N,t Input to the spatiotemporal feature aggregation module.

[0053] The spatiotemporal feature aggregation module is used to aggregate target information from multi-step pulse features, and to enhance the early time step features with later time step features to obtain the aggregated target feature A. t , will A t Input into the target area generation module.

[0054] The target region generation module is configured to generate a predicted target region at a current time t according to an input A t The pulse target feature restores the feature size using an interpolation operation, determines the target category to which each pixel belongs through a Sigmoid function, and obtains a region Pred predicted at the current time t Then the membrane potential of all pulse neurons is updated, and the next cycle iteration is performed until the spatiotemporal event stream acquisition module stops outputting data representation I. During the target region generation process, the time step of the pulse neuron is gradually reduced according to the accuracy of the prediction result, and the energy consumption of the system is reduced.

[0055] Advantages:

[0056] 1. The target segmentation method and system based on the pulse neural network disclosed by the application can convert continuous spatiotemporal event pulse streams into input representations suitable for pulse neural networks through an event pulse interval conversion relationship, so as to process continuous high-speed spatiotemporal event pulse streams.

[0057] 2. The target segmentation method and system based on the pulse neural network disclosed by the application uses pulse neurons in the pulse neural network as an activation function, extracts information and remembers past features, and improves the feature extraction capability of the target segmentation model for continuous spatiotemporal event pulse inputs.

[0058] 3. The target segmentation method and system based on the pulse neural network disclosed by the application fuses information between multiple time steps through a spatiotemporal feature aggregation module, improves the utilization rate of features of the target segmentation model and the robustness of the target segmentation model, and further improves the accuracy of the prediction result.

[0059] 4. The target segmentation method and system based on the pulse neural network disclosed by the application reduces the dependence of the target segmentation model on time steps through a multi-step loss Multi-Step Loss, and a single time step can produce a good prediction result. Not only can the generalization capability of the target segmentation model be improved, but also the energy consumption of the target segmentation model can be significantly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of the target segmentation method and system based on the pulse neural network disclosed by the application.

[0061] Figure 2 is a framework diagram of the target segmentation method and system based on the pulse neural network disclosed by the application.

[0062] Figure 3 is a target segmentation result diagram of the target segmentation method and system based on the pulse neural network disclosed by the application for spatiotemporal event pulse stream input.

[0063] Figure 4 is a target segmentation result graph of the target segmentation method and system based on the pulse neural network in the high light and low light scene.

[0064] Figure 5 is a segmentation result graph of the target segmentation method and system based on the pulse neural network in the single time step. DETAILED DESCRIPTION

[0065] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While example embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0066] Example 1:

[0067] Figure 1 The target segmentation method flowchart provided by the embodiment of the present application is shown in Figure 1 The target segmentation method based on the pulse neural network disclosed in the embodiment is implemented as follows:

[0068] Step 101: Collecting a spatiotemporal event pulse stream E by an event pulse camera. In order to adapt to the input form of the pulse neural network, the pulse interval is used to represent the brightness difference of the object, and the continuous spatiotemporal event pulse stream E is converted into an input representation I suitable for the pulse neural network according to the pulse interval conversion relationship.

[0069] In the embodiment, the continuous spatiotemporal event pulse stream E uses pulse data generated by a pulse camera. The spatiotemporal event pulse stream E is an N x H x W matrix, and E ∈ {0, 1} N×H×W , where N is the number of times of recording pulse data with a time interval of T, H x W is the spatial resolution of the camera, and H = 250 and W = 400. In order to adapt to the input form of the neural network, according to the characteristics of the pulse data recording cumulative light intensity, the continuous spatiotemporal event pulse stream E is converted into an input representation I ∈ R H×W using the pulse interval conversion relationship shown in formula (1).

[0070]

[0071] , where C is the maximum gray value set, C is set to 255 in the embodiment, x and y are the spatial positions of the pulse data, and Δt (x,y) is the interval time of the pulse firing at the current position.

[0072] Step 102: Using the pulse neuron as the activation function, simulating the pulse firing, reducing the energy consumption of the target segmentation model and improving the running efficiency.

[0073] The light intensity of the scene affects the density and sparsity of the event pulse data. The light intensity of the scene is measured using a light intensity scale (LIS). The specific formula of LIS is:

[0074]

[0075] Where M is the number of pulse firings in each frame of event pulse data, and HxW is the spatial resolution of the camera. In this embodiment, the threshold value of LIS is set to 0.031 according to the average light intensity of the scene, and a LIF pulse neuron is used. The neural dynamics equation of LIF is as follows:

[0076]

[0077] S[t] = Θ(H[t] - V threshold ) (4)

[0078] V[t] = H[t] · (1 - S[t]) + V reset · S[t] (5)

[0079] Where V[t-1] is the membrane potential of the neuron at time t-1, X[t] is the input of the neuron at time t, H[t] is the membrane potential after charging the neuron, S[t] is the current output of the neuron, V threshold is the pulse firing threshold set for the neuron, Θ is the step function, which outputs 1 when the input is greater than 0, and outputs 0 in other cases, and V reset is the reset membrane potential of the LIF pulse neuron, and V[t] is the membrane potential of the neuron after discharge.

[0080] Step 103: Using an N-layer pulse neural network to perform multi-size feature extraction on the input I t at time t, according to the set pulse neuron time step Step, to obtain multi-step features F i,t , and update the membrane potential mem i,t of the pulse neuron, thereby improving the feature extraction capability of the target segmentation model for continuous event pulse input.

[0081] Spiking neural networks (SNNs) extract features by utilizing the membrane potential of spiking neurons. The accumulation of membrane potential states is beneficial for processing continuous, high-speed event pulse data. When the input data is sparse, preventing spiking neurons from extracting effective features, the input data is iterated Step times to update the membrane potential, ensuring the output has sufficient feature information. In this embodiment, the number of layers N in the spiking neural network is set to 4, and the time step Step of the spiking neuron is set to 5. The specific formula for feature extraction at each step is as follows:

[0082]

[0083] Layer i For each layer of the spiking neural network, s is the iteration step number of the current neuron. When s = 1 and t = 1, It is initialized to a matrix of all zeros; This represents the neuron membrane potential at time t-1 when iterating to step s-1. Let H′ and W′ be the output features of each iteration in the i-th layer, and H′ and W′ be the resolutions of the multi-size features. The features from each step are aggregated and concatenated to obtain the multi-step feature F. i,t ∈{0,1} Step×B×C×H′×W′ In this embodiment, each layer consists of a Conv convolutional layer, a BatchNorm layer, a LIF spiking neuron, and a Max Pooling layer. Max Pooling is used to perform four downsampling operations to obtain multi-size features.

[0084] Step 104: The multi-step features F from the last layer in step 103... N,t E, as the input of the spatiotemporal feature aggregation module t The target features are aggregated simultaneously in the time and space dimensions to obtain the aggregated output A. t This improves the target segmentation model's ability to match targets and its utilization rate of features.

[0085] In this embodiment, a Recurrent Spiking Transformer (RST) is used as the feature aggregation module. Since the features extracted by the spiking neural network contain richer information with each iteration of the time step, RST strengthens the features from earlier time steps by using features from later time steps to compensate for the lack of information in earlier time steps. The target segmentation model first generates features for each time step, and then aggregates these multi-step features using the spatiotemporal feature aggregation module RST. RST consists of multiple linear layers, spiking neurons, and an MLP, as shown in the following formula:

[0086]

[0087]

[0088]

[0089] in For target features at adjacent time steps, W Q W K W V ∈R d×C Learnable parameters for each Linear layer, SN Q ,SN K ,SN V These are spiking neurons, and BN is a BatchNorm layer. The corresponding pulse features are obtained after passing through spiking neurons. Then, a multi-head self-attention operation is used to calculate the attention relationship matrix, the specific formula of which is:

[0090]

[0091] in for Features obtained from multi-head attention, For the corresponding dimension, It is the scaling factor of the eigenvalues, AQ ′ t ∈{0,1} B×H′W′×d′ The output is for each multi-head attention. In this embodiment, d is set to 256 and head is set to 8. All outputs are merged to obtain the attention value. The input to the MLP is then obtained through residual OR operations. OR operations can not only supplement feature information but also reduce energy consumption. The specific formula is as follows:

[0092]

[0093]

[0094] in As the output features of the MLP, the output of the spatiotemporal feature aggregation module A is obtained by aggregating the output of each step of the MLP. t ∈{0,1} Step×B×d×H′×W′ .

[0095] Step 105: Use the multi-size feature F from step 103 N-1,t ,F N-2,t The output A of step 104 t As input to the pyramid upsampling module, the fused target features O are obtained. t, improve the feature fusion capability of the target segmentation model for coarse-grained and fine-grained information, and further improve the accuracy of the target segmentation result.

[0096] The up-sampling module Decoder includes an UNet Decoder class ANN up-sampling module and an SNN up-sampling module. Multi-size features have rich semantic information. The semantic information of features of different sizes is fused using a pyramid up-sampling module to improve the utilization capability of the target segmentation model for features, and further improve the accuracy of the overall prediction result. In the embodiment, a pulse up-sampling module SNN Decoder is used to fuse multi-size features, and the multi-size features are input for fusion. The SNN Decoder is composed of two layers of pulse neural networks, each layer is composed of a Conv convolutional layer, a BatchNorm layer and a LIF pulse neuron, and the SNN Decoder formula is as follows:

[0097] O t =SNNDecoder(A t ,F N-1,t ,F N-2,t ) (13)

[0098] Where F N-1,t ,F N-2,t are the intermediate two layers of features of the Encoder module, is the fused target feature.

[0099] Step 106: use the output O t of step 105 as the input of the segmentation head, restore the pulse size through the interpolation operation, and use the Sigmoid function to obtain the multi-step prediction result Pred t at time t.

[0100] The feature O t ∈{0,1} Step×B×d×H′×W′ is a pulse feature. The segmentation head converts the pulse feature into a floating point feature in a convolution mapping manner, and then obtains a target saliency matrix through a Sigmoid function. The formula is as follows:

[0101] Pred t =Sigmoid(upsample(Conv head (O t ))) (14)

[0102] Where Pred t ∈R Step×B×1×H×W is the target saliency matrix predicted at each step, unsample is an interpolation up-sampling operation, and Conv head is the mapping convolution kernel of the segmentation head.

[0103] Step 107: the multi-step prediction result Pred of step 106 is used as a constraint target to generate the early time step feature of the target segmentation model, reduce the dependence of the target segmentation model on the time step, and improve the robustness of the target segmentation model. t , using Multi-Step Loss to constrain the early time step feature generated by the target segmentation model, reduce the dependence of the target segmentation model on the time step, and improve the robustness of the target segmentation model.

[0104] The feature aggregation module strengthens the late time step feature to the early time step feature, and then increases the weight of the early time step result during training to constrain the utilization efficiency of the feature by the target segmentation model again, so as to obtain accurate results at a lower time step and improve the robustness of the target segmentation model. The loss function is as follows:

[0105]

[0106] Where GT t is the target object label at time t, and in this embodiment is the loss weight at the s-th step.

[0107] Step 108: repeat steps 101-107 to process the continuous event pulse stream, use the pulse neuron as the activation function to improve the operation rate of the target segmentation model and reduce the energy consumption of the target segmentation model; use the membrane potential V of the pulse neuron as the hidden state to transmit the spatio-temporal information and improve the feature extraction capability of the target segmentation model for the continuous event pulse input; and through the spatio-temporal feature aggregation module, the feature information between multiple time steps is fused to improve the utilization rate of the feature by the target segmentation model, thereby improving the accuracy and robustness of the target segmentation result.

[0108] The embodiment discloses a target segmentation system based on a pulse neural network. Figure 2 The system module of the embodiment is shown in Figure 2 The target segmentation system based on the pulse neural network includes a spatio-temporal event stream acquisition module, a pulse neural network feature extraction module, a spatio-temporal feature aggregation module, and a target region generation module.

[0109] The spatio-temporal event stream acquisition module acquires the spatio-temporal event stream E through an event pulse camera, converts the continuous spatio-temporal event pulse stream E into an input representation I suitable for the pulse neural network according to an event pulse interval conversion relationship, and inputs the current time input representation I t to the feature extraction module.

[0110] The pulse neural network feature extraction module is used to extract continuous event pulse features in the time and space dimensions, memorize long-time information through the pulse neuron, and output multi-scale features F containing target information at the current time while updating the internal membrane potential.i,t , the feature F N,t is input to a spatiotemporal feature aggregation module.

[0111] The spatiotemporal feature aggregation module is configured to aggregate target information in the multi-step pulse feature, reinforce the early time step feature by the later time step feature, and obtain the aggregated target feature A t , which is input to a target region generation module. t

[0112] The target region generation module is configured to generate a predicted target region at the current time t according to the input A t , the pulse target feature, restore the feature size using an interpolation operation, determine the target category to which each pixel belongs by using a Sigmoid function, and obtain the predicted region Pred t at the current time. Then, the membrane potential of all pulse neurons is updated, and the next cycle iteration is performed until the spatiotemporal event stream acquisition module stops outputting the data representation I. During the target region generation process, the time step of the pulse neuron is gradually reduced according to the accuracy of the prediction result, and the energy consumption of the system is reduced.

[0113] This embodiment will illustrate the effectiveness of the present application by the results of target segmentation.

[0114] 1. Experimental conditions

[0115] The hardware test conditions of this experiment are: Intel Xeon Gold 6240C, memory 32G. The GPU is NVIDIA GeForce RTX 3090, with 24G of video memory, and the CUDA version is 11.1; the camera used is Spike Vidar pulse camera.

[0116] In order to ensure the fairness of the comparative experiment, the spatiotemporal event pulse stream collected by the pulse camera is used as the SVS data set to train the current target segmentation method based on the pulse neural network SNN. All methods are adjusted to adapt to the pulse input representation, and the original training hyperparameters are used to retrain the same number of rounds on the SVS pulse saliency target segmentation data set.

[0117] 2. Experimental results

[0118] In order to quantitatively measure the quality of the target segmentation result, the mean absolute error (Mean Absolute Error, MAE), the structure similarity (Structure Similarity Index Measure, SSIM), and the average value mF β and the maximum value ​To measure the accuracy of the predicted target region, while according to the time step used by the model during inference, the comparison cases are divided into single time step and multi-time step.

[0119] Table 1 shows the target segmentation index results of the input pulse feature representation data method after training on the SVS pulse saliency target detection data set with different time step settings. From the table, it can be seen that the method of the present application can achieve better segmentation results than other comparison methods, whether in single time step or multi-time step, which shows that the spatiotemporal feature aggregation module and Multi-Step Loss can significantly improve the performance of the model.

[0120]

[0121] Table 1 shows the target segmentation index results of the input pulse feature representation data method after training on the SVS pulse saliency target detection data set with different time step settings. From the table, it can be seen that the method of the present application can achieve better segmentation results than other comparison methods, whether in single time step or multi-time step, which shows that the spatiotemporal feature aggregation module and Multi-Step Loss can significantly improve the performance of the model. Figure 3 The results of the present method for pulse stream input are shown in the figure, whether in single step or multi-step inference setting, the method of the present application can accurately segment the target region, and is more stable than the comparison method, and the target region is more accurate.

[0122] Figure 4 The segmentation results of the present method in high light and low light scenes are shown in the figure, from which it can be seen that the target segmentation method of the present application can adapt to different lighting environments, and can not only predict accurate results in high light scenes, but also effectively detect target regions in sparse pulse streams in low light environments; Figure 5 The segmentation results of the present method in single time step case for continuous pulse input are shown in the figure, with the accumulation of neuron membrane potential, the present method can achieve relatively accurate segmentation results using single time step, which can save more energy in actual deployment.

[0123] In summary, the target segmentation method and system based on pulse neural network proposed in the present embodiment can convert data representation, store past information using pulse neurons, and use spatiotemporal feature aggregation module to strengthen target features, which significantly improves the utilization of continuous spatiotemporal pulse stream, and successfully applies pulse neural network to the field of target segmentation in high speed and continuous data modal. The target segmentation method and system based on pulse neural network proposed in the present embodiment has lower energy consumption than traditional ANN method, which means it is easier to deploy to actual application scenarios. The present embodiment not only broadens the application scenarios of target segmentation tasks, but also has important application value in low light, high speed, human-computer interaction, target tracking, automatic driving and other fields.

[0124] The above detailed description of the specific description, the purpose, technical scheme and beneficial effects of the application are further described in detail, it should be understood that the above description is only a specific embodiment of the application, and is not used to limit the protection scope of the application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A method for object segmentation based on a spiking neural network, characterized in that: The method comprises the following steps of: Step 101: acquiring a spatio-temporal event pulse stream E by an event pulse camera, converting the continuous spatio-temporal event pulse stream E into an input representation I suitable for a pulse neural network according to an event pulse interval conversion relationship, so as to adapt to an input form of the pulse neural network and utilize the characteristics of the pulse interval capable of representing brightness differences of objects; Step 102: using a pulse neuron as an activation function to simulate pulse firing, so as to reduce energy consumption and improve operation efficiency of a target segmentation model; Step 103: using an N-layer spiking neural network to input representation I at time t t Multi-size feature extraction is performed, and multi-step features F are obtained according to a set spiking neuron time step Step i,t Meanwhile, the membrane potential mem of the spiking neuron is updated i,t The feature extraction capability of the target segmentation model for continuous event spiking input is improved; Step 104: obtaining the multi-step feature F of the last layer in step 103 N,t As the input E of the spatio-temporal feature aggregation module t In the time and space dimensions, the target feature is aggregated to obtain the aggregated output A t Improve the matching ability of the target segmentation model to the target and the utilization rate of the feature Step 105: using the multi-size features F of step 103 N-1,t ,F N-2,t With the output A of step 104 t As the input of the pyramid upsampling module, the fused target feature O is obtained t , improve the feature fusion capability of the target segmentation model to coarse-grained and fine-grained information, and improve the accuracy of the target segmentation result; Step 106: using the output O of step 105 t As the input of the split head, the pulse size is recovered by the interpolation operation, and the multi-step prediction result Pred at time t is obtained using the Sigmoid function t ; Step 107: using the multi-step prediction result Pred of step 106 t The multi-step loss constraint target segmentation model uses the early time step features generated by the target segmentation model to reduce the dependence of the target segmentation model on the time step and improve the robustness of the target segmentation model. Step 108: repeating steps 101-107 to process continuous event pulse streams, using a pulse neuron as an activation function to improve operation speed and reduce energy consumption of the target segmentation model, using a membrane potential V of the pulse neuron as a hidden state to transmit spatio-temporal information and improve feature extraction capability of the target segmentation model on continuous event pulse inputs, and using a spatio-temporal feature aggregation module to fuse feature information between multiple time steps, so as to improve utilization of features and improve accuracy and robustness of a target segmentation result. 2.The target segmentation method based on the pulse neural network according to claim 1, wherein: The continuous spatio-temporal event pulse stream E in step 101 comprises event data generated by an event camera and pulse data generated by a pulse camera; The spatiotemporal event pulse stream E is an N x H x W matrix, and E e {0, 1} N×H×W , where N is the number of times of recording pulse data at a time interval T, H x W is the spatial resolution of the camera; in order to adapt to the input form of the neural network, according to the characteristics of the pulse data record cumulative light intensity, the continuous spatiotemporal event pulse stream E is converted into an input representation I e R H×W using the pulse interval conversion relationship shown in formula (1). where C is a set maximum gray value, x, y are spatial positions of the pulse data, Δt (x,y) is the interval time of the two pulse firings at the current position.

3. The method of claim 2, wherein: In step 102, the pulse neuron used to simulate pulse firing is an IF, LIF or QLIF; the intensity of light of a scene affects the density and sparsity of event pulse data, and a light intensity ratio LIS is used to measure the intensity of light of the scene, and a specific formula of the LIS is as follows: Wherein, M is the number of pulse firing in each frame of event pulse data, and HxW is the spatial resolution of the camera; For a scene with low light intensity, the camera cannot obtain sufficient event pulse data, resulting in that the event pulse data is sparsely distributed in space and the information in the input representation I is insufficient; Based on the characteristics of the event pulse data and the process of simulating biological signal firing of the human brain by the pulse neural network, a pulse neuron is used as an activation function, and a membrane potential of the pulse neuron is used as a hidden state to remember pulse features; the pulse neuron comprises three steps of charging, discharging and state resetting, and specific formulas are as follows: H[t]=f(V[t-1],X[t]) (3) S[t] = Θ(H[t] - V threshold ) (4) V[t] = H[t] - V threshold • S[t] (5) Where V[t-1] is the membrane potential of neuron at t-1, X[t] is the input of neuron at t, H[t] is the membrane potential of neuron after charging, S[t] is the output of neuron at t, V[t] is the membrane potential of neuron after discharging. threshold Where V[t-1] is the membrane potential of neuron at t-1, X[t] is the input of neuron at t, H[t] is the membrane potential of neuron after charging, S[t] is the output of neuron at t, V[t] is the membrane potential of neuron after discharging.

4. The method of claim 3, wherein: In step 103, the N-layer pulse neural network is used to extract features, and the state accumulation of the membrane potential is conducive to processing continuous high-speed event pulse data; when the input data is sparse and the pulse neuron cannot extract effective features, the input data is iterated Step times to update the membrane potential, so that the output has sufficient feature information, and a specific formula for each step of feature extraction is as follows: where Layer i is the pulse neural network of each layer, s is the iteration step number of the current neuron, when s = 1, t = 1, is initialized as a full 0 matrix; is the membrane potential of the neuron at the s-1 step of iteration at t-1 moment, is the output feature of each step iteration of the i-th layer, H' and W' are the resolutions of multi-size features, and the multi-step features F are obtained by aggregating and concatenating each step feature i,t ∈{0, 1} Step×B×C×H′×W′ .

5. The method of claim 4, wherein: The spatio-temporal feature aggregation module of step 104 is a feature aggregation module based on SNN pulse neural network, and uses a recurrent full pulse Transformer as the feature aggregation module; since the features extracted by the pulse neural network contain rich information with the iteration of time steps, the RST reinforces the early time step features by the late time step features to make up for the lack of early feature information; the target segmentation model first generates features at each time step, and then aggregates the multi-step features through the spatio-temporal feature aggregation module RST, which is composed of multiple Linear linear layers, pulse neurons and MLP, and the specific formula is as follows: wherein is the target feature of the adjacent time step, W Q ,W K ,W V ∈R d×C is the learning parameter of each Linear layer, SN Q ,SN K ,SN V is the spiking neuron, BN is the BatchNorm layer, is the corresponding spiking feature after the spiking neuron; The attention relationship matrix is calculated using a multi-head self-attention operation, and the specific formula is as follows: wherein is with head multi-head attentions, is a corresponding dimension, is a scaling factor of the feature value, AQ ′ t ∈ {0, 1} B×H′W′×d′ is an output of each multi-head attention; combining all outputs to obtain attention values obtaining the input of the MLP through the residual or operation again The or operation can not only supplement feature information but also reduce energy consumption, and the specific formula is as follows: wherein is the output feature of the MLP, the output of the spatio-temporal feature aggregation module A is obtained by aggregating each step output of the MLP t ∈{0,1} Step×B×d×H′×W′ .

6. The method of claim 5, wherein: The step 103 is used in step 105 N-1,t ,F N-2,t The output A of step 104 t As the input of the pyramid upsampling module, the feature fusion ability of the model to coarse-grained and fine-grained information is improved, and the accuracy of target segmentation is further improved. The up-sampling module Decoder includes: UNet Decoder class ANN up-sampling module, SNN up-sampling module; Multi-size features have rich semantic information, and the pyramid up-sampling module is used to fuse the semantic information of different size features to improve the utilization ability of the target segmentation model to the features, and thus improve the accuracy of the overall prediction result, and the formula is as follows: wherein is the fused target feature.

7. The method of claim 6, wherein: The output O of step 105 is used in step 106 t As the input of the split head, the pulse size is recovered by the interpolation operation, and the multi-step prediction result Pred at time t is obtained using the Sigmoid function t ; Features O of a multi-size fusion module t ∈ {0, 1} Step×B×d×H′×W′ For the pulse feature, the segmentation head converts the pulse feature into a floating-point feature in the manner of convolution mapping, and then obtains a target saliency matrix through a Sigmoid function, and the formula is as follows: Pred t = Sigmoid (upsample (Conv head (O t ) ) ) (14) where Pred t ∈ R Step×B×1×H×W is the predicted saliency map for each step, unsample is an interpolation up-sampling operation, Conv head is the mapping convolution kernel of the segmentation head.

8. The method of claim 7, wherein: The multi-step prediction result Pred of step 106 in step 107 t The Multi-Step Loss constraint target segmentation model produces early time step features, reduces the target segmentation model's dependence on time steps, and improves the robustness of the target segmentation model. The feature aggregation module strengthens the high time step features to the low time step features, and on this basis, the utilization efficiency of the target segmentation model to the features is further constrained by increasing the weight of the early time step results during training, so as to obtain accurate results at a lower time step and improve the robustness of the target segmentation model, and the loss function is as follows: where GT t is the target object label at time t, a s is the weight of the loss at step s.

9. The method of claim 8, wherein: In step 108, the spatio-temporal information is memorized by the pulse neuron membrane potential, and the target segmentation area of the continuous spatio-temporal event pulse stream is obtained by using a recurrent iteration method.

10. A spiking neural network based object segmentation system, implemented based on the spiking neural network based object segmentation method according to any one of claims 1 to 9. It includes a spatio-temporal event stream acquisition module, a pulse neural network feature extraction module, a spatio-temporal feature aggregation module and a target region generation module: The spatiotemporal event stream acquisition module acquires the spatiotemporal event stream E through an event pulse camera, converts the continuous spatiotemporal event pulse stream E into an input representation I suitable for a pulse neural network according to an event pulse interval conversion relationship, and inputs the input representation I of the current moment to the feature extraction module t . The pulse neural network feature extraction module is used for extracting continuous event pulse features in time and space dimensions, memorizing long-time information through pulse neurons, outputting multi-scale features F containing target information at the current time while updating the internal membrane potential i,t The features F N,t are input to the space-time feature aggregation module; The spatio-temporal feature aggregation module is configured to aggregate target information in the multi-step pulse features, to strengthen the early time step features by the late time step features, and to obtain aggregated target features A t , A t is input to the target region generation module. The target region generation module is configured to generate a predicted target region at a current time t according to an input A t The pulse target feature restores the feature size using an interpolation operation, determines the target category to which each pixel belongs through a Sigmoid function, and obtains a predicted region Pred t ; then update the membrane potential of all pulse neurons, and perform the next cycle iteration until the spatiotemporal event stream acquisition module stops outputting data representation I; during the target region generation process, gradually reduce the time step of the pulse neurons according to the accuracy of the prediction result, and reduce the energy consumption of the system.

Images