Training method based on biology-information collaborative spiking neural network and neuromorphic optical flow prediction method based on training method

Abstract

The invention discloses a training method based on a biology-information collaborative spiking neural network and a neuromorphic optical flow prediction method based on the training method, which are applied to the technical field of computer vision and aim at the problem that efficient and accurate optical flow prediction is difficult to realize in the prior art. According to the method, model parameters trained by the ANN are directly migrated to the SNN model in a cross-model manner, fine adjustment is performed by using a'space-time direction propagation 'method of the SNN model, and network weights based on information constraints and biological parameters based on biological constraints are cooperatively trained. According to the method, the technical difficulty of converting the ANN model into the SNN model is greatly reduced, and the performance and the stability of the converted model are improved. In the aspect of a neuromorphic data optical flow estimation task, the biological-information collaborative spiking neural network effect is superior to the previous result.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a neuromorphic optical flow estimation technology. Background Art

[0002] Optical flow estimation is a fundamental topic in the field of computer vision and has broad application prospects. Event-based neuromorphic cameras can asynchronously record changes in light intensity and can effectively address the challenges posed by scenes with high dynamic range and rapid changes, while avoiding the errors introduced by the "pixel intensity invariance assumption" in traditional optical flow estimation. Therefore, compared with traditional methods, optical flow estimation based on neuromorphic vision has significant advantages. Current methods usually reconstruct neuromorphic data into frame-based images so that optical flow estimation can be performed using traditional artificial neural network (ANN) models. These methods often ignore the errors caused by the reconstruction process. For example, images aggregated over long time windows often show obvious motion blur. At the same time, dynamic features embedded in the time domain are also difficult to fully utilize.

[0003] Unlike artificial neural networks, spiking neural networks (SNNs) are inspired by real brain mechanisms, using biologically reliable spiking neurons as the basic structure for information processing, and encoding and transmitting information through discrete spike events in the temporal direction. This precise time-based information processing and encoding method makes SNNs an effective tool for processing complex spatiotemporal information. Therefore, using SNN models for neuromorphic visual optical flow estimation shows a natural advantage. However, using backpropagation methods to train SNNs faces the problem of poor training results, which limits their performance in practical tasks. Currently, the "ANN to SNN" method is mainly used to improve the performance of SNN models. However, the conversion method itself has theoretical flaws, resulting in the need to manually set biological parameters for the converted SNN model to ensure accuracy. This greatly reduces the operability of the "ANN to SNN" method, making efficient and accurate optical flow prediction difficult. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a training method based on a bio-informatics collaborative pulse neural network and a neuromorphic optical flow prediction method based on the same. Through bio-informatics collaborative constraints, efficient and rapid training of SNN is achieved, and the performance of SNN in neuromorphic visual optical flow prediction tasks is improved.

[0005] One of the solutions adopted by the present invention is: a training method based on a biological-information collaborative pulse neural network, comprising:

[0006] S1. Build ANN and SNN models with the same structure. The ANN model uses sigmoid neurons, and the SNN model uses leaky-integrate-release neurons.

[0007] S2. Neuromorphic image asynchronously records the brightness change p on each photosensitive element k , the image is represented as:

[0008]

[0009] Where K represents p k The number of (x k ,y k , t k ) represents the k-th brightness change recorded at the x, y pixel at time t;

[0010] All brightness changes are divided into N groups, and each group of brightness changes is divided into two frames: enhancement and reduction, which are expressed as:

[0011]

[0012] in,

[0013] The ANN model input is: The input of the SNN model is

[0014] S3. Using the ANN model input obtained in step S2, the ANN model is trained, and the weight parameters of the ANN model after training are saved;

[0015] S4. Specific parameters of the SNN model: membrane time constant τ and spike firing threshold V th Perform random initialization and migrate the weight parameters of the ANN model saved in step S3 to the SNN model;

[0016] S5. Use the SNN model input obtained in step S2 to train the SNN model obtained in step S4; obtain a trained SNN model.

[0017] The second technical solution adopted by the present invention is: a neuromorphic optical flow prediction method based on a biological-information collaborative pulse neural network, which processes the neural state data to be estimated into the SNN model input form, inputs it into the trained SNN model, and obtains the optical flow image and label optical flow.

[0018] Beneficial effects of the present invention: Traditional ANN model training is efficient and accurate, and the present invention obtains information-constrained network weight knowledge through training of neuromorphic data and ANN model. Similar to the previous "ANN to SNN" method, the present invention still constructs an SNN model with the same structure as the ANN model. However, the difference is that the present invention does not use complex mathematical derivation to infer the biological parameters of the model, but proposes a method based on bio-information collaboration to fine-tune the model. Specifically, the present invention directly migrates the model parameters trained by the ANN across models to the SNN model, and uses the "time-space directional propagation" method of the SNN model for fine-tuning, and collaboratively trains the network weights based on information constraints and the biological parameters based on biological constraints. The present invention greatly reduces the technical difficulty of converting the ANN model to the SNN model, and improves the performance and stability of the converted model. In the task of optical flow estimation of neuromorphic data, the optical flow estimation accuracy and prediction efficiency of the bio-information collaborative pulse neural network are better than previous results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the flow chart of the neuromorphic optical flow prediction method based on biological-informatics collaborative spiking neural network according to an embodiment of the present invention.

[0020] Figure 2 Schematic diagram of the pulse neural network structure for optical flow prediction according to an embodiment of the present invention.

[0021] Figure 3 Graph showing the results of the neuromorphic optical flow prediction method based on biological-informatics collaborative spiking neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0023] The training process of the bio-information collaborative spiking neural network involved in the present invention mainly includes five parts: model construction, neuromorphic data processing, artificial neural network (ANN) model training, ANN information parameter migration, and spiking neural network (SNN) biological parameter fine-tuning.

[0024] like Figure 1 As shown, the complete steps of the specific embodiment of the present invention are as follows:

[0025] S1: Network model construction: In order to achieve subsequent information parameter migration, the present invention needs to build ANN and SNN models with the same structure. Specifically, the ANN model is composed of sigmoid neurons, and its neuron model is defined as follows:

[0026]

[0027] Among them, x represents the neuron input information, Y ann Represents the output of a sigmoid neuron. Sigmoid neurons accept continuous analog values ​​for information processing and output continuous analog values ​​to be passed to subsequent neurons. The SNN model is composed of leaky-integrate-and-fire (LIF) neurons and uses discrete pulse trains to encode and transmit information. The LIF neuron model is defined as follows:

[0028]

[0029] Among them, I t is the input current of the neuron at time t, u t and o t Indicates the membrane potential and pulse emission of neurons at time t, α t represents the membrane potential decay coefficient, τ represents the membrane time constant, V th In the present invention, the neuron threshold v th and the membrane time constant τ are both trainable parameters.

[0030] Sigmoid and LIF neurons are used to build ANN and SNN network models respectively. The model structure consists of five parts: convolutional gated recurrent unit, encoding layer, residual module, decoding layer, and generator. Specifically, the data first enters convolutional gated recurrent unit 1, and then enters encoding layers 1 to 4 and residual modules 1 to 2 in sequence to extract features of different scales. The feature maps of different scales generated by the four encoding layers and the feature map generated by residual module 2 are spliced ​​and upsampled, and then sent to the decoding layer, generator, and convolutional gated recurrent unit 2 in sequence for optical flow prediction, ultimately generating the predicted optical flow.

[0031] like Figure 2 As shown, the output of the gated recurrent unit The expression is:

[0032]

[0033] Among them, ⊙ represents the matrix multiplication operation, O t-1 Represents the output of the previous moment, R t , Z t and H t Represent the reset gate, update gate and hidden state respectively. The specific calculation process is:

[0034] R t =σ1(εR (O t-1 , I t ))

[0035] Z t =σ1(ε Z (O t-1 , I t ))

[0036] H t =σ2(ε I (R t ⊙O t-1 , I t ))

[0037] Among them, σ1 and σ2 represent sigmoid and tanh activation functions respectively, ε R , ε Z and ε I Represent the convolution operation, I t Indicates the current moment input.

[0038] The encoding and decoding layers are commonly used convolution operations, and the residual module is a standard residual module in residual neural networks. The generator is a convolution operation with an output layer of 2 to generate optical flow information.

[0039] In this structure, the generator output is amplified four times and passed to the subsequent convolutional gated recurrent unit 2, thereby aligning the historical optical flow information with the current optical flow information. This aligns the output of the structure with the input of the structure, facilitating subsequent feedback enhancement operations. Finally, the optical flow generated by the convolutional gated recurrent unit 2 is fed back into the convolutional gated recurrent unit 1 as input to enhance the subsequent input data.

[0040] S2: Neuromorphic data processing: Neuromorphic images asynchronously record the brightness changes p on each photosensitive element. k , the brightness increases beyond the threshold value p k =p + , the brightness decreases beyond the threshold value p k =p - , the image is represented as:

[0041]

[0042] Where K represents p k The number of (x k ,y k , t k ) indicates that the kth brightness change is recorded at the x, y pixel at time t. To facilitate network processing, we divide all brightness changes into N groups, and divide each group of brightness changes into two frames: enhancement and reduction, which are expressed as:

[0043]

[0044] in, In ANN, the model input is: In SNN, the model input is

[0045] S3: Training ANN model: Use the ANN model established in S1 and the neuromorphic data processed in S2 to train the ANN model. During the training process, the network loss value is calculated using the endpoint error loss function (EPE loss).

[0046] Loss EPE =MSE(Flow pre -Flow label )

[0047] Among them, Flow pre and Flow label Denote the optical flow estimated by the ANN model and the true label optical flow, respectively. MSE denotes the mean squared error. This loss function is used with the backpropagation method to train the ANN model. Network training ends when the model loss function stabilizes.

[0048] S4: Initialize the SNN model: the membrane time constant τ and the pulse firing threshold V are the specific parameters of the SNN model. th Perform random initialization. Note that the parameter definition interval is τ∈(0,1), V th ∈(0,+∞).

[0049] S5: Model weight migration: Migrate the ANN weights trained in S3 to the SNN model.

[0050] S6: SNN model fine-tuning: Use the neuromorphic data processed in S2 to train the SNN model. When the model loss function Loss EPE Model fine-tuning ends when the model reaches stability, that is, when the value of the model loss function stops decreasing. The SNN model is trained using the "time-space directional propagation" method to fine-tune the model weights.

[0051] Figure 3 This figure illustrates the results of a neuromorphic optical flow prediction method based on a bio-informatics collaborative spiking neural network, using an embodiment of the present invention. The figure shows the input neuromorphic data, the model-generated optical flow image, and the label optical flow. Experimental results demonstrate that the present method can accurately estimate optical flow using neuromorphic images.

[0052] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A training method based on biological-information synergistic pulse neural network, characterized in that: include: S1. Construct ANN and SNN models with the same structure. The ANN model uses sigmoid neurons, and the SNN model uses leakage-integration-release neurons. S2. Neuromorphic image asynchronously records the brightness change p on each photosensitive element k , the image is represented as: Where K represents p k The number of k ,y k , t k ) represents the k-th brightness change recorded at the x, y pixel at time t; All brightness changes are evenly divided into N groups, and each group of brightness changes is divided into two frames: enhancement and reduction, which are expressed as: in, The ANN model input is: The input of the SNN model is S3, using the ANN model input obtained in step S2 to train the ANN model, and saving the weight parameters of the ANN model after the training is completed; S4. Parameters specific to the SNN model: membrane time constant τ and spike firing threshold V th Perform random initialization and migrate the weight parameters of the ANN model saved in step S3 to the SNN model; S5. Use the SNN model input obtained in step S2 to train the SNN model obtained in step S4; obtain a trained SNN model.

2. The training method based on biological-information collaborative pulse neural network according to claim 1 is characterized in that: The structure of the ANN and SNN models specifically includes: a first convolutional gated recurrent unit, multiple encoding layers, multiple residual modules, multiple decoding layers, a generator, and a second convolutional gated recurrent unit; the input data first enters the convolutional gated recurrent unit, and then enters multiple encoding layers and multiple residual modules in turn; the feature maps of different scales generated by each encoding layer except the last encoding layer and the feature map generated by the last residual module are spliced ​​and upsampled, and then sent to each decoding layer, the generator, and the second convolutional gated recurrent unit in turn for optical flow prediction.

3. The training method based on biological-information synergistic pulse neural network according to claim 2 is characterized in that: It also includes taking the optical flow generated by the second convolutional gated recurrent unit as input and feeding it back to the first convolutional gated recurrent unit to enhance subsequent input data.

4. The training method based on biological-information synergistic pulse neural network according to claim 3 is characterized in that: The endpoint error loss is used to calculate the network loss value during the training of ANN models and SNN models.

5. A neuromorphic optical flow prediction method based on a biological-informatics collaborative spiking neural network, characterized in that: By processing the neural state data to be estimated into the SNN model input form, inputting it into the training method based on the biological-information collaborative pulse neural network described in any one of claims 1-4, and obtaining the optical flow image and label optical flow in the trained SNN model.

Images