A video frame interpolation method based on spiking neural network and event camera

Abstract

The application discloses a video interpolation method based on a pulse neural network and an event camera, and comprises the following steps: acquiring two adjacent frame images in a low-frame-rate RGB video and event streams of a corresponding time period between the two frames collected by an event camera; dividing the event streams according to a certain interframe time of a required reconstructed image, and processing positive polarity event information and negative polarity event information obtained by the division into event grids that are uniformly divided according to time; constructing and training a video interpolation model based on a pulse neural network, including an image processing module, an event stream processing module, an attention module based on event guidance, a pulse fusion module and an image reconstruction module; and performing video interpolation on event grids corresponding to frame images and event streams by using the trained video interpolation model, so that high-energy-efficient video interpolation can be realized under lower calculation amount and energy consumption.

Description

Technical Field

[0001] This invention belongs to the field of image processing and neuromorphic computing, and specifically relates to a video frame interpolation method based on a spiking neural network and an event camera. Background Technology

[0002] Video frame interpolation is an important application topic in computer vision. Its goal is to predict video frames at inter-frame intervals using known low-frame-rate video, thereby increasing the video's frame rate and enhancing the smoothness and continuity of motion. Traditional low-frame-rate RGB cameras struggle to capture high-speed motion information; therefore, frame interpolation techniques based on pure RGB video produce significantly flawed results when dealing with high-speed and non-linear motion scenes.

[0003] To address this shortcoming, event-based video interpolation methods introduce additional event sensors to assist in RGB video interpolation. Unlike traditional RGB cameras that capture images at fixed time intervals, event sensors generate event information based on changes in pixel brightness within the scene, offering extremely high temporal resolution and sensitivity, thus enabling precise capture of high-speed and non-linear motion. With the aid of this additional high temporal resolution information, event-based video interpolation methods exhibit significant performance advantages when dealing with high-speed and complex motion.

[0004] Currently, most video frame interpolation methods based on event cameras are based on artificial neural network (ANN) algorithms. These algorithms' neural network models themselves have a large number of parameters, requiring high computational and energy consumption during inference.

[0005] An event camera, as a neuromorphic visual sensor, has an independent photoelectric sensing module at each pixel. When the brightness change at that pixel exceeds a set threshold, it generates and outputs event data (sometimes called pulse data). It features low power consumption, and its output event stream is sparse and asynchronous. In contrast, ANNs rely on large-scale synchronous and intensive data computation, making it difficult to effectively utilize the sparsity and asynchronicity of the event stream itself. Since only a portion of adjacent image frames typically generate an event stream, while most areas do not, ANN-based methods need to perform calculations even in areas without events, resulting in a large amount of unnecessary computation.

[0006] Spiking Neural Networks (SNNs) are third-generation neural networks inspired by biology. They use models that best fit the mechanisms of biological neurons for computation, achieving a higher level of bio-neural simulation. Unlike ANNs, SNN computation is event-triggered, making them more suitable for handling asynchronous and sparse neuromorphic sensing data. SNN-based algorithms have significantly lower computational and energy consumption compared to ANN-based algorithms. Summary of the Invention

[0007] In view of the above, the purpose of this invention is to provide a video frame interpolation method based on spiking neural networks and event cameras, which overcomes the shortcomings of current ANN-based methods that cannot effectively utilize the sparsity of event cameras, resulting in a large amount of unnecessary computation. This method can achieve high-energy-efficiency video frame interpolation with lower computational load and energy consumption.

[0008] To achieve the above-mentioned objectives, an embodiment provides a video frame interpolation method based on a spiking neural network and an event camera, comprising the following steps:

[0009] Acquire two adjacent frames from a low frame rate RGB video, and an event stream for the corresponding time period between the two frames captured by the event camera;

[0010] The event stream is divided according to a certain inter-frame time of the image to be reconstructed, and the positive and negative events obtained are processed into event grids that are uniformly divided in time.

[0011] The video frame interpolation model is constructed and trained based on a spiking neural network, including an image processing module that generates image feature pulses based on frame images, an event stream processing module that generates event feature pulses based on event grids corresponding to event streams, an event-guided attention module that guides event attention on image feature pulses based on event feature pulses and generates sparse image feature pulses, a pulse fusion module that generates multi-scale and multi-modal feature pulses based on event feature pulses and sparse image feature pulses, and an image reconstruction module that generates intermediate frame images based on multi-scale and multi-modal feature pulses.

[0012] The trained video interpolation model is used to interpolate video frames based on the event grid corresponding to the frame image and the event stream to obtain high frame rate video.

[0013] Preferably, the spiking neural networks corresponding to the image processing module, event stream processing module, and spiking fusion module employ leaking and firing neurons, and the topological connections between neurons are in the form of convolution.

[0014] Preferably, the image processing module and the event stream processing module generate image feature pulses and event feature pulses through multi-layer downscaling; the event-guided attention module uses the event feature pulses of each layer to guide the reduction of the firing frequency of image feature pulses to generate sparse image feature pulses of each layer, thus obtaining multi-scale sparse image feature pulses; the pulse fusion module fuses the event feature pulses and sparse image feature pulses of each layer to generate multi-scale multimodal feature pulses.

[0015] Preferably, in the event-guided attention module, the process of generating sparse image feature pulses based on event pulse features at each layer is as follows:

[0016]

[0017] Here, the symbol ○ represents the Hadamard product, which is the element-wise multiplication operation of tensors, V t+1 and V t S represents the membrane potential at the next time step t+1 and the current time step t, respectively. e and S I These are the input event feature pulses and image feature pulses, respectively, τ leak W is the membrane potential leakage coefficient. e and W I S represents e and S I The corresponding weights;

[0018] Based on the membrane potential V at time t+1 t+1 Sparse image feature pulses are generated through leakage and firing of neurons.

[0019] Preferably, the process of fusing event feature pulses and sparse image feature pulses for each layer by the pulse fusion module is as follows:

[0020] Event feature pulses and sparse image feature pulses are convolved and then concatenated. The concatenated result is then processed by leak and firing neurons to obtain bidirectional multimodal feature pulses.

[0021] Preferably, the image reconstruction module includes a pulse decoder and a pulse deformable convolutional unit;

[0022] Among them, the pulse decoder performs upscaling fusion on multi-scale multimodal feature pulses to obtain bidirectional multimodal feature pulses with the same scale as the input frame image;

[0023] The pulse deformable convolution unit is based on bidirectional multimodal feature pulses. It calculates the bidirectional deformable convolution kernel parameters separately, and uses the bidirectional deformable convolution kernel parameters to perform deformable convolution calculations on two adjacent input frames to obtain two unidirectional prediction images. The two unidirectional prediction images are then fused through image content visibility to obtain the predicted intermediate frame image.

[0024] Preferably, the step of calculating the bidirectional deformable convolution kernel parameters based on bidirectional multimodal feature pulses includes:

[0025] For forward multimodal feature pulses and backward multimodal feature pulses, spatial offset, convolution kernel weights, and convolution kernel biases are calculated based on the weights of the calculated convolution kernel parameters. The spatial offset, convolution kernel weights, and convolution kernel biases constitute the deformable convolution kernel parameters.

[0026] Preferably, the image content visibility is obtained in the following manner:

[0027] V 0t =sigmoid(W0S 0t +W1S 1t )

[0028] V 1t =1-V 0t

[0029] Among them, S 0t and S 1t These represent the forward multimodal characteristic pulse and the backward multimodal characteristic pulse, respectively, with W0 and W1 representing the weighting parameters, and V... 0t V represents the weight of the visible portion of the intermediate frame image predicted from the previous frame image. 1t V represents the weight of the visible portion of the intermediate frame image predicted from the subsequent frame image. 0t and V 1t This is collectively referred to as image content visibility.

[0030] Preferably, the predicted intermediate frame image is obtained by fusing the two unidirectional predicted images through image content visibility, as shown below:

[0031] I t =V 0t I 0t +V 1t I 1t

[0032] Among them, I 0t I represents the forward predicted image calculated based on forward multimodal feature impulses. 1t I represents the inverse predicted image calculated based on inverse multimodal feature impulses. t This represents the predicted intermediate frame image.

[0033] Preferably, when training the video frame interpolation model, supervised learning is used with L1 loss, and the weight parameters are updated using the impulse gradient descent method. For the non-differentiable unit impulse step function in the leaking and firing neurons, a surrogate gradient is used instead.

[0034] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0035] This invention applies a spiking neural network to the video frame interpolation process based on an event camera. While leveraging the high temporal resolution information of the event camera itself to improve the video frame interpolation effect, it also fully utilizes the sparsity and asynchronicity of pulse information, significantly reducing the computational load and power consumption during frame interpolation. By using a spiking neural network to process both images and event streams simultaneously, and introducing an event-guided attention module, the pulse firing frequency of the image processing module is reduced, further decreasing computational load and power consumption. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a video frame interpolation method based on a spiking neural network and an event camera, provided in the embodiment.

[0038] Figure 2 This is a schematic diagram of the module structure of the video frame interpolation model provided in the embodiment;

[0039] Figure 3 This is a schematic diagram of the specific network structure of the video frame interpolation model provided in the embodiment;

[0040] Figure 4 This is a schematic diagram of the structure of the pulse deformable convolutional unit provided in the embodiment;

[0041] Figure 5 This is a flowchart illustrating the training process of the video frame interpolation model used in the embodiment.

[0042] Figure 6 This is a comparison chart of the video frame interpolation results from the embodiment and the results from other methods. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0044] The video frame interpolation scheme based on spiking neural networks and event cameras provided in this invention is designed, implemented, and tested using computer simulation. The simulation experiments are based on a Python 3.8 environment, using the PyTorch framework to simulate the spiking neural network. Figure 1 As shown in the embodiment, the video frame interpolation method based on a spiking neural network and an event camera includes the following steps:

[0045] S1, acquire two adjacent frames in a low frame rate RGB video, and the event stream of the corresponding time period between the two frames captured by the event camera.

[0046] Specifically, two adjacent frames I0 and I1 are acquired from a low frame rate RGB video, where I0 is the previous frame and I1 is the next frame. Event stream messages e are acquired from the event camera within the corresponding time period T∈[0,1]. 01 The goal of the video frame interpolation task is to generate an intermediate frame image I at a certain time t between consecutive image frames I0 and I1. t .

[0047] S2, divide the event stream according to a certain time interval of the image to be reconstructed, and process the obtained positive and negative event information into event grids that are uniformly divided over time.

[0048] In the embodiment, the event stream e is first divided according to the inter-frame time t. 01 Events are divided into events based on their chronological order. 0t and event stream e t1 For event stream e t1 Event flow reversal is required. First, reverse the chronological order, then reverse the polarity of the events. The reversed event flow is e. 1tThe inverted event stream differs from the uninverted event stream in that: the temporal order is reversed, and the positive and negative polarities of the same event are reversed. When an event stream contains positive polarity events, the inverted stream becomes a negative polarity event, and vice versa. The preprocessing stage requires converting the event stream from discrete asynchronous event address format (AER) to a grid representation. The event grid representation has a fixed number of time steps and spatial dimensions, allowing for better processing with image data. This invention grids positive and negative polarity events separately, resulting in an event grid size of 2×N×H×W, where N is the number of time steps (16 in this invention), and H and W are the spatial resolution of the event camera. The event grid uses E... 0t and E 1t This indicates that the final preprocessing results in data I0, I1, and E. 0t and E 1t The image is fed into the trained video interpolation model to predict the intermediate frame image I. t .

[0049] S3 is a video frame interpolation model built and trained based on a spiking neural network.

[0050] This invention utilizes a spiking neural network to implement a video frame interpolation method based on an event camera, achieving a complementary advantage between event cameras and RGB cameras. Applying the event-based computation paradigm of the spiking neural network itself better leverages the sparsity characteristics of the event camera. Combined with a neuromorphic computing chip, it enables low-power video frame interpolation applications, showing promise for edge device applications. Furthermore, this invention broadens the application areas of neuromorphic computing and spiking neural networks, and is of significant value for research into the application of spiking neural networks.

[0051] Specifically, such as Figure 2 As shown, the video frame interpolation model based on a spiking neural network includes an image processing module, an event stream processing module, an event-guided attention module, a pulse fusion module, and an image reconstruction module. The image processing module generates image feature pulses based on the frame image; the event stream processing module generates event feature pulses based on the event grid corresponding to the event stream; the event-guided attention module performs event attention on the image feature pulses guided by the event feature pulses and generates sparse image feature pulses; the pulse fusion module generates multi-scale, multi-modal feature pulses based on the event feature pulses and sparse image feature pulses; and the image reconstruction module generates intermediate frame images based on the multi-scale, multi-modal feature pulses. The image processing module, event stream processing module, event-guided attention module, and pulse fusion module serve as the encoding part, based on the input I0 and E... 0t Obtaining positive multi-scale multimodal pulse features F0t , by I1 and E 1t Obtaining the inverse multi-scale multimodal pulse feature F 1t The image reconstruction module, as the decoding part, is based on F. 0t and F 1t Decoding and reconstruction yields intermediate frame image I t .

[0052] In this embodiment, the spiking neural networks corresponding to the image processing module, event stream processing module, and spiking fusion module employ Leaky Integrate-And-Fire Models (LIF), and the topological connections between neurons are implemented using convolution. The basic parameters of a LIF neuron include: synaptic weight w, membrane potential time constant τ, membrane potential V, and membrane potential threshold V. th For each input pulse at each time step, the input current is first obtained based on the synaptic weights. Then, the membrane potential voltage is updated according to the membrane potential time constant and the input current. Finally, it is determined whether the membrane potential threshold is exceeded. If the threshold is exceeded, a pulse is fired and the membrane potential is reset. Let the layer number of the neuron be l, and the time step be t. The update formula for a time-discrete LIF neuron is as follows:

[0053] I in =WS in

[0054]

[0055]

[0056]

[0057] Among them, S in Indicates the input pulse, I in This represents the input current to the neuron cell body after the input pulse passes through the synapse. and This represents the membrane potential values ​​at two adjacent moments. S is the accumulated membrane potential value after the input pulse. out H represents the output pulse, and H represents the step function, with the following specific form:

[0058]

[0059] like Figure 3As shown, Conv-LIF is short for LIF spiking neural network layer with convolutional connections, EA is short for event-guided attention layer, SF is short for spiking fusion layer, and DConv is short for spiking variable convolution. The image processing module and event stream processing module generate image feature spiking and event feature spiking through multi-layer downscaling. Due to the density of image data, the firing rate of neurons in the image processing module is much higher than that in the event processing module. Since event information itself indicates the range of changing regions in the image, neurons can be guided to focus on areas where events have occurred. This invention embeds an event-guided attention module in the image processing module, using the spiking output from the event processing module neurons to guide the firing of neurons in the image processing module, thus reducing the firing frequency of the image processing module's spiking while serving as an attention mechanism. Specifically, the event-guided attention module uses the event feature spiking of each layer to guide the reduction of the firing frequency of image feature spiking, generating sparse image feature spiking for each layer, resulting in multi-scale sparse image feature spiking. The spiking fusion module fuses the event feature spiking and sparse image feature spiking of each layer to generate multi-scale, multi-modal feature spiking.

[0060] like Figure 3 As shown, in the event-guided attention module, the inputs are event feature pulses and image feature pulses. The attention tensor is calculated from the event feature pulses and applied to the membrane potential tensor. That is, the process of generating sparse image feature pulses based on event pulse features in each layer is as follows:

[0061]

[0062] Among them, symbols V represents the Hadamard product, which is an element-wise multiplication operation of tensors. t+1 and V t S represents the membrane potential at the next time step t+1 and the current time step t, respectively. e and S I These are the input event feature pulses and image feature pulses, respectively, τ leak W is the membrane potential leakage coefficient. e and W I S represents e and S I The corresponding weights;

[0063] Based on the membrane potential V at time t+1 t+1 Sparse image feature pulses are generated after processing by LIF neurons.

[0064] like Figure 3 As shown, in the pulse fusion module, the fusion process of event feature pulses and sparse image feature pulses for each layer is as follows:

[0065] In each layer, the neuronal input currents obtained after passing the event feature pulses and sparse image feature pulses through convolutional synapses are summed, and then the summed input currents are processed by LIF neurons to obtain bidirectional multimodal feature pulses.

[0066] like Figure 3 As shown, the image reconstruction module includes a pulse decoder and a pulse deformable convolutional unit. The pulse decoder performs upscaling fusion on multi-scale multimodal feature pulses to obtain bidirectional multimodal feature pulses with the same scale as the input frame image. The corresponding spiking neural network also uses leaky and firing neurons.

[0067] like Figure 4 As shown, the pulse deformable convolution unit, based on bidirectional multimodal feature pulses, calculates bidirectional deformable convolution kernel parameters separately. Using these parameters, it performs deformable convolution calculations on two adjacent input frames to obtain two unidirectional prediction images. These two unidirectional prediction images are then fused using image content visibility to obtain the predicted intermediate frame image. The specific process is as follows:

[0068] (a) Predicting spatial offset from pulses:

[0069] Δx=W x S

[0070] Δy=W y S

[0071] Where S represents the input pulse, which is the positive multimodal characteristic pulse S0. ot Or inverse multimodal characteristic pulse S 1t W x W y The weights represent the predicted spatial offsets, and Δx and Δy represent the predicted spatial offsets of the deformable convolution kernel.

[0072] (b) Predicting weights from impulses:

[0073] W d =W w S

[0074] Among them, W w W represents the weights used to predict deformable convolution kernel weights. d These are the predicted convolutional kernel weights;

[0075] (c) Predicting bias from pulses:

[0076] b = W b S

[0077] Among them, W brepresents the weights for predicting deformable convolution kernel bias, and b represents the predicted convolution kernel bias;

[0078] (d) From F 0t Calculate the convolution kernel K0, and then perform convolution on I0 using this kernel to obtain I. 0t Similarly from F 1t Calculate the convolution kernel K1 and perform convolution on I1 to obtain I. 1t The deformable convolution is calculated as follows:

[0079]

[0080] Among them, W i Δx i Δy i b are the deformable convolution kernel parameters calculated from the pulses mentioned above;

[0081] (e) Due to the presence of occlusion, it is necessary to comprehensively consider F t0 and F t1 Image content visibility is calculated as follows:

[0082] V 0t =sigmoid(W0S 0t +W1S 1t )

[0083] V 1t =1-V 0t

[0084] Among them, S 0t and S 1t These represent the forward multimodal characteristic pulse and the backward multimodal characteristic pulse, respectively, with W0 and W1 representing the weighting parameters, and V... 0t V represents the weight of the visible portion of the intermediate frame image predicted from the previous frame image. 1t V represents the weight of the visible portion of the intermediate frame image predicted from the subsequent frame image. 0t and V 1t This is collectively referred to as image content visibility.

[0085] (f) Based on image content visibility, bidirectional prediction I 0t and I 1t The final predicted intermediate frame image I is obtained by blending. t :

[0086] I t =V 0t I 0t +V 1t I 1t

[0087] Among them, I 0tI represents the forward predicted image calculated based on forward multimodal feature impulses. 1t This represents the inverse prediction image calculated based on inverse multimodal feature pulses.

[0088] The above video frame interpolation model was trained on a public dataset, and the training flowchart is as follows: Figure 5 As shown. The dataset contains consecutive high frame rate video images and event streams between video image frames. A preprocessed training example includes: consecutively spaced video frames I0, I1, and a bidirectional event stream grid E. 0t and E 1t and real intermediate frame I t During training, the spatial resolution of both the image and event stream is uniformly set to 256×256, and data augmentation is performed using random horizontal and vertical flipping, random time reversal, and random rotation. This invention employs supervised learning to train the spiking neural network, using L1 loss as the training loss function. This invention uses the Impulsive Gradient Descent (STBP) method to update the weight parameters. For non-differentiable unit impulse step functions, a surrogate gradient is used instead. The surrogate gradient function used in this invention is as follows:

[0089]

[0090] Where 'a' is a hyperparameter that controls the shape of the surrogate gradient function, and in this embodiment, it is set to 10.0.

[0091] S4 utilizes the trained video interpolation model to perform video interpolation based on the event grid corresponding to the frame image and the event stream, in order to obtain high frame rate video.

[0092] In this embodiment, based on the above-described spiking neural network structure and training method, the performance and energy consumption of the video frame interpolation model were tested on a public dataset. The spatial resolution of the test data was uniformly set to 256×256, and the number of time steps for the spiking neural network was 4.

[0093] Table 1 shows a comparison of metrics between the proposed video interpolation method (Spike-EFI) based on spiking neural networks and event cameras and other event interpolation algorithms based on artificial neural networks. Figure 6 The paper presents and compares the interpolation results of the proposed video interpolation method based on spiking neural networks and event cameras with those of other video interpolation algorithms based on artificial neural networks.

[0094] Table 1

[0095] Method Name PSNR / db SSIM Time-Lens 28.36 0.932 Time-lens++ 28.56 -- Spike-EFI 29.12 0.889

[0096] Table 2 lists the average spike firing rate (ASFR) of each SNN layer in the video frame interpolation model proposed in this invention during testing, where H = 256 and W = 256.

[0097] Table 2

[0098]

[0099]

[0100] Table 3 compares the computational load and energy consumption of the spiking neural network-based method proposed in this invention with other artificial neural network-based methods.

[0101] Table 3

[0102] Method Name Parameter quantity / M Number of operations / G <![CDATA[Energy consumption / 10 -3 J]]> Time-Lens 72.2 103.4 475.6 Time-Lens++ 53.9 85.8 395.9 Spike-EFI 1.35 25.9 44.2

[0103] The metrics in Table 1 above are the peak signal to noise ratio (PSNR) and structural similarity (SSIM).

[0104] The specific formula for calculating the average pulse rate of fire (ASFR) in Table 2 above is as follows:

[0105]

[0106] Where N is the number of neurons, T is the number of time steps, which is set to 4 in the embodiment, and S(n,t) represents the pulse firing of the nth neuron at time t. Numerically, 1 indicates that a pulse was fired, and 0 indicates that no pulse was fired.

[0107] The number of operations in Table 3 above represents the number of multiply-accumulate operations (MACs) performed by the neural network during the entire process of interpolating an image. For convolutional connections, let the kernel size be K. W ×K H Number of input channels C in Number of output channels C out If the output size is W×H, then its MACs calculation formula is as follows:

[0108] MACs Conv =K W ×K H ×C in ×C out ×W×H

[0109] For a fully connected layer, let the input dimension be C.in The output dimension is C out The formula for calculating MACs is:

[0110] MACs Dense =C in ×C out

[0111] For the SNN layer of the LIF-based video frame interpolation model, the computational operation consists of two parts: synaptic weight sum calculation and membrane potential update calculation. In the synaptic weight sum calculation phase, the MACs are the MACs of the corresponding ANN connections multiplied by the pulse firing rate. In the membrane potential update phase, each neuron needs to perform an additional MAC. Assuming the SNN layer pulse firing rate is SR and the number of neurons is N, the formula for calculating the MACs for one time step of the synaptic weight sum calculation is as follows:

[0112] MACs SNN =MACs ANN ×SR

[0113] The formula for calculating MACs for membrane potential update over one time step is as follows:

[0114] MACs SNN-MP =N

[0115] The energy consumption calculation principle in Table 3 above is to multiply the number of MAC calculations by the energy consumption required for a single MAC calculation. Due to its event-based computational nature, SNNs do not actually require multiplication of activation values ​​and weights for weights and stages; they only need to add the weights triggered by pulses. The actual energy consumption is the energy consumed by the synaptic weight accumulation operation (Accumulate, AC) and membrane potential update calculation. Therefore, the energy required for a single MAC calculation in an SNN is significantly lower than that required for an ANN calculation. It is known that under 45nm CMOS technology, the energy consumption for a single MAC calculation for a 32-bit floating-point number is 4.6pJ, and the energy consumption for a single AC calculation is 0.9pJ. Therefore, in the testing of this invention, the energy consumption for a single MAC calculation in an ANN is taken as e. ANN =4.6pJ, the energy consumption for a single MAC calculation for the synaptic weights of an SNN is taken as e. SNN =0.9pJ, the energy consumption of a single MAC calculation for membrane potential update is taken as e SNN-MP =4.6pJ.

[0116] Based on the above settings, the energy consumption E of ANN and SNN is... ANN and E SNN The calculation formula is as follows:

[0117] E ANN =MACs ANN×e ANN

[0118] E SNN =MACs SNN ×e SNN +MACs SNN-MP ×e SNN-MP

[0119] For the attention module, since the attention weights need to be multiplied by the membrane potential, an additional element-wise multiplication operation is performed. In this invention, the energy consumption e of the 32-bit floating-point multiplication operation is taken during testing. MUL = 3.7pJ, the attention module has N neurons, and its additional energy consumption ΔE EA The calculation formula is as follows:

[0120] ΔE EA =N×e MUL

[0121] The test results above show that the video interpolation method based on spiking neural networks and event cameras proposed in this invention maintains the performance indicators of event-based video interpolation methods while significantly reducing energy consumption, enabling low-power video interpolation applications. Combined with neuromorphic computing hardware and event cameras, it has potential application prospects.

[0122] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video frame interpolation method based on a spiking neural network and an event camera, characterized in that, Includes the following steps: Acquire two adjacent frames from a low frame rate RGB video, and an event stream for the corresponding time period between the two frames captured by the event camera; The event stream is divided according to a certain inter-frame time of the image to be reconstructed, and the positive and negative events obtained are processed into event grids that are uniformly divided in time. The video frame interpolation model is constructed and trained based on a spiking neural network, including an image processing module that generates image feature pulses based on frame images, an event stream processing module that generates event feature pulses based on event grids corresponding to event streams, an event-guided attention module that guides event attention on image feature pulses based on event feature pulses and generates sparse image feature pulses, a pulse fusion module that generates multi-scale and multi-modal feature pulses based on event feature pulses and sparse image feature pulses, and an image reconstruction module that generates intermediate frame images based on multi-scale and multi-modal feature pulses. The trained video interpolation model is used to perform video interpolation based on the event grid corresponding to the frame image and the event stream to obtain high frame rate video. The image reconstruction module includes a pulse decoder and a pulse deformable convolutional unit; Among them, the pulse decoder performs upscaling fusion on multi-scale multimodal feature pulses to obtain bidirectional multimodal feature pulses with the same scale as the input frame image; The pulse deformable convolution unit is based on bidirectional multimodal feature pulses. It calculates the bidirectional deformable convolution kernel parameters separately, and uses the bidirectional deformable convolution kernel parameters to perform deformable convolution calculations on two adjacent input frames to obtain two unidirectional prediction images. The two unidirectional prediction images are then fused through image content visibility to obtain the predicted intermediate frame image.

2. The video frame interpolation method based on a spiking neural network and an event camera according to claim 1, characterized in that, The spiking neural networks corresponding to the image processing module, event stream processing module, and spiking fusion module employ leaking and firing neurons, and the topological connections between neurons are in the form of convolution.

3. The video frame interpolation method based on a spiking neural network and an event camera according to claim 1, characterized in that, The image processing module and the event stream processing module generate image feature pulses and event feature pulses through multi-layer downscaling; the event-guided attention module uses the event feature pulses of each layer to guide the reduction of the firing frequency of image feature pulses to generate sparse image feature pulses of each layer, thus obtaining multi-scale sparse image feature pulses; the pulse fusion module fuses the event feature pulses and sparse image feature pulses of each layer to generate multi-scale multimodal feature pulses.

4. The video frame interpolation method based on a spiking neural network and an event camera according to claim 1, characterized in that, In the event-guided attention module, the process of generating sparse image feature pulses based on event pulse features at each layer is as follows: Here, the symbol ○ represents the Hadamard product, which is the element-wise multiplication operation of tensors. and They represent the next moment. and the current moment membrane potential, and These are the input event feature pulses and image feature pulses, respectively. The membrane potential leakage coefficient is... and express and The corresponding weights; based on membrane potential at time 1 Sparse image feature pulses are generated through leakage and firing of neurons.

5. The video frame interpolation method based on a spiking neural network and an event camera according to claim 3, characterized in that, The process by which the pulse fusion module fuses the event feature pulses and sparse image feature pulses of each layer is as follows: Event feature pulses and sparse image feature pulses are convolved and then concatenated. The concatenated result is then processed by leak and firing neurons to obtain bidirectional multimodal feature pulses.

6. The video frame interpolation method based on a spiking neural network and an event camera according to claim 1, characterized in that, The bidirectional multimodal feature pulse-based approach calculates the bidirectional deformable convolution kernel parameters, including: For forward multimodal feature pulses and backward multimodal feature pulses, spatial offset, convolution kernel weights, and convolution kernel biases are calculated based on the weights of the calculated convolution kernel parameters. The spatial offset, convolution kernel weights, and convolution kernel biases constitute the deformable convolution kernel parameters.

7. The video frame interpolation method based on a spiking neural network and an event camera according to claim 1, characterized in that, The visibility of the image content is obtained in the following ways: in, and These represent the forward multimodal characteristic pulse and the backward multimodal characteristic pulse, respectively. and Represents the weight parameters. This represents the weight of the visible portion of the intermediate frame image predicted from the previous frame image. This represents the weight of the visible portion of the intermediate frame image predicted from the subsequent frame image. and This is collectively referred to as image content visibility.

8. The video frame interpolation method based on a spiking neural network and an event camera according to claim 6, characterized in that, The intermediate predicted frame image is obtained by fusing two unidirectional predicted images through image content visibility, and is represented as follows: in, This represents a forward-predicted image calculated based on forward multimodal feature impulses. This represents the inverse predicted image calculated based on inverse multimodal feature impulses. This represents the predicted intermediate frame image.

9. The video frame interpolation method based on a spiking neural network and an event camera according to claim 6, characterized in that, When training the video frame interpolation model, supervised learning is used with L1 loss, and the weight parameters are updated using the impulse gradient descent method. For the non-differentiable unit impulse step function in the leaking and firing neurons, a surrogate gradient is used instead.

Images