An event camera optical flow estimation method and device based on a self-attention mechanism

Abstract

The application provides an event camera optical flow estimation method and device based on a self-attention mechanism, and relates to the technical field of computer vision, and comprises the following steps: acquiring an original data stream and converting the original data stream into a discretized event frame; inputting the event frame into a pulse neural network; processing the event frame according to an encoding layer in the pulse neural network to obtain pulse output corresponding to the encoding layer, wherein the encoding layer processes the event frame according to a cross-shaped attention module and a time attention module; processing the pulse output according to a residual block and a decoding layer to determine a predicted optical flow, wherein the decoding layer performs up-sampling using transpose convolution; and determining the total loss of the predicted optical flow according to the trained hybrid neural network. The input object is processed by the cross-shaped attention module and the time attention module, the complete image dependency is captured, the information is selectively focused, the frames irrelevant to the final result are automatically filtered, the optical flow prediction capability is improved, and the calculation efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an event camera optical flow estimation method and device based on a self-attention mechanism. BACKGROUND

[0002] Optical flow estimation is an important research direction in image processing and computer vision research, which allows us to visualize the motion field in the scene and lay the foundation for more complex tasks such as action recognition, motion segmentation and target tracking. In the past few years, optical flow estimation has been dominated by traditional computer vision algorithms based on frame cameras, but due to the low temporal resolution, high latency and large amount of data of traditional frame cameras, it is difficult to capture the rapid changes of objects in challenging scenes such as high-speed motion, and the output image frames will have motion blur, repeated exposure and other problems, resulting in a sharp decline in the performance of optical flow estimation algorithms.

[0003] Event cameras are a new type of neuromorphic visual sensor inspired by biological vision systems, which independently detect the brightness changes of each pixel point and generate asynchronous event stream data (x, y, t, p), which includes timestamp t, pixel address (x, y) and brightness change polarity p, so it can overcome the limitations of traditional frame cameras and has unique advantages such as high temporal resolution (microsecond level), low latency (microsecond level), low power consumption (10mW), high dynamic range (120-143dB), and can obtain instantaneous motion data, which is beneficial for continuous optical flow estimation.

[0004] The third generation of neural networks, Spiking Neural Networks (SNN), uses a more biologically interpretable spiking neuron model as the basic unit, provides an event-driven computing form and takes advantage of the inherent sparsity of spike events to respond to events in a low-power and almost delay-free manner, so the combination of spiking neural networks and event cameras has natural advantages. However, due to the spike disappearance phenomenon and the lack of suitable training techniques, the performance problem of spiking neural networks has become the biggest obstacle to their application in various tasks. To solve this problem, some researchers have proposed a deep hybrid neural network (SNN-ANN) architecture that integrates spiking neural networks and analog neural networks to efficiently estimate the optical flow output by sparse event cameras. SUMMARY

[0005] To solve the above problems, an event camera optical flow estimation method and device based on a self-attention mechanism are proposed.

[0006] The first aspect of the present application proposes an event camera optical flow estimation method based on a self-attention mechanism, comprising:

[0007] obtaining an original data stream and converting the original data stream into discretized event frames, inputting the event frames into a spiking neural network;

[0008] processing the event frames according to an encoding layer in the spiking neural network to obtain a spiking output corresponding to the encoding layer, wherein the encoding layer processes the event frames according to a cross-attention module and a temporal attention module;

[0009] processing the spiking output according to a residual block and a decoding layer to determine a predicted optical flow, wherein the decoding layer performs up-sampling using a transpose convolution;

[0010] determining a total loss of the predicted optical flow according to the trained hybrid neural network.

[0011] Optionally, the obtaining an original data stream and converting the original data stream into discretized event frames comprises:

[0012] tracking the original data stream by an event camera;

[0013] discretizing the original data stream into two groups of event frames in a time dimension according to a preset threshold value set by a preset setting, wherein each group of event frames contains N event frames obtained by accumulating events from a timestamp of a previous frame to a current timestamp, and each event frame is composed of two channels.

[0014] Optionally, the processing the event frames according to an encoding layer in the spiking neural network to obtain a spiking output corresponding to the encoding layer, wherein the encoding layer processes the event frames according to a cross-attention module and a temporal attention module, comprises:

[0015] convolving the event frames by the encoding layer to obtain a current membrane potential;

[0016] merging the current membrane potential with a membrane potential accumulated after a previous encoding layer to determine a merged membrane potential;

[0017] processing the merged membrane potential by the cross-attention module to determine an attention map;

[0018] processing the attention map by the temporal attention module to determine the spiking output.

[0019] Optionally, the processing the merged membrane potential by the cross-attention module to determine an attention map comprises:

[0020] for a given merged membrane potential, the cross-attention module applies two filtered encoding layers to generate two feature maps, respectively ​ and ,in, , It is the number of channels less than C;

[0021] The feature map is obtained by clustering. and Obtaining attention maps For feature maps Each location in the spatial dimension Determine the vector Meanwhile, in position From the same row or column of the feature map Extracting feature vectors to obtain a set ,in, yes The One element, The clustering operation is defined by the following expression:

[0022] ,

[0023] in, Features and The correlation between them , .

[0024] Optionally, determining the pulse output includes:

[0025] The statistical vector is calculated using the following formula:

[0026] ,

[0027] in It is the first The layer in the first A spatial input tensor with a time step of C, where C is the number of channels. ;

[0028] Based on the statistical vector, the attention values ​​for the training and inference phases are calculated, and the formula is as follows:

[0029] ,

[0030] in , It is a trainable parameter matrix. This is the set attention threshold, δ is the ReLU activation function, and σ is the Sigmoid activation function. It is the Heaviside step function.

[0031] Optionally, before the determining the total loss of the predicted optical flow, comprising:

[0032] training the hybrid neural network according to a self-supervised learning method.

[0033] Optionally, the determining the total loss of the predicted optical flow, comprising:

[0034] obtaining a photometric consistency loss, formulated as follows:

[0035] ,

[0036] wherein, is a pixel intensity of the first grayscale image, is a pixel intensity of the second grayscale image, is a Charbonnier loss function, and r and η are constant values set during training;

[0037] obtaining a smoothness loss, formulated as follows:

[0038] ,

[0039] wherein, and are a height and a width of the predicted optical flow, respectively;

[0040] obtaining the total loss of the predicted optical flow according to the photometric consistency loss and the smoothness loss, formulated as follows:

[0041] ,

[0042] wherein, represents a weight factor.

[0043] The second aspect of the present application proposes an event camera optical flow estimation device based on a self-attention mechanism, comprising:

[0044] an input module configured to obtain an original data stream and convert the original data stream into a discretized event frame, and input the event frame into a spiking neural network;

[0045] an encoding module configured to process the event frame according to an encoding layer in the spiking neural network to obtain a spiking output corresponding to the encoding layer, wherein the encoding layer processes the event frame according to a cross-attention module and a time-attention module;

[0046] a decoding module configured to process the spiking output according to a residual block and a decoding layer to determine a predicted optical flow, wherein the decoding layer performs up-sampling using a transpose convolution;

[0047] The loss obtaining module is configured to determine the total loss of the predicted optical flow according to the trained hybrid neural network.

[0048] In a third aspect of the present application, an event camera is provided, which is a neuromorphic vision sensor. When a pixel point independently detects a change in illumination intensity and asynchronously outputs a raw event stream signal, the method according to any one of the first aspect is implemented.

[0049] The technical solutions provided by the embodiments of the present disclosure at least have the following beneficial effects:

[0050] The first aspect introduces a cross-shaped attention module to capture complete image dependency, make up for the defects of sparse event stream, generate dense and pixel-level scene information, and reduce the algorithm time and space complexity. The second aspect introduces a time attention module to selectively focus on information, automatically filter frames irrelevant to the final result, effectively extract time and space features from the event stream, process the event stream more effectively than the pure SNN structure while retaining the event triggering characteristics of the SNN, improve the optical flow prediction capability, and improve the calculation efficiency.

[0051] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0052] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, in which:

[0053] Figure 1 FIG. 1 is a flowchart of an event camera optical flow estimation method based on a self-attention mechanism according to an example embodiment of the present application;

[0054] Figure 2 FIG. 2 is a flowchart of another event camera optical flow estimation method based on a self-attention mechanism according to an example embodiment of the present application;

[0055] Figure 3 FIG. 3 is a network structure diagram of an event camera optical flow estimation method based on a self-attention mechanism according to an example embodiment of the present application;

[0056] Figure 4 FIG. 4 is a visualization diagram of a cross-shaped attention mechanism in an example embodiment of the present application;

[0057] Figure 5 FIG. 5 is a block diagram of an event camera optical flow estimation device based on a self-attention mechanism according to an example embodiment of the present application;

[0058] Figure 6 FIG. 6 is a schematic diagram of an event camera hardware in an example embodiment of the present application. Detailed Implementation

[0059] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0060] Figure 1 This is an example of an event camera optical flow estimation method based on a self-attention mechanism, as illustrated in the exemplary embodiments of this application. Figure 1 As shown, it includes:

[0061] Step 101: Obtain the raw data stream and convert it into discretized event frames, then input the event frames into the spiking neural network.

[0062] Brightness changes at event camera pixels are caused by the movement of the target object or sensor. When multiple pixels simultaneously request event output, these events are output asynchronously with sub-microsecond delays, forming an event stream. The event stream data is represented as follows: This includes the event's pixel coordinates (x and y positions), trigger time (t), and the polarity (p) of the brightness change. Here, each ON / OFF polarity corresponds to an increase or decrease in pixel intensity, respectively. The event camera tracks the logarithmic intensity at each pixel array element. The change in logarithmic strength is considered as long as it exceeds a set threshold. This will result in the following discrete events:

[0063] .

[0064] In step 101, a discretized input encoding scheme is employed to preserve the spatial and temporal information of events for the SNN. Specifically, the temporal dimension within a time window is discretized into two groups, each containing N event frames obtained by accumulating events from the timestamp of the previous frame to the current timestamp. Each of these event frames also consists of two channels for the ON / OFF polarity of the event, and the input representation is as follows: Figure 3 As shown in (a). Therefore, the sequence of event frames from these four channels passes sequentially through the spiking neural network portion of the entire network.

[0065] Step 102: Process the event frame according to the coding layer in the spiking neural network to obtain the pulse output corresponding to the coding layer. The coding layer processes the event frame according to the cross-attention module and the temporal attention module.

[0066] like Figure 2 As shown, step 102 also includes:

[0067] Step 201, the encoding layer convolves the event frame to obtain the current membrane potential.

[0068] First, the encoding layer convolves the input event frame to determine the current membrane potential.

[0069] Step 202, the current membrane potential is merged with the membrane potential accumulated after the previous encoding layer to determine the merged membrane potential.

[0070] The merged membrane potential of the current membrane potential and the membrane potential accumulated after the previous encoding layer is merged as the input of the cross-attention module.

[0071] Step 203, the merged membrane potential is processed by the cross-attention module to determine the attention map.

[0072] In the embodiment of the application, the four encoding layers of the pulse neural network are respectively pixel modeled by the cross-attention module, all relevant information is collected, and down-sampling is performed at each layer. The accumulated output of each encoding layer has a spatial size of , which is respectively input to the cross-attention module, and a new reduced membrane potential is generated after processing. The feature map collects the scene information of each pixel on its cross-attention path. Then, the dense target feature is connected with the local representation feature, and one or more convolutional layers are traversed.

[0073] As shown in Figure 4 , the input feature map of the cross-attention module has a spatial size of , and the scene information is collected by using the weighted sum method. Each position in the feature map is sparsely connected to other positions in the same row and column, resulting in a predicted attention feature map with weights, as shown in Figure 3 , the cross-attention module captures all rich scene information in the horizontal and vertical directions with a spatio-temporal complexity of .

[0074] In addition, as shown in Figure 4 , for a given merged membrane potential , the cross-attention module applies two filtered encoding layers on to generate two feature maps and , where , is a channel number less than C;

[0075] The attention map is obtained by clustering the feature maps and , and for the feature map each position in the spatial dimension , determine vector At the same time, in the position extract the feature vector from the feature map get the set wherein, is the th element of , The clustering operation defines the expression as:

[0076] ,

[0077] wherein, is the correlation between the features and , , .

[0078] In addition, another layer of filter convolution layer is applied to generate . Similar to , at each position Figure 4 of the feature map , the vector and the set are obtained, and the scene information is collected through the following clustering operation:

[0079] ,

[0080] wherein is the feature vector in the output feature map at position . is the scalar value at channel .

[0081] The scene information can be added to the local feature H to enhance the pixel representation.

[0082] In addition, after processing the input feature map using the cross-shaped attention module, the data is further processed using the IF neuron for further processing.

[0083] wherein the IF neuron model describes the action potential as an event, and transmits the output signal in the form of a pulse event over time. At each time step, the input pulse of each encoding layer is weighted to produce a current, which is then integrated into the neuron membrane potential ( ). It is calculated as follows:

[0084] ,

[0085] wherein​ represents the membrane potential at the time step represents the synaptic weight, represents the pulse event of the previous layer. When the membrane potential exceeds the set activation threshold , the neuron emits a pulse signal, and the membrane potential is reset to 0.

[0086] Step 204, processing the attention map through the time attention module to determine the pulse output.

[0087] As Figure 3 shown in (c) of the present application, for the time attention module proposed in the present application, the concept of attention is extended to the temporal input to obtain the statistical characteristics of events at different time points, generate attention scores, and then weight the events according to the scores. The attention score should not only be based on the statistical characteristics of the input at the current time step, but also consider information from adjacent frames. Specifically, for each step of the spatial input, the statistical vector is calculated through the compression operation without considering the batch size:

[0088] ,

[0089] where is the spatial input tensor of the th layer at the th time step, C is the number of channels, .

[0090] According to the statistical vector s, the attention value d in the training and inference stages is calculated as follows:

[0091] ,

[0092] where , is a trainable parameter matrix, is a set attention threshold, δ andσ are ReLU and Sigmoid activation functions respectively, is the Heaviside step function, which satisfies: when , , otherwise , the calculated d value is multiplied by the spatial input to obtain a new spatial input .

[0093] Thus, the output pulse of the membrane potential in the cross-attention module-IF neuron model layer is processed and passed to the subsequent simulated neural network layer.

[0094] ​The pulse outputs of the encoding layers are collected in their corresponding output accumulators after being processed by cross-attention and temporal attention mechanisms, respectively, until all consecutive event images are processed.

[0095] At step 103, the pulse outputs are processed according to the residual blocks and the decoding layers to determine the predicted optical flow, wherein the decoding layers perform up-sampling using transpose convolution.

[0096] In the embodiments of the present application, the pulse outputs from the last encoding layer pass through two residual blocks, while the pulse outputs of the first three encoding layers are used as the inputs of the corresponding decoding layers. Then each decoding layer performs up-sampling using transpose convolution to generate an intermediate flow prediction, and the intermediate flow prediction, the transpose convolution output and the corresponding activation output of the encoding layer are all connected together to construct the input of the next decoding layer, and the last decoding layer generates a full-scale optical flow prediction with the same dimension as the input frame.

[0097] At step 104, the total loss of the predicted optical flow is determined according to the trained hybrid neural network.

[0098] In the embodiments of the present application, the hybrid neural network proposed in the present application is trained by using a self-supervised learning method.

[0099] The total loss of the predicted optical flow includes a photometric consistency loss and a smoothness loss, and the calculation formula is:

[0100] ,

[0101] wherein, is a weight factor.

[0102] The photometric consistency loss can realize object motion by tracking the pixel intensity between images, which is calculated by a pair of grayscale images (start and end) and the predicted optical flow. The spatial transformer uses the optical flow estimates in the current horizontal and vertical directions to warp the second grayscale image to obtain the first grayscale image . The photometric consistency loss minimizes the difference between the first frame image and the second frame image. It is calculated by the following equation:

[0103] ,

[0104] wherein, is the pixel intensity of the first grayscale image, is the pixel intensity of the second grayscale image, is the Charbonnier loss function, and , r and η are constant values set during training.

[0105] The smoothness loss can enhance the spatial co-linearity of adjacent optical flows by adding regularization to the predicted optical flow to minimize the deviation between adjacent pixels, and is calculated as follows:

[0106] ,

[0107] wherein, and are the height and width of the predicted optical flow, respectively.

[0108] In addition, the Average end-point error (AEE) indicator is selected for the evaluation and determination of the predicted optical flow. The Average end-point error describes the average value of the Euclidean distance between the standard optical flow field (V) and the predicted optical flow (V). ) of all pixels. The calculation formula is as follows:

[0109] .

[0110] The first aspect of the present application introduces a cross-shaped attention module to capture complete image dependency, make up for the defects of sparse event stream, generate dense and pixel-level scene information, and reduce the space-time complexity of the algorithm; the second aspect introduces a time attention module to selectively focus on information, automatically filter frames irrelevant to the final result, effectively extract time and space features from the event stream, while preserving the event triggering characteristics of SNN, can process event stream more effectively than pure SNN structure, improve the optical flow prediction ability, and improve the calculation efficiency.

[0111] Figure 5 An event camera optical flow estimation device 500 based on a self-attention mechanism is shown according to an exemplary embodiment of the present application, comprising: an input module 510, an encoding module 520, a decoding module 530 and a loss obtaining module 540.

[0112] The input module 510 is configured to obtain an original data stream and convert the original data stream into a discretized event frame, and input the event frame into a spiking neural network.

[0113] The encoding module 520 is configured to process the event frame according to an encoding layer in the spiking neural network to obtain a spiking output corresponding to the encoding layer, wherein the encoding layer processes the event frame according to a cross-shaped attention module and a time attention module.

[0114] The decoding module 530 is configured to process the spiking output according to a residual block and a decoding layer to determine a predicted optical flow, wherein the decoding layer performs up-sampling using a transpose convolution.

[0115] The loss obtaining module 540 is configured to determine a total loss of the predicted optical flow according to the trained hybrid neural network.​

[0116] With respect to the apparatus in the above-described embodiments, in which the specific manner in which the various means perform operations has been described in detail in the embodiments relating to the method, no detailed elaboration will be given here.

[0117] Figure 6 A hardware schematic of an example event camera DVS that can be used to implement embodiments of the present disclosure is shown. An event camera is a new type of dynamic vision sensor that is sensitive to light changes, can effectively acquire motion information, and constitutes an effective complement to traditional cameras that acquire still information of a scene, and can also be used independently to complete efficient output of motion information; and has characteristics of low delay, small data volume, high dynamic range, extremely low power consumption, etc., and is widely used in computer vision fields such as target detection, image segmentation, etc. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] As Figure 6 shown, a DVS is a time-based asynchronous dynamic vision sensor that represents changes in scene light intensity in the form of asynchronous spatiotemporal pulse signals, and responds to parts with light intensity changes, but does not respond to parts without light intensity changes. The DVS converts these motion change information into a spatially sparse, temporally dense event stream. The single pixel structure and working principle of the DVS are shown in Figure 6 . The DVS pixel is composed of a fast logarithmic photoreceptor, a differencing circuit, and two comparators. The photoreceptor mainly consists of a photodiode, a transistor in series, and a negative feedback loop, and can sense light intensity changes and respond in a timely manner. The differencing circuit can amplify the output of the photoreceptor and reduce errors caused by transistor process. The threshold comparison circuit mainly realizes the output of ON / OFF event pulses by comparing the voltage changes through the comparator. If the light becomes brighter, the voltage change is greater than the threshold of the ON event, and an ON event pulse is output; if the light becomes dimmer, the voltage change is less than the threshold of the OFF event, and an OFF event pulse is output. If the voltage reaches the saturation value, the reset signal will reset the voltage, and no ON or OFF signal will be generated.

[0119] Two mainstream event cameras, ATIS (Asynchronous Time-based Image Sensor) and DAVIS (Dynamic and Active Pixel Vision Sensor), are improved based on DVS. ATIS adds a pulse width modulation link to output the light intensity information of the corresponding point while outputting the light intensity change information. Like ATIS, DAVIS can also output grayscale images at a rate of about 50 frames per second to assist image-event calibration and reduce the delay of generating grayscale images on events by using parallel sensors.

[0120] In addition, the chip based on the pulse neural network structure is called a brain-like chip. The time of pulse generation carries important information, and the pulse neural network naturally has the ability to process time sequence information, which is very consistent with the event camera based on the event stream output of the timestamp.

[0121] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which is not limited herein.

[0122] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for event camera optical flow estimation based on self-attention mechanism, characterized in that, The method comprises: acquiring an original data stream and converting the original data stream into a discretized event frame, and inputting the event frame into a spiking neural network; processing the event frame according to an encoding layer in the spiking neural network to obtain a spiking output corresponding to the encoding layer, wherein the encoding layer processes the event frame according to a cross-attention module and a temporal attention module; processing the spiking output according to a residual block and a decoding layer to determine a predicted optical flow, wherein the decoding layer performs up-sampling using transposed convolution; determining a total loss of the predicted optical flow according to a trained hybrid neural network; the processing of the event frame according to the encoding layer in the spiking neural network to obtain the spiking output corresponding to the encoding layer, wherein the encoding layer processes the event frame according to the cross-attention module and the temporal attention module, comprises: the encoding layer convolves the event frame to obtain a current membrane potential; merging the current membrane potential with a membrane potential accumulated after a previous encoding layer to determine a merged membrane potential; processing the merged membrane potential through the cross-attention module to determine an attention map; processing the attention map through the temporal attention module to determine the spiking output.

2. The method of claim 1, wherein, the acquiring of the original data stream and the converting of the original data stream into the discretized event frame, comprises: outputting the original data stream by an event camera; discretizing the original data stream into two groups of event frames in a time dimension according to a preset threshold, wherein each group of event frames contains N event frames obtained by accumulating events from a timestamp of a previous frame to a current timestamp, and each event frame is composed of two channels.

3. The method of claim 1, wherein, before the determining of the total loss of the predicted optical flow, comprising: training the hybrid neural network according to a self-supervised learning method.

4. An event camera optical flow estimation device based on self-attention mechanism, characterized in that, The method comprises: an input module, configured to acquire an original data stream and convert the original data stream into a discretized event frame, and input the event frame into a spiking neural network; an encoding module, configured to process the event frame according to an encoding layer in the spiking neural network to obtain a spiking output corresponding to the encoding layer, wherein the encoding layer processes the event frame according to a cross-attention module and a temporal attention module; a decoding module, configured to process the spiking output according to a residual block and a decoding layer to determine a predicted optical flow, wherein the decoding layer performs up-sampling using transposed convolution; a loss obtaining module, configured to determine a total loss of the predicted optical flow according to a trained hybrid neural network; the processing of the event frame according to the encoding layer in the spiking neural network to obtain the spiking output corresponding to the encoding layer, wherein the encoding layer processes the event frame according to the cross-attention module and the temporal attention module, comprises: the encoding layer convolves the event frame to obtain a current membrane potential; merging the current membrane potential with a membrane potential accumulated after a previous encoding layer to determine a merged membrane potential; processing the merged membrane potential through the cross-attention module to determine an attention map; processing the attention map through the temporal attention module to determine the spiking output.

Images