An event camera motion segmentation method based on graph neural network

Abstract

The present application belongs to the technical field of computer vision, and particularly relates to an event camera motion segmentation method based on a graph neural network. For the motion segmentation task based on event camera data, first, an event camera-based motion segmentation dataset is prepared, and simultaneously, a motion segmentation algorithm based on a graph neural network is proposed based on the dataset to accurately segment the motion target. The present application can fully exert the sparsity and high time resolution characteristics of event data, and effectively integrate the graph structure and content information by introducing an attention mechanism to learn potential features.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a graph neural network and a method for motion segmentation using event data output by an event camera, in particular to an event camera motion segmentation method based on a graph neural network. BACKGROUND

[0002] Motion segmentation is an important topic in computer vision, which aims to segment dynamic regions from complex backgrounds and assign the same kind of moving objects to the same category. Convolutional neural network (CNN) based methods have shown excellent performance in this field, and most methods rely on traditional frame images (RGB images or grayscale images) to complete motion segmentation. However, the segmentation effect of frame image based motion segmentation algorithms will decrease dramatically in harsh conditions (e.g. low light, fast motion, etc.).

[0003] Event cameras are a novel type of bio-inspired vision sensor that asynchronously measures the intensity changes in the scene and outputs events. Therefore, it provides very high temporal resolution (up to 1MHz) and very low power consumption. Since the intensity change is calculated in a logarithmic scale, it can work in a very high dynamic range (140dB). When the logarithmic scale pixel intensity change is higher or lower than the threshold, the event camera triggers the formation of "ON" and "OFF" events. Most current methods for processing event data first accumulate and stack event data into frame images, and then use CNN for processing. However, these steps discard the sparsity and high temporal resolution characteristics of events, making it difficult to fully utilize the advantages of event data. Graph neural networks (GNNs) process events as "static" spatio-temporal graphs, which are essentially sparse and can maintain the asynchronous and sparse nature of events. Therefore, it provides a possibility for solving the event camera motion segmentation task. Next, the related background technology in this field will be introduced in detail.

[0004] (1) Event frame image based motion segmentation

[0005] Since the output of the event camera is a sparse asynchronous event stream, the storage form of the event data is very different from the traditional frame image form, so the event data is usually stacked, and CNN is used for motion segmentation on the stacked event image frame. However, this stacking form discards the sparsity of event data and quantizes event timestamps, which does not take advantage of the high temporal resolution of events, and also leads to redundant computation.

[0006] (2) Motion segmentation based on spiking neural networks

[0007] Current research on motion segmentation based on spiking neural networks (SNN) is still in its early exploratory stage. SNN is an instance of filtering models that attempt to learn task rules in a data-driven manner. Although processing event data into spike form can maintain the low latency characteristics of event data, its novel design and sensitivity to adjustments make it difficult to train and currently achieve unsatisfactory accuracy on advanced tasks. Compared with gradient-based methods, the training of SNN-based methods is more complex and difficult to operate.

[0008] (3) Motion segmentation based on graph neural networks

[0009] Methods based on graph neural networks are an ideal solution for processing asynchronous and sparse event data. Instead of processing event data into dense image tensors, event data is processed into the form of a graph, which can only consider the sparse connections between event data and limit motion message passing to these connections. This enables it to effectively process events without sacrificing their sparsity and high temporal resolution. In addition, like dynamic systems, some methods update the state immediately upon the arrival of events, achieving minimal latency for each event and significantly reducing computational complexity.

[0010] (4) Event motion segmentation dataset

[0011] Mitrokhin et al. collected two event-based motion segmentation datasets: the EED dataset and the EV-IMO dataset. However, the EED dataset only contains two motion target categories, only 179 frames (7.8 seconds) of grayscale images and corresponding event frame images, and the scene design is relatively simple, making it difficult to demonstrate the advantages of event cameras. The EV-IMO dataset increases the event data acquisition frequency to 200Hz, but only contains three motion target categories and provides only frame image-level annotations, lacking event data-level annotations. SUMMARY

[0012] To solve the above problems, the present application provides an event camera motion segmentation method based on graph neural networks. For the motion segmentation task based on event camera data, first, an event camera-based motion segmentation dataset is prepared, and at the same time, based on the dataset, a motion segmentation algorithm based on graph neural networks is proposed to accurately segment the motion target. The present application can fully exploit the sparsity and high temporal resolution characteristics of event data, and effectively integrate graph structure and content information by introducing an attention mechanism to learn latent features.

[0013] Technical scheme of the present application:

[0014] An event camera motion segmentation method based on graph neural networks, the steps are as follows:

[0015] (1) Construction of dataset

[0016] The dataset includes various moving targets and moving scenes; the dataset provides synchronous event data and frame images, as well as synchronous moving object masks, depth data and semantic information by means of the Carla simulator, and provides category labels and motion parameter information for each event point. According to the scene classification, the dataset is divided into: low-light scene, fast-moving scene with motion blur and fast-moving scene without motion blur; according to the number of moving targets and whether the camera is stationary, the dataset is divided into three scenes: single target moving and camera moving, multiple targets moving and camera stationary, and multiple targets moving and camera moving.

[0017] (2) Graph construction

[0018] The event data ε is expressed by the following formula:

[0019]

[0020] wherein (x k ,y k ) is the pixel coordinate of the event, t k is the time stamp of the event, p k = ±1 is the polarity of the event, and k represents the kth event point in the event data;

[0021] In addition, the construction of the graph node combines the four-parameter motion model parameters of each event point The obtained event graph node X is expressed as follows:

[0022]

[0023] wherein is the offset of the event point in the horizontal direction and the vertical direction, is the scaling amount of the event point, is the rotation amount of the event point;

[0024] An event graph G = {V, E} is constructed based on the event graph node, wherein represents the set of nodes, represents the set of edges in the graph structure, i.e. the adjacency matrix of the graph, e ij represents the edge of a pair of nodes (X i ,X j ). For each pair of nodes (X i ,X j ), the Euclidean distance between them is calculated, and if they are within the spatiotemporal distance R, i.e. D(X i ,X i)≤R, then the corresponding edge is generated, otherwise there is no edge between the two nodes. At the same time, a sliding window ΔT is introduced in the graph construction process, which can process data asynchronously by event point, fully consider the spatio-temporal characteristics of events, and also can mine local event information in the window. The Euclidean distance is expressed as follows:

[0025]

[0026] wherein, represents the kth feature of node X i . represents the kth feature of node X j .

[0027] (3) Motion segmentation based on graph neural network

[0028] The constructed graph structure first extracts high-order features from event data carrying motion information through an event data feature extraction module. In order to better extract the effective information of event data, the attention mechanism is introduced to focus on the adjacent nodes of the node to learn the potential features of the current node. The potential features of the current graph node are calculated by the attention coefficient and the learnable linear transformation parameter. The attention coefficient measures the importance of the neighbor nodes to the current node, considering both attribute values and topological distance. From the perspective of attribute value, the attention attribute value coefficient is calculated by the single-layer feedforward neural network of the concatenated current node and neighbor node; from the perspective of topological distance, the topological weight adjacency matrix is obtained by calculating the neighbor nodes of the t-order neighborhood in the graph. For the attention coefficient, the topological weight matrix and the activation function are added to the attention attribute value coefficient, and then the Softmax function is used to normalize between all adjacent nodes of the current node to obtain the final result. The final node potential feature can be obtained by the following formula:

[0029]

[0030] M = (B + B 2 + … + B t ) / t (5)

[0031]

[0032]

[0033]

[0034] wherein c ij is the attention attribute value coefficient, || denotes the concatenation operation, is the weight vector, W and W (l)are parameters of linear transformation of node features, aiming to enhance the expression ability, and l is the number of feature extraction module layers, M is a topological weight adjacency matrix, B is a transition matrix, and t is the order of the neighborhood of the graph. If e ij ∈E, then B ij =1 / d i , otherwise B ij =0, and d i is the degree of node i. If M ij >0, then node j is a neighbor node of node i. Alpha ij is an attention coefficient, delta is a LeakyReLU activation function, and N i represents all neighbor nodes of node i. is the output feature of the l-th layer feature extraction module of node j, and sigma is a Sigmoid function. For the initial node output feature, it can be represented as , thereby generating high-order features.

[0035] Secondly, the obtained high-order features are sent to a classifier module, and the classifier module sequentially passes through a multi-layer perceptron (MLP) and a Softmax function to obtain the motion segmentation result of the node.

[0036] The beneficial effects of the present application are:

[0037] (1) Large motion segmentation event dataset

[0038] The motion segmentation task based on deep learning relies on a large number of labeled datasets. Since the output of the event camera is an asynchronous stream, it is difficult to label event data. The present application provides synchronous event data and motion object masks for each event point by the Carla simulator, and provides motion model parameters for each event point by calculation, thereby making a large-scale motion segmentation dataset. The dataset will be beneficial to subsequent research on event camera-based motion segmentation algorithms.

[0039] (2) Event-based composition method

[0040] Due to the high temporal resolution and sparsity of event data, the present application first proposes an event data and motion model parameter combined sliding window composition method. The composition method proposed by the present application fully utilizes the spatiotemporal characteristics of event data, and provides the motion model of event data to provide more motion information for the connection between nodes. At the same time, the sliding window composition can more effectively provide event information within the window and reduce the amount of calculation.

[0041] (3) Attention mechanism-based graph node feature extraction

[0042] The graph neural network usually assigns the same weight to each neighbor node of a node, and the application introduces an attention mechanism to assign different weights to each neighbor node of a node. By focusing on the neighbor nodes of the node to learn the potential features of the current node, and combining the attribute value with the topological distance to calculate the attention coefficient, the application can effectively extract the features of the event data to solve the motion segmentation problem. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 It is a graph neural network-based event camera motion segmentation network structure diagram. DETAILED DESCRIPTION

[0044] The application will be further described in detail below in conjunction with specific embodiments, but the application is not limited to the specific embodiments.

[0045] As shown in the figure, an event camera motion segmentation method based on a graph neural network includes data set making and network model training and testing. Figure 1

[0046] (1) Training data set making

[0047] The data set contains 87 sequences, a total of 6 motion target categories, and the motion targets include vehicle, pedestrian and bicycle, etc. Motion object categories cover low light, motion blur and other complex scenes.

[0048] In order to give the event data a category label, the mask of the motion object on the frame image needs to be obtained first, and the synchronous event data and the mask are obtained through the Carla simulator, and the event data falling within the mask in the given timestamp range is given the category of the motion object. Then the interest point matching is performed on the current event point and the event point in the next timestamp range, and the four-parameter motion model of the current two event points is calculated through the two groups of paired event points of the same motion object category, and the formula is:

[0049]

[0050] Where (x, y) is the current event point coordinate, (x ′ ,y′) is the paired next timestamp event point coordinate, and t is the timestamp of the current event point.

[0051] (2) Network training

[0052] The size of the sliding window ΔT is set to 2000 in the network construction process, the sliding window step is set to 200, and the space-time distance R is set to 3. The graph structure adopts a three-order neighborhood, that is, the size of t is set to 3, and the adjacency matrix M of the t-order neighborhood neighbor nodes of the graph is represented as M=(B+B 2 +B 3 ​) / 3, the number of layers of the feature extraction module is set to 3, that is, the maximum value of l is 3. The batch size of the model is set to 4. In order to train this network, Adam is used as an optimizer to update the model parameters, and the number of iterations is set to 80 times. The learning rate is initialized to 0.001, the decay factor of the learning rate is set to 0.2, the learning rate is decayed once every 15 iterations, and the loss function uses a cross entropy loss function. In the test stage, the input event data sequence needs to be motion segmented, and after the input event data is constructed into a graph, it will pass through the feature extraction module and the classifier module in turn, and finally output the class of the event data predicted by the network, that is, the motion segmentation result.

Claims

1. A method for event camera motion segmentation based on graph neural networks, characterized in that, The steps are as follows: (1) Constructing the dataset The dataset includes various moving targets and moving scenes; the dataset provides synchronous event data and frame images, as well as synchronous moving object masks, depth data and semantic information, by means of the Carla simulator, and also provides category labels and motion parameter information for each event point; according to scene classification, the dataset is divided into: low-light scenes, fast-moving scenes with motion blur and fast-moving scenes without motion blur; according to the number of moving targets and whether the camera is stationary, the dataset is divided into three scenes: single target movement and camera movement, multiple target movement and camera stationary, and multiple target movement and camera movement; (2) Graph construction The event data ε is expressed by the following formula: where (x k ,y k ) is the pixel coordinate of the event, t k is the timestamp of the event, p k = ±1 is the polarity of the event, and k denotes the kth event point in the event data; Furthermore, the construction of the graph nodes incorporates the four parameter motion model parameters of each event point The obtained event graph node X is represented as follows: wherein, is an offset amount of the event point in the horizontal direction and the vertical direction, is a scaling amount of the event point, is a rotation amount of the event point; An event graph G = {V, E} is constructed based on the event graph nodes, where represents a set of nodes, represents a set of edges in the graph structure, i.e., the adjacency matrix of the graph, e ij represents an edge of a pair of nodes (X i , X j ); for each pair of nodes (X i , X j ), the Euclidean distance is calculated, and if they are within the spatiotemporal distance R, i.e., D(X i , X i )≤R, a corresponding edge is generated, otherwise there is no edge between the two nodes; at the same time, a sliding window ΔT is introduced in the graph construction process, and the data is processed asynchronously by event point; the Euclidean distance is represented as follows: wherein, represents the kth feature of node X i ; represents the kth feature of node X j ; (3) Motion segmentation based on graph neural network The constructed graph structure first extracts high-order features from the event data carrying motion information through the event data feature extraction module; in order to better extract the effective information of the event data, the attention mechanism is introduced to focus on the adjacent nodes of the node to learn the latent features of the current node; the latent features of the current graph node are calculated by the attention coefficient and the learnable linear transformation parameter; the attention coefficient measures the importance of the neighbor nodes to the current node, considering both attribute values and topological distances; from the perspective of attribute values, the attention attribute value coefficient is calculated by the concatenated current node and neighbor nodes through a single-layer feedforward neural network; from the perspective of topological distance, the topological weight adjacency matrix is obtained by calculating the t-order neighborhood of the graph; for the attention coefficient, the topological weight matrix and the activation function are added to the attention attribute value coefficient, and then the Softmax function is used to normalize between all adjacent nodes of the current node to obtain the final result; the latent features of the final node are obtained by the following formula: M = (B + B 2 +…+B t ) / t (5) where c ij is the attention property value coefficient, || denotes the cascade operation, is the weight vector, W and W (l) is the parameter of the linear transformation of the node feature, l is the number of feature extraction module layers; M is the topology weight adjacency matrix, B is the transition matrix, t is the neighborhood order of the graph; if e ij ∈E, then B ij =1 / d i , otherwise B ij =0, d i is the degree of node i; if M ij >0, then node j is the neighbor node of node i; α ij is the attention coefficient, δ is the LeakyReLU activation function, N i represents the neighbor node of node i; is the output feature of the l-th layer feature extraction module of node j, σ is the Sigmoid function; for the initial node output feature, it is represented as from which high-order features are generated; Secondly, the obtained high-order features are sent to the classifier module, and the motion segmentation result of the node is obtained by sequentially passing through the multi-layer perceptron MLP and the Softmax function in the classifier module.

Images