Event-based visual simultaneous localization and mapping method based on neural networks

Abstract

The invention relates to a simultaneous localization and mapping method based on a computational model utilizing neural networks, employing only a single event-based camera as the sensor. Within the method, three different neural networks are trained to estimate the changes in the camera's position and orientation, depth, and the presence of loops. All of the defined estimations are combined to construct a map and calculate the robot's position relative to it. Detected loops reset cumulative errors, enhancing the consistency and accuracy of the method.

Description

[0001] EVENT-BASED VISUAL SIMULTANEOUS LOCALIZATION AND MAPPING METHOD BASED ON NEURAL NETWORKS

[0002] TECHNICAL FIELD

[0003] The invention relates to a simultaneous localization and mapping method based on a computational model utilizing neural networks and employing only a single eventbased camera as a sensor. Within the method, three different neural networks are trained to estimate the change in the position and orientation of the camera, the depth, and the presence of loops. All defined estimations are combined to construct a map and calculate the robot's position relative to the map. The detected loops reset cumulative errors, thereby enhancing the consistency and accuracy of the method.

[0004] BACKGROUND

[0005] Simultaneous localization and mapping is currently a research area of significant commercial and academic importance. Autonomous vacuums and cars are prominent examples in this field. For mobile robots to perform autonomous tasks, they must be able to perceive their environment, create a map of the surroundings, and determine their own position relative to this map. This enables robotic platforms to execute autonomous tasks under varying environmental conditions. This problem is known in the literature as simultaneous localization and mapping. Various solutions are being developed depending on the type of sensor used by the robot, with cameras being widely utilized in this field.

[0006] RGB or depth sensor RGBD cameras used in solving the problem are affected by certain technical challenges. Some of these challenges include motion blur caused by fast-moving objects, high sensitivity to light changes due to narrow dynamic range, very low temporal resolution, and high energy consumption. Event-based camera technology, which has emerged in recent years, eliminates all of these technical issues. Therefore, within the scope of our invention, the simultaneous localization and mapping problem has been addressed using event-based cameras. This enables mobile robots to perform the same tasks under more variable conditions, with less energy consumption, in faster scenarios, and in more dynamic environments. Given the potential for the developed solutions to impact a wide range of applications, from industry to smart vacuums in our homes, this topic is considered to be of significant importance.

[0007] The first examples of event-based cameras began to emerge in 2018 and are still in the development stage today. Solutions for simultaneous localization and mapping based on such cameras have only appeared in recent years. The most notable work in the literature is titled "EVO: Event-based Visual Odometry" (WO / 2018 / 037079). This work primarily focuses on odometry and does not include the loop closure solution defined in our invention. Therefore, it is not a complete solution for simultaneous localization and mapping. Additionally, the method uses a limited number of points when constructing the map of the environment, resulting in a sparse map structure. This reduces the method's applicability for robotic tasks. The method is developed based on a filter, which means that the proposed solution is iterative. Iterative solutions may not always allow for hardware acceleration during implementation. As a result, some technical challenges can arise during the implementation of the method. Furthermore, all estimations in our invention are based on neural networks and rely solely on event-based camera data. The EVO method, on the other hand, requires an image for initialization, which also distinguishes it from our invention.

[0008] AIM OF THE INVENTION

[0009] The method described in our invention is an innovative solution based on neural networks. Neural networks can be easily implemented today and can be accelerated using appropriate hardware such as GPUs or TPUs. During the training of neural networks, the weights converge to specific values. After this phase, the operations required for a trained network involve only multiplication and summation with the determined weights. As a result, these operations can be performed in parallel with ease. This allows neural network-based methods to achieve prediction times on the order of around one hundred microseconds.

[0010] In the method described in our invention, the estimations required for localization, mapping, and loop closure are obtained using neural networks, and the solution relies solely on event data. This eliminates the need for additional sensors, providing an advantage over the methods present in the literature.

[0011] Within the method described in the invention, the presence of the loop closure component continuously reduces cumulative error, keeping the global error of the method under control. Reducing cumulative error, commonly referred to as "drift," is one of the most significant aspects that differentiates the method in the invention from the state of the art.

[0012] The mapping component in the method of the application is based on depth estimation from events. The method estimates a depth map from the set of events, resulting in a map containing more information. This allows the map to be constructed as dense or semi-dense. Therefore, it is possible to generate a more accurate and detailed map compared to the state of the art.

[0013] In the invention, the event data is not used directly but is subjected to a preprocessing step that makes the data more compact. This allows for faster results to be achieved while maintaining the same level of performance.

[0014] LIST OF FIGURES

[0015] Figure 1. Block Diagram of the Method Subject to the Invention

[0016] Figure 2. Event Data

[0017] Figure 3. Conversion of Event Data into Input Tensor

[0018] Figure 4. Event-Based Relative Pose Estimation Network Architecture

[0019] Figure 5. Event-Based Depth Estimation Network Architecture

[0020] Figure 6. Event-Based Loop Closure Feature Prediction Network Architecture

[0021] Figure 7a. Predicted Trajectory with Loop Closure Disabled

[0022] Figure 7b. Predicted Trajectory with Loop Closure Enabled Figure 8a. Predicted Trajectory with Loop Closure Disabled Figure 8b. Predicted Trajectory with Loop Closure Enabled

[0023] DETAILED DESCRIPTION OF THE INVENTION

[0024] Simultaneous localization and mapping is a problem that must be solved to develop autonomous robots. The solution developed for this problem depends on the sensor used. Therefore, the selected sensor should be capable of fast response, low energy consumption, high temporal resolution, minimal sensitivity to environmental conditions, and should produce minimal yet meaningful data. Event-based cameras, which offer a relatively new technology and approach, meet all of these requirements. As such, they hold great potential for solving the problem of simultaneous localization and mapping. Event-based cameras are an electrical model of the biological retina, and the data they produce resembles the spike signals in the nervous system. Therefore, processing this data with neural networks provides a cohesive solution and offers performance advantages. For these reasons, a simultaneous localization and mapping solution has been proposed that is based on a computational model using neural networks and employs only a single event-based camera as a sensor. In this solution, three different neural networks have been trained, validated, and tested to estimate the change in the camera’s position and orientation, depth, and the presence of loops. The method combines all estimations to construct a map and calculate the robot’s position relative to it. Detected loops reset cumulative errors, enhancing the consistency and accuracy of the method.

[0025] The method described in the invention is an event-based visual simultaneous localization and mapping solution based on neural networks. The solution essentially consists of three main components: the event-based camera pose estimation network, the event-based depth estimation network, and the event-based loop closure feature estimation network. The system architecture of the method is shown in the block diagram in Figure 1. As seen in the block diagram, the method relies solely on monocular event data.

[0026] Event-based cameras report the coordinates of pixels where brightness changes occur, along with the time and polarity of the change. Therefore, a single event data point is represented as a four-dimensional vector. To explain the structure of event data, two RGB images and the events occurring between them are shown in Figure 2. In the figure, it can be observed that, unlike traditional cameras, the events are distributed between two frames. Events with different polarities are represented by points occurring between two time steps.

[0027] The event data undergoes a pre-processing step to be compressed in time and converted into a common input tensor to be used in all networks. The pre-processing process is illustrated in Figure 3. After the event data is separated into two groups based on the polarity value, which indicates whether the brightness change is an increase or a decrease, it is transformed into the event tensor shown in Figure 3. The event tensor consists of three dimensions, specifying the pixel coordinates (H, W) and the time (t) of the change. Since event data is reported with microsecond resolution, the events are accumulated in millisecond-sized chunks and digitized. This vector can be considered the derivative of two traditional RGB images. As a result, converting events into images through an integral operation is a common approach. However, this approach does not preserve time information. Therefore, in the method described in the invention, a tensor called the time multiplier has been introduced. The events are reduced to two dimensions by applying a weighted sum with the time multiplier, preserving the time information. In this way, while the time information is retained, it eliminates the need to apply three-dimensional convolution operations.

[0028] The primary objective of the neural networks designed within the method described in the invention is to extract certain features from the events and use them for pose estimation and mapping. The event-based camera pose estimation network is trained to predict relative translation and rotational changes from the extracted features. The event-based depth estimation network is trained to use the same data to estimate logarithmic depth information. Additionally, the event-based loop closure feature estimation network is trained to estimate loop closure probabilities from the same event data.

[0029] To localize the robot, its pose must be estimated. The robot's pose consists of its position and orientation in three-dimensional Cartesian space. The event-based camera pose estimation network is the neural network designed for predicting the robot's position and orientation. It calculates the robot's positional changes and orientation shifts within the map. The event data originates from pixel changes, which are a result of the movement of the robot. This ensures that the event data contains valuable information for predicting pose changes. Once the changes are calculated, the current pose can be determined using the previous pose information. The architecture of the convolutional neural network designed for this purpose is shown in Figure 4. The network, which takes the event tensor as input, predicts features from this data through convolutional blocks. The "Inception" architecture, commonly used for feature prediction in the literature, has been adapted to work with event data and is employed in the deeper layers of the network. The predicted features pass through a fully connected localization layer, where the translational change (T) in Cartesian space and the rotational change (Q) in Quaternion space are estimated.

[0030] The event-based depth estimation network is the neural network used to estimate a depth map of the environment based on the event data generated from the surroundings and movement of the robot. It is crucial for generating dense maps. A depth map is a single-channel matrix, image-like in structure and size, that represents the logarithmic or metric depth for each pixel of the image. Therefore, U-shaped networks, commonly used for image generation in the literature, serve as good examples. The event-based depth estimation network designed as part of the method described in the invention is shown in Figure 5. In the designed network, the goal is to reduce the resolution of the image-like event data using convolutional blocks, thereby increasing the number of extracted features. After passing through the bottleneck convolutional block, the image size is increased through upsampling to produce output data of the same dimensions as the input data. In this way, the trained neural network learns to generate a depth map from the event data. Since the depth can vary from a few meters to several hundred meters, it is estimated logarithmically and later converted to metric depth.

[0031] Since relative pose estimation is performed, errors in the estimations accumulate over time, causing the predicted robot trajectory to deviate further from the actual path. This issue is known as loop closure in the simultaneous localization and mapping (SLAM) literature. To the best of our knowledge, there is currently no loop closure solution based on event data and neural networks in the existing literature. Therefore, a neural network using the same data has been designed and trained to address this problem. The designed event-based loop closure feature estimation network employs an original approach. The loop closure problem has been redefined using a triplet loss function, offering a novel solution compared to the existing literature. This network is one of the most critical components of the invention, as it helps keep global error under control. The architecture of the designed neural network is presented in Figure 6. The network has two inputs. The first input, the event tensor, is transformed into a 448-dimensional feature vector through double-step convolutional layers. The second input, which is the absolute camera pose, is transformed from a 12-dimensional vector into a 64-dimensional vector, and these two vectors are then merged. A final linear layer maps the combined vector into a 256-dimensional space, forming the output vector. Similar to the other networks, the designed network utilizes convolutional blocks, normalization, and activation layers.

[0032] The training of the designed relative camera pose estimation network was conducted in a supervised manner. The actual camera poses were converted into position and quaternion vectors and directly used in the training process. The loss function defined for training the localization network is given in Equation (1 ). The orientation-position pair predicted by the network and the actual orientation-position pair are denoted as (q, t) and (q, t), respectively. Since the error in orientation is significantly smaller than the error in position, the loss function was balanced with a parameter p to account for this discrepancy.

[0033] The training of the depth estimation network was also performed in a supervised manner. The loss function defined for this purpose is given in Equation (2). In the equation, MSSSIM refers to the multi-scale structural similarity, represents the Li norm between the predicted depth map IPand the ground truth depth map IR. The term Ls, referred to as the smoothness loss, is commonly known in the literature as depth smoothness.

[0034] L IR,Ip) = 0.85 * MSSSIM + 0.15 (2)

[0035] The goal of the designed loop closure feature estimation network is to bring the vectors produced for tensor pose pairs containing loops closer together while distancing them from other vectors. By minimizing the distances between vectors that include loops, the presence of the loop can be detected. To train such a network, the triplet loss approach is used, which is known as triplet loss function. The loss function used is given in Equation (3). In the equation, A, N, and P represent the anchor, positive, and negative predicted vectors, respectively. The defined loss function pulls the anchor and positive example closer together while pushing the negative example away from the anchor. The targeted distance between the anchor and the negative is called the margin, which is represented by a in the equation.

[0036] £(A,P,N = max \\A - P||2- - ||2+ a, 0) (3)

[0037] A general mechanism is required to transform the predictions obtained from the trained neural networks into a complete SLAM solution. In the proposed method, the event data generated by the camera is processed through the pose estimation network to calculate the robot's trajectory and create a global environmental model. This allows the relative pose changes to be estimated, and these predictions are then converted into absolute camera poses using the previous pose information. Sequentially collected event data clusters contain very similar information about the environment. Therefore, similar to traditional visual SLAM solutions, a keyframe-based approach has been adopted. Whether an event cluster is selected as a keyframe is determined based on the estimations of each event cluster. For an event cluster to be selected as a keyframe, the total distance traveled and the total rotation angle must exceed certain threshold values. For event clusters that are not selected as keyframes, only the camera pose is estimated.

[0038] The metric depth map is estimated using the depth estimation network for event clusters selected as keyframes. The estimated depth is then converted into a set of 3D points using multi-view geometry and the camera pose. At this stage, statistical noise removal and normal vector estimation are applied to the generated point cloud. Pointbased fusion methods, which have been previously used to merge point clouds, are employed to integrate the point clouds. This approach aims to globally merge sequentially predicted point clouds (global registration) and transform them into a single model. Inspired by this method, a similar approach has been developed for event-based mapping. All visible points in the global model are projected into 3D Cartesian space based on the estimated camera pose. This projection is performed using the matrices from the "pinhole" camera model and the camera's calibration matrices, as required by multi-view geometry. Using proximity relationships between the projected points and the newly predicted points, neighbors are identified, and the points are merged. The positions of the merged points are calculated as the average position of the points.

[0039] In the final stage, the global model is maintained as an index list, which includes point positions, normals, and the time steps for updating the points. The time index is updated for each merged point, keeping a record of the history of operations. To simplify the model, points that have not been updated for an extended period are removed from the map. For each event tensor, a loop closure prediction is made, and the loop closure probability is tracked. A loop closure feature vector is predicted for each tensor and compared with other vectors within a certain threshold distance. This threshold varies depending on the data. The distances between predicted vectors are minimized for looped tensors, enabling the detection of loop closures. In this dataset, loops are grouped together, and thus the predicted loop closure matches are grouped accordingly. In the frontend, absolute pose estimations are used to create a pose graph, and for each group, the SLAM problem is solved by optimizing the pose graph. The nodes of the pose graph represent the poses, while the edges represent the estimated relative pose changes. The newly calculated pose information from the optimization is then used to update and refine the global model, improving its accuracy.

[0040] When creating the pose graph, only the poses with the same loops and their frame indices are known. Since only the event data and a small portion of the global model are available, calculating the transformation between these poses is quite challenging. Therefore, when constructing the pose graph, it is assumed that these poses are the same. As a result, the identity matrix is used for the transformation matrix between these poses. While this assumption may not be entirely accurate, it has yielded fairly good results. Due to the absence of RGB images, loop closure had to be simplified using this approach.

[0041] In the backend, the pose graph optimization is performed. During the implementation of the method, the g2o (general graph optimization) library was used for optimizing the pose graph. Since the camera poses in the proposed method are represented as the special Euclidean group SE(3), the LinearSolverCSparseSE3 was chosen as the block solver.

[0042] The designed and trained networks, along with the loop closure solution, form a complete SLAM infrastructure. To process the information obtained from these components together, the final optimization step was performed using the g2o framework, thus completing the SLAM solution.

[0043] The obtained loop closure information was added as edges to the constructed pose graph in order to create constraints. It was assumed that the transformation matrix between the input tensors identified as loops and the previous tensors is the identity matrix. Using this information, the pose graph was optimized. The state of the pose graph at steps 2 and 17 is provided as examples in Figures 9a - 9b and Figures 10a - 10b, respectively. On the left side of the figures, the trajectories are plotted without optimizing the pose graph, while on the right side, the trajectories obtained after optimizing the pose graph are shown.

Claims

CLAIMS1. The method of event-based visual simultaneous localization and mapping based on neural networks, using monocular event data, is characterized by the following features:- Capturing the image (event data) using a single event-based camera,- Compressing the event data in time and subjecting it to a preprocessing step to convert it into a common input tensor, reducing the event data into two dimensions via a weighted sum with the time multiplier, referred to as the time of occurrence for use in all networks,- Applying a pose estimation network trained to estimate relative translational and rotational changes based on the extracted features,- Applying a depth estimation network to estimate logarithmic depth information,- Applying a loop closure feature estimation network trained to estimate loop closure probabilities from the same data,- Converting the estimations into a pose graph in the frontend and optimizing it in the backend,- Performing the optimization of the pose graph,- Generating a 3D map as a result of the optimization process.

2. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 1 , characterized by, in the pre-processing step, the event data is divided into two groups based on the polarity value, indicating whether the change is an increase or decrease in brightness.

3. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 2, characterized by the reduction of the event tensor into two dimensions while preserving the time information through the use of a time multiplier.

4. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to any of the preceding claims, characterized by the accumulation and digitization of events in millisecond-sized chunks due to the event data being reported with microsecond resolution. io5. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 1 , characterized by the use of the "Inception" architecture during the application of the pose estimation network, where convolutional blocks are applied to the event tensor to predict features, in order to calculate the robot's position and orientation changes within the map.

6. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 5, characterized by the estimation process in which the features pass through a fully connected localization layer, predicting the translational change (T) in Cartesian space and the rotational change (Q) in Quaternion space.

7. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 1 , characterized by, during the application of the depth estimation network, reducing the resolution of the image-like event data with convolutional blocks to increase the number of extracted features, followed by passing through a bottleneck convolutional block, after which the image size is increased through upsampling to produce output data of the same dimensions as the input data, enabling the neural network to learn to generate a depth map from the event data.

8. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 7, characterized by the estimation of depth using the depth estimation network, where the depth can range from a few meters to several hundred meters, and therefore the depth is estimated logarithmically and subsequently converted to metric depth.

9. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 1 , characterized by the use of a triplet loss function during the application of the loop closure feature estimation network.

10. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 9, characterized by the loop closure feature estimation network having two combined inputs during its application.

11. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 10, characterized by the first input, the event tensor, being transformed into a 448- dimensional feature vector using double-step convolutional layers, and the second input, the absolute camera pose, being transformed from a 12- dimensional vector into a 64-dimensional vector.

12. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to any of the claims 9 to 11 , characterized by the combined vectors being merged through a final linear layer, mapped into a 256-dimensional space, and generating the output vector.

13. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 1 , characterized by the use of the g2o (general graph optimization) library for the optimization of the pose graph.

14. The method for training the pose estimation network to be used in event-based visual simultaneous localization and mapping using monocular event data based on neural networks, characterized by the use of real camera poses directly in the training process by converting them into position and quaternion vectors, applying the loss function given in Equation 1 , and balancing the loss function with the parameter , as the orientation error is significantly smaller than the position error.

15. The method for training the depth estimation network to be used in event-based visual simultaneous localization and mapping using monocular event data based on neural networks, characterized by the application of Equation (2) with multi-scale structural similarity (MSSSIM), thenorm between the predicted depth map IPand the real depth map IR, and the depth smoothness term Ls.

16. The method for training the loop closure feature estimation network to be used in event-based visual simultaneous localization and mapping using monocular event data based on neural networks, characterized by the application of Equation (3), where the anchor (A), positive (P), and negative (N) predicted vectors are used, and the target margin distance between the anchor and negative, represented by a, is applied.

17. The method of event-based visual simultaneous localization and mapping using monocular event data based on neural networks according to claim 1 , characterized by relying solely on event data.

Images