Method for generating 3D reference points in a scene map

Abstract

A method for supplementing a scene map using 3D reference points, comprising four steps. In the first step, data is collected and recorded based on samples from at least one of an optical sensor, GNSS, and IMU. The second step includes: generating an initial pose by processing the collected sensor data to provide a trajectory of the vehicle pose. The pose is based on a specific dataset, at least one dataset recorded before the dataset, and at least one dataset recorded after the dataset. The third step includes: performing SLAM processing on the initial pose and the collected optical sensor data to generate key frames with feature points. In the fourth step, 3D reference points are generated by fusion and optimization of feature points that use future feature points and past feature points together with the feature points at the processing point. The second step and the fourth step provide significantly better results than known SLAM or VIO methods from the prior art because the second step and the fourth step are based on the recorded data. Wherein ordinary SLAM or VIO algorithms can only access past data, and in these steps, it is also possible to process by looking at the previous positions and by using the recorded data.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a method for generating point landmarks / 3D reference points / map points for a 3D map of a scene (e.g., a data road map that can be used by an autonomous driving vehicle). Background Art

[0002] Improved driver systems and autonomous vehicles require highly accurate maps of roads and other areas on which the vehicle can travel. Autonomous vehicles need to determine the position of the vehicle on the road with high accuracy, which cannot be achieved by conventional navigation systems such as GNSS (Global Navigation Satellite System), e.g., GPS, Galileo, GLONASS, or other known positioning techniques such as triangulation, etc. However, when an autonomous vehicle is traveling on a multi-lane road, it is necessary to accurately determine the position of the vehicle on one of the lanes.

[0003] Regarding high-precision navigation, it is necessary to access a digital map in which objects related to the safe driving of an autonomous vehicle are captured. Tests and simulations of autonomous vehicles have shown that a very detailed understanding of the vehicle's environment and road specifications is required.

[0004] However, conventional digital maps of the road environment currently used in combination with GNSS tracking of vehicle movement may be sufficient to support the navigation of driver-controlled vehicles, but they are not detailed enough for autonomous vehicles. Scanning roads using dedicated scanning vehicles provides more details, but is extremely complex, time-consuming, and expensive.

[0005] In robot mapping and navigation, Simultaneous Localization and Mapping (SLAM) is well-known for generating an accurate map of a scene. According to the SLAM process, the position of an agent within an unknown environment is tracked while constructing or updating a map of the environment. Another approach is sensor fusion, which uses information from multiple sensor systems (e.g., a monocular camera, a satellite navigation system such as GPS, an Inertial Measurement Unit (IMU), etc.) to provide a mobile robot trajectory and 3D reference points for generating a map of the scene. However, the known methods for generating vehicle trajectories and 3D reference points have some limitations.

[0006] Especially in large-scale scenarios, due to cumulative errors and lack of texture (grain) for image matching, SLAM usually performs poorly, especially over long periods. Traditional SLAM systems use data in an online / real-time manner and thus lose some of the intrinsic value of background data from different sensors (such as the image sensor of a camera, satellite navigation sensor, sensors of an inertial measurement unit (IMU), etc.). Therefore, traditional SLAM-based systems may be prone to losing track for various reasons (such as images with blurred features) and thus generate very short and unusable data.

[0007] In addition, SLAM systems are initialized from the very beginning of the data (image data, satellite navigation data, IMU data, etc.), which may take a long time to provide usable data, especially when the initial data accuracy is poor. Basically, known SLAM systems cannot perform fast and accurate initialization.

[0008] The problem with the sensor fusion system mentioned above is that the sensor fusion of data from different sensor systems (such as monocular cameras, satellite navigation sensors, inertial measurement unit sensors, etc.) depends to a large extent on the accuracy of GPS data. However, when the GPS data is inaccurate due to signal loss, obstruction, etc., it may generate unreliable vehicle trajectories and 3D reference points. Summary of the Invention

[0009] The problem to be solved by the present invention is to provide a method for generating point landmarks / map points / 3D reference points as part of or in a scene map with high precision and a low probability of tracking loss.

[0010] The solution to the problem is described in the independent claims. The dependent claims relate to further improvements of the present invention.

[0011] An embodiment of a method for generating point landmarks / map points / 3D reference points as part of or in a scene map is detailed in claim 1, which allows generating a map with high precision and almost eliminating tracking loss.

[0012] One embodiment includes four steps. The first step includes: collecting data for background or deferred processing. The vehicle collects and records samples of at least one of optical sensor or camera data, such as images from the environment, position data from GNSS (Global Navigation Satellite System) which may be GPS, and data from IMU (Inertial Measurement Unit). Further sensor data collected and recorded may include wheel tick and / or steering angle and / or any other sensor data or environmental condition data indicating the motion characteristics of the vehicle, such as environmental temperature and / or windshield wiper status. These data are collected over a certain period of time, distance, or number of samples. For example, 100 to 300 or even up to several thousand samples can be collected at intervals of 1 meter. The optical sensor mentioned herein may be or may include one or more monocular cameras. It may also include other optical sensors, such as stereo cameras and / or lidar and / or radar.

[0013] The second step includes: generating a trajectory of an initial vehicle pose by processing the collected sensor data. The vehicle pose may also be referred to as pose herein. Based on the quality of the sampled data, at least one of the image, GNSS, and IMU data can be selected from each sample, or at least two of the image, GNSS, and IMU can be fused to provide a trajectory of the initial vehicle pose. In addition, filters can be applied to provide a smooth trajectory and / or an accurate initial vehicle pose. If the second step is performed after the first step and applied to the data collected in step 1, the second step is most efficient. This can allow for better filtering of longer sample sequences, rather than just based on past recorded samples. Instead, for each sample, additional samples sampled earlier and later can be used for filtering. This can only be done when filtering the recorded data from the first step. In one embodiment, the operation of the second step can be performed with a certain delay amount after collecting the sample of each data in the first step.

[0014] The third step includes: constructing a sequence of key frames, where the key frames include feature points generated by a SLAM algorithm or other VIO algorithms. In addition, a sequence of improved vehicle poses can be generated. The feature points may be dot-like features. Such features can be detected by image processing using an appropriate feature detector (such as a corner or edge detector or other detector). Descriptors that describe the optical characteristics of the dot-like features relative to their environment can be used to describe the feature points so that they can be detected again from different distances or angles from the optical sensor.

[0015] This third step can be carried out after the second step. The third step can also be carried out with a certain delay amount after the second step so that sufficient data is available.

[0016] This third step provides significantly better results than known SLAM or VIO (Visual Odometry) methods from the prior art, which process the latest data in real time, because it is based on the recorded and filtered initial pose data of the second step. Additionally, the fourth step is also based on the recorded data as it uses future key frames.

[0017] The fourth step includes generating 3D reference points. Such 3D reference points are sometimes simply referred to as 3D points or map points or point landmarks. A 3D reference point can be a point in 3D space. It can be derived from at least one feature point in at least one 2D image. In this case, it is the same point, but in 3D space, while the feature point is its view in the optical sensor image. A 3D reference point can also have a descriptor. The 3D reference points are based on key frames and the feature points contained therein, and can also be based on their associated refined poses. Each processed point is based on the selected key frame and its associated refined pose, and further uses at least one key frame and its associated refined pose before the processed point and at least one key frame and its associated refined pose after the processed point.

[0018] 3D reference points can be generated through the fusion and optimization of feature points and by optionally using refined poses. One way to perform such refinement or optimization can be bundle adjustment.

[0019] In summary, the method can be basically divided into two main parts. In the first part (initial pose generation), smooth trajectories and initial / preliminary but already accurate vehicle pose trajectories are generated in step 2 using the data sampled from various sensor systems in step 1. In the second main part (3D reference point construction), the generated initial pose is used in step 3 by a SLAM system / program or a visual odometry system / program to provide key frames including feature points and refined vehicle poses. Later, in step 4, these key frames are used to construct point landmarks / map points / 3D reference points, while also continuously optimizing the poses and 3D reference points using appropriate data optimization algorithms (such as the bundle adjustment algorithm).

[0020] According to one embodiment, the initial vehicle pose is generated by using data from non-optical sensor systems (such as data from inertial measurement unit sensors and satellite navigation sensors).

[0021] In one embodiment, only data from non-optical sensors is used.

[0022] According to another embodiment, the initial pose of the optical sensor is generated by using data provided by an inertial measurement unit and a satellite navigation system and also by using image data of at least one image sequence of the optical sensor recorded in step 1.

[0023] According to another embodiment, in a second step, a smooth trajectory of an initial vehicle pose can be generated by using and fusing data from various sensor systems based on an Extended Kalman Filter (EKF) framework. Due to the nature of the EKF itself, the accuracy of multi-sensor fusion (e.g., the fusion of image data from an optical sensor, satellite navigation data from a satellite navigation system, and data from an Inertial Measurement Unit) is no longer highly dependent on a specific one of these sensors, resulting in the system being more robust to noise. Specifically, the accuracy of multi-sensor fusion is no longer largely dependent on the accuracy of GPS data. The pose can initially be generated by cinematic modeling and then optimized through several visual and physical observations.

[0024] Due to the use of the Extended Kalman Filter, a confidence level can be provided for the pose of each frame before running feature point construction in a third step. The confidence level can be used for SLAM initialization, making the initialization of the system faster and more likely to succeed. Additionally, considering the confidence level of the generated initial pose can avoid unnecessary data processing. Specifically, if each generated initial pose has a confidence level, it can be appropriately decided when and where to initialize the feature point construction process in the third step of the method.

[0025] After initialization, each initial pose obtained from the second step (pose generation) can be used as the initial / preliminary pose for subsequent improvement of the vehicle pose and key frame construction in the third step. The key frames include feature points and this step is performed by a SLAM system / program or a visual odometry system / program. Then, in a fourth step, both the improved pose and the key frames with feature points are used to construct point landmarks / map points / 3D reference points and generate an optimized vehicle pose. This can be done by applying appropriate optimization algorithms, e.g., local or global bundle adjustment algorithms. In this fourth step, the data recorded in the first step can also be used, such as checking the reprojection of the generated 3D reference points onto the original 2D optical sensor images in order to check and minimize the reprojection error. The optimized point landmarks / map points / 3D reference points can be used to generate or supplement a scene map. This can be done in or after the fourth step.

[0026] In one embodiment, during bundle adjustment, constraints on the reprojection error and on the rotation between the position and pose of the optical sensor can be imposed. Thus, the method can eliminate scale drift and orientation deviation to ensure the accuracy of scale and orientation.

[0027] The method of generating 3D reference points in a scene map and using them for vehicle navigation can be applied in, for example, the fields of autonomous vehicle navigation, autonomous underwater vehicle navigation, or autonomous unmanned aerial vehicle (UAV) navigation.

[0028] In one embodiment, a computer or a mobile robot in a vehicle may perform any of the steps of the methods disclosed herein.

[0029] Additional features and advantages are set forth in the detailed description which follows. It is to be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide an overview or framework for understanding the nature and character of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Hereinafter, the present invention will be described by way of example based on embodiments with reference to the accompanying drawings, without limiting the general concept of the present invention.

[0031] Figure 1 A flowchart showing method steps for a method of generating point landmarks / map points / 3D reference points in a scene map is shown.

[0032] Figure 2 A dataset that can be accessed by a SLAM algorithm is shown.

[0033] Figure 3 The computational inaccuracy of standard SLAM is shown.

[0034] Figure 4 The reduced computational inaccuracy of modified SLAM is shown.

[0035] Figure 5 The optimized vehicle pose for method step 4 is shown. DETAILED DESCRIPTION

[0036] Figure 1 An embodiment is shown. As shown in the drawings, a method for generating landmarks / 3D reference points in a scene map may include four steps S1, S2, S3, and S4.

[0037] In a first step S1, a dataset is recorded, where each dataset includes sampling data of at least one optical sensor and at least one of GNSS (Global Navigation Satellite System) and IMU (Inertial Measurement Unit). For example, 100 to 300 or thousands of samples may be collected at intervals of 1 meter or shorter or longer. Each dataset may include one sample.

[0038] In a second step S2, the collected data from the first step is processed to provide a trajectory of an initial vehicle pose. The initial vehicle pose associated with the dataset is based on the dataset, the datasets recorded before the dataset, and the datasets recorded after the dataset. In other words, if the initial vehicle pose at the processing point is based on a specific dataset, the initial vehicle pose is further based on at least one dataset recorded before the processing point and at least one dataset recorded after the processing point.

[0039] In a real-time system, which means a system where there is no record of step 1 according to the prior art, only the data sets generated before this data set will be available. According to an embodiment, data that allows access to future data can be used. By using past and future data, a smoother, continuous, plausible, and more accurate trajectory of the initial vehicle pose can be generated. This pre-generated trajectory of the initial vehicle pose allows for correct initialization and smooth processing of the SLAM algorithm in the next step without losing the trajectory and with high precision.

[0040] In the third step S3, a sequence of improved vehicle poses and key frames is generated by the SLAM algorithm. The key frames include feature points. Another VIO algorithm can be used instead of the SLAM algorithm.

[0041] In the fourth step S4, 3D reference points are constructed based on the feature points and the improved vehicle poses from the third step. The 3D reference points and the optimized vehicle poses are based on the key frames and can also be based on their related improved vehicle poses related to the processing points, at least one key frame and its related improved vehicle poses before the processing point, and at least one key frame and its improved vehicle poses after the processing point. This method also uses key frames before the processing point, thus allowing for a more precise determination of the 3D reference points / landmarks.

[0042] In an embodiment of step S2, the initial vehicle pose is generated by specifically using the data of a non-optical sensor system. For example, the initial vehicle pose can be generated by using the data of an inertial measurement unit (IMU) and a satellite navigation system (such as GPS).

[0043] In step S3, feature extraction, matching, and pose improvement using descriptors can be performed based on the captured images of the optical sensor. Feature tracking can be performed by applying the simultaneous localization and mapping (SLAM) algorithm.

[0044] In step S4, triangulation can be performed by evaluating the optimized poses and results generated from feature extraction and matching to generate 3D points or features in 3D space.

[0045] According to an embodiment, in the fourth step S4, at least one of the generated 3D reference points and the generated improved vehicle poses can be optimized by applying an optimization algorithm (e.g., global bundle adjustment algorithm).

[0046] Compared with traditional SLAM systems that use data in an online / realtime manner, according to a first embodiment of the method for generating 3D reference points in a scene map, the SLAM system is now a background system that uses data from various sensor systems previously stored in a storage system to generate an initial vehicle pose. Thus, the SLAM system evaluates data from various sensor systems in a background manner, avoiding loss of tracking even in scenarios that are very difficult for realtime SLAM systems, such as scenarios with only few features that produce feature points that can be tracked over many optical image frames. Consequently, the generated vehicle pose trajectory, the generated sequence of key frames, and the generated 3D reference point map are much longer and much more accurate than those generated using traditional realtime SLAM systems without a pre-generated initial vehicle pose trajectory.

[0047] This method is well-suited for large-scale scenarios for generating large road databases. Due to the low computational complexity, the SLAM system can generate vehicle poses and 3D reference points precisely and efficiently. The SLAM system runs very fast because the initial vehicle pose provided during pose generation is computationally more efficient. Additionally, the proposed method for generating 3D reference points in a scene map is robust to unreliable sensor data.

[0048] In one embodiment, in step S2, the initial pose of the optical sensor is generated by using data from an inertial measurement unit (IMU) and a satellite navigation system (GNSS) (e.g., a GPS system) and additionally by using image data of an image sequence captured by an optical sensor. According to one embodiment, in step S2, the initial pose of the optical sensor can be generated by filtering data from the inertial measurement unit and the satellite navigation system and image data of the image sequence of the optical sensor by using an extended Kalman filter (EKF).

[0049] In one embodiment, features can be extracted from the image sequence, and in a second step S2, feature tracking is performed by evaluating the optical flow. The initial pose of the optical sensor can be updated based on the optical flow tracking result in step S2. However, at this stage, the tracking result may not be used for triangulation. The feature tracking result can be used as measurement information for updating the initial vehicle pose. In summary, the feature tracking result can be understood as an epipolar constraint between images.

[0050] In an alternative embodiment, features can be extracted from the image sequence and in a second step S2, feature tracking is performed by matching feature point descriptors, triangulating the feature points, and generating an initial vehicle pose based on the matching and triangulation results.

[0051] In one embodiment, in step S4, the generated 3D reference points and the improved poses of the generated optical sensors can be optimized by applying an optimization algorithm, such as a local bundle adjustment algorithm.

[0052] Steps S1, S2, S3, and step S4 of optimizing the generated 3D reference points and the generated vehicle poses, and the step of generating or supplementing a scene map based on the optimized generated 3D reference points and all other processing steps can be executed by a processor of a computer. Such a computer can advantageously be in the vehicle, but may also be in a backend that collects different sensor data, such as sensor data from a satellite navigation system, sensor data from an inertial measurement unit (IMU), and image data from an optical sensor, and then processes the sensor data. The method for generating point landmarks / map points / 3D reference points in a scene map can be implemented as a computer program product embodied on a computer-readable medium. The computer program product includes instructions for causing a computer to execute the method steps S1 and S2 of the method for generating landmarks / 3D reference points in a scene map.

[0053] Figure 2 Shows a data set that can be accessed by a SLAM (Simultaneous Localization and Mapping) algorithm. Known SLAM algorithms from the prior art are used for real-time localization. Localization is combined with mapping because an environmental map is needed for precise localization within it. A normal SLAM algorithm obtains a data sample, which can successively contain image, GNSS, and IMU information. Depending on the speed and processing power, the sampling rate can be between 1 / s and 60 / s. When the SLAM algorithm receives a new sample, new calculations can be started based on the new sample and past samples. This can only generate an orbit / trajectory and vehicle poses ending with the new sample. There can be extrapolation from previous samples to the new sample and combination with the values of the new sample. This is indicated by arrow 110 showing the use of samples N, N - 1, N - 2, N - 3, N - 4, and other samples in sample 100.

[0054] The method of step 2 can also access "future" samples before the new sample because the sensor data has already been recorded. This can better smooth the generated trajectory. In this embodiment, real-time localization is not required, so the recorded data can be used only for background map construction. This will be shown in more detail in the following figure. Additionally, this method step allows access to "future" key frames. Here, the same graph can be applied by referring to key frames instead of samples.

[0055] Figure 3Shows the trajectory of the vehicle pose known in the prior art. Points N-1 to N-4 indicate the past vehicle poses / positions of the trajectory 200. SLAM can identify the new vehicle pose N, while the true pose may be XN.

[0056] If feature points cannot be identified or new feature points do not match the previous ones, the SLAM system known in the prior art cannot work properly. Sometimes, non-optical systems such as IMU or GPS can help improve accuracy and / or recover lost trajectories. However, there are still some cases where IMU and GPS cannot help and SLAM cannot initialize or recover lost trajectories.

[0057] Figure 4 Shows the initial vehicle pose of method step 2. Here, the trajectory of the initial vehicle pose is generated based on the data set collected in the first step. In order to generate the initial vehicle pose P at the processing point (which may be related to the data set N), not only the past data sets N-1 to N-4 are used, but also the future data sets N+1 to N+4 are used. This makes the trajectory of the initial vehicle pose 210 relatively smooth and accurate.

[0058] Such a trajectory of the initial vehicle pose provides better initial conditions for SLAM and avoids trajectory loss because it has provided a basic reasonable trajectory.

[0059] Figure 5 Shows the optimized vehicle pose of method step 4. Here, the trajectory of the optimized vehicle pose is generated based on the improved vehicle pose of the third step and the key frames including feature points. The 3D reference points are generated and optimized together with the optimized vehicle pose. However, for a simpler picture and better understanding, the 3D reference points are omitted in this figure. Used to generate the optimized vehicle pose P at the processing point O , which may be related to the improved vehicle pose and key frame P, not only uses the past improved vehicle poses and key frames P-1 to P-4, but also uses the future improved vehicle poses and key frames P+1 to P+4. This results in a relatively smooth and accurate trajectory of the improved vehicle pose 220 and the 3D reference points.

Claims

1. A method for generating 3D reference points, comprising the following steps: Receive a data set including data sampled by at least one sensor, where the at least one sensor includes at least one of an optical sensor or a non-optical sensor; Process the data set and determine an initial vehicle pose trajectory, where each of the initial vehicle poses is associated with a processing point, and each item of the initial vehicle pose is generated based on one of the data sets recorded at the processing point, at least one of the data sets recorded before the processing point, and at least one of the data sets recorded after the processing point; Generate a sequence of improved vehicle poses and key frames based on the application of a SLAM algorithm to a part of the data set and the initial vehicle poses, where the key frames include feature points; and Generate 3D reference points and optimized vehicle poses based on the feature points and the improved vehicle poses, where each of the 3D reference points and the optimized vehicle poses is generated based on the key frame associated with a corresponding one of the processing points and one of the improved vehicle poses, the key frames before the corresponding one of the processing points and at least one of the improved vehicle poses, and the key frames after the corresponding one of the processing points and at least one of the improved vehicle poses.

2. The method according to claim 1, wherein: The non-optical sensor includes an inertial measurement unit (IMU), a global satellite navigation system (GNSS), a wheel scale generator, or a steering angle sensor; and The determination includes generating the initial vehicle pose based on a first subset of the data set including data sampled by the non-optical sensor.

3. The method according to claim 2, wherein The determination further includes: Generating the initial vehicle pose based on image data sampled by the optical sensor, where the optical sensor includes a monocular camera, a stereo camera, a lidar unit, or a radar unit.

4. The method according to claim 1, wherein The determination includes: Generating the initial vehicle pose based on applying an extended Kalman filter to data sampled by the non-optical sensor and image data sampled by the optical sensor.

5. The method according to claim 1, wherein: The data set includes image data sampled by the optical sensor, and the image data includes an image sequence; The method further includes: extracting features from the image sequence and performing feature tracking based on an evaluation of optical flow; and The determination includes generating the initial vehicle pose based on the evaluation of the optical flow.

6. The method according to claim 1, wherein The data set includes image data sampled by the optical sensor, and the image data includes an image sequence; The method further includes: extracting feature points from the image sequence and performing feature point tracking by matching feature point descriptors, triangulating the feature points, and generating the initial vehicle pose based on the matching and triangulation results.

7. The method according to claim 1, wherein The data set includes image data sampled by the optical sensor, and the image data includes an image sequence; The method further includes: performing feature point extraction, key frame generation, and vehicle pose improvement using descriptors based on the image data and further data received in addition to the data set and based on the initial vehicle pose.

8. The method according to claim 7, wherein: The method further includes performing feature point tracking based on matching of the feature point descriptors and triangulation of the feature points; and Generating the 3D reference point includes generating the 3D reference point based on an evaluation of the refined vehicle pose and the results from the feature point tracking.

9. The method according to claim 7, wherein: The method further includes performing feature point tracking based on matching of the feature point descriptors and triangulation of the feature points; and Generating the 3D reference point includes generating the 3D reference point based on an evaluation of the refined vehicle pose, the results extracted from the feature point tracking, and a portion of the data set.

10. The method according to claim 1, wherein Generating the 3D reference point and the optimized vehicle pose includes: Generating the 3D reference point and the optimized vehicle pose by applying an optimization process to a portion of the key frames and the refined vehicle pose, the optimization process including a global bundle adjustment process or a local bundle adjustment process.

11. The method according to claim 10, further comprising: Performing the following: generating a portion of the scene map based on the generated 3D reference point.

12. The method according to claim 1, wherein Generating the 3D reference point and the optimized vehicle pose includes: Performing the following: checking a reprojection of the generated 3D reference point onto the data sampled by the optical sensor, and minimizing a reprojection error based on the checked reprojection of the generated 3D reference point.

13. The method according to claim 1, further comprising: Generating a portion of the scene map based on the 3D reference point and the optimized vehicle pose; And Performing vehicle navigation operations based on at least a portion of the scene map.

14. The method according to claim 1, further comprising: Sending at least a subset of the 3D reference point and the optimized vehicle pose to a computing system, the computing system performing the following: generating a portion of the scene map based on the subset of the 3D reference point and the optimized vehicle pose.

15. The method according to claim 1, wherein The 3D reference point is used to implement vehicle navigation operations.

16. A non - transitory machine - readable storage medium storing instructions which, when executed by at least one processor of a server, cause the at least one processor to perform an operation comprising the following steps: Receiving a data set comprising data sampled by at least one sensor, the at least one sensor including at least one of an optical sensor or a non - optical sensor; Processing the data set and determining an initial vehicle pose trajectory, wherein Each of the initial vehicle poses is associated with a processing point, and each item of the initial vehicle poses is generated based on one of the data sets recorded at the processing point, at least one of the data sets recorded before the processing point, and at least one of the data sets recorded after the processing point; Based on the application of a SLAM algorithm to a portion of the data set and the initial vehicle poses, generating a sequence of refined vehicle poses and key frames, the key frames including feature points; and Based on the feature points and the refined vehicle poses, generating a 3D reference point and an optimized vehicle pose, wherein each of the 3D reference point and the optimized vehicle pose is based on the key frame and one of the refined vehicle poses corresponding to a respective one of the processing points, the key frames and at least one of the refined vehicle poses before the respective one of the processing points, and the key frames and at least one of the refined vehicle poses after the respective one of the processing points.

17. An apparatus for generating 3D reference points, comprising: Communication interface; A non-transitory machine-readable storage medium that stores instructions; and at least one processor coupled to the communication interface, and the non-transitory machine-readable storage medium, the at least one processor being configured to execute instructions to perform the following operations: Receive a data set including data sampled by at least one sensor, the at least one sensor including at least one of an optical sensor or a non-optical sensor; Process the data set and determine an initial vehicle pose trajectory, wherein each of the initial vehicle poses is associated with a processing point, and each entry of the initial vehicle pose is generated based on one of the data sets recorded at the processing point, at least one of the data sets recorded before the processing point, and at least one of the data sets recorded after the processing point; Generate a sequence of improved vehicle poses and key frames based on the application of a SLAM algorithm to a portion of the data set and the initial vehicle poses, the key frames including feature points; and Generate 3D reference points and optimized vehicle poses based on the feature points and the improved vehicle poses, wherein each of the 3D reference points and the optimized vehicle poses is generated based on the key frame associated with the corresponding one of the processing points and one of the improved vehicle poses, the key frames before the corresponding one of the processing points and at least one of the improved vehicle poses, and the key frames after the corresponding one of the processing points and at least one of the improved vehicle poses.

18. The apparatus according to claim 17, wherein: The non-optical sensors include an inertial measurement unit (IMU), a global satellite navigation system (GNSS), a wheel scale generator, or a steering angle sensor; and The at least one processor is further configured to execute instructions to perform the following operation: generate the initial vehicle pose based on a first subset of the data set including data sampled by the non-optical sensors.

19. The apparatus according to claim 18, wherein The at least one processor is further configured to execute instructions to perform the following operations: Generate the initial vehicle pose based on image data sampled by the optical sensor, the optical sensor including a monocular camera, a stereo camera, a lidar unit, or a radar unit.

20. The apparatus according to claim 17, wherein The at least one processor is further configured to execute instructions to perform the following operations: Generate the initial vehicle pose based on applying an extended Kalman filter to data sampled by the non-optical sensors and image data sampled by the optical sensors.

21. The apparatus according to claim 17, wherein: The data set includes image data sampled by the optical sensor, the image data including an image sequence; and The at least one processor is further configured to execute instructions for the following operations: Extract features from the image sequence and perform feature tracking based on an evaluation of the optical flow; and Generate the initial vehicle pose based on the evaluation of the optical flow.

22. The apparatus according to claim 17, wherein: The data set includes image data sampled by the optical sensor, the image data including an image sequence; and The at least one processor is further configured to execute instructions to perform the following operations: Extract feature points from the image sequence and perform feature point tracking by matching feature point descriptors, triangulating the feature points, and generating the initial vehicle pose based on the matching and triangulation results.

23. The apparatus according to claim 17, wherein: The data set includes image data sampled by the optical sensor, the image data including an image sequence; and The at least one processor is further configured to execute instructions for performing the following operations: Perform feature point extraction using descriptors, key frame generation, and vehicle pose improvement based on the image data and further data received in addition to the data set and based on the initial vehicle pose.

24. The apparatus according to claim 23, wherein, The at least one processor is further configured to execute instructions for performing the following operations: Perform feature point tracking based on the matching of the feature point descriptors and the triangulation of the feature points; and Generating the 3D reference point includes generating the 3D reference point based on an evaluation of the refined vehicle pose and the results from the feature point tracking.

25. The apparatus according to claim 23, wherein, The at least one processor is further configured to execute instructions for performing the following operations: Perform feature point tracking based on the matching of the feature point descriptors and the triangulation of the feature points; and Generate the 3D reference point based on an evaluation of the refined vehicle pose, the results extracted from the feature point tracking, and a portion of the data set.

26. The apparatus according to claim 17, wherein, The at least one processor is further configured to execute instructions for performing the following operations: Generate the 3D reference point and the optimized vehicle pose by applying an optimization process to a portion of the key frame and the refined vehicle pose, the optimization process including a global bundle adjustment process or a local bundle adjustment process.

27. The apparatus according to claim 26, wherein, The at least one processor is further configured to execute instructions for performing the following operations: Perform the following operation: supplement the scene map based on the generated 3D reference point.

28. The apparatus according to claim 17, wherein, The at least one processor is further configured to execute instructions for performing the following operations: Perform the following operations: check the reprojection of the generated 3D reference point to the data sampled by the optical sensor, and minimize the reprojection error based on the checked reprojection of the generated 3D reference point.

29. The apparatus according to claim 17, wherein, The at least one processor is further configured to execute instructions for performing the following operations: Generate a portion of the scene map based on the 3D reference point and the optimized vehicle pose; and Perform vehicle navigation operations based on at least a portion of the scene map.

30. The apparatus according to claim 17, wherein, The at least one processor is further configured to execute instructions for performing the following operations: send at least a subset of the 3D reference point and the optimized vehicle pose to a computing system, the computing system performing the following operations: generate a portion of the scene map based on the subset of the 3D reference point and the optimized vehicle pose.

31. The apparatus according to claim 17, wherein, The 3D reference point is used to implement vehicle navigation operations.

Images