Track high integrity path extraction with autonomous sensors

Abstract

A method for path extraction of a vehicle on a guide rail, the vehicle having two or more sensors. Two or more sensor inputs are received from two or more sensors, including at least one active 3D sensor input from at least one active 3D sensor and at least one passive 2D sensor input from at least one passive 2D sensor. At least one active 3D sensor path is extracted based on the at least one active 3D sensor input and at least one passive 2D sensor path is extracted based on the at least one passive 2D sensor input. At least one 3D sensor ground surface model is generated based on the at least one passive 2D sensor path. At least one passive 3D path is generated based on the at least one passive 2D sensor path and the at least one 3D sensor ground surface model. The at least one passive 3D path and the at least one active 3D sensor path are fused to produce a merged 3D path. At least one supervision check is performed in the path extraction pipeline to provide integrity to the merged 3D path.

Description

[0001] Preference Statement

[0002] This application claims priority to U.S. Provisional Application No. 63 / 368,910, filed on July 20, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] Path / track extraction in the rail industry is performed via (a) offline survey-grade data collection or (b) online onboard data processing.

[0004] Offline survey-grade data processing uses surveying equipment such as total stations (SBB software), EM-SAT, high-density LiDAR surveys (with the help of ground control points (GCPs)), global navigation satellite systems (GNSS) and / or inertial measurement units (IMUs), and GNSS / IMU surveys. However, surveying techniques are costly because they rely on special equipment, manual operations, and extensive post-processing. If the position of the train is accurately known, offline construction of paths using surveying methods can be used, but this is not applicable in the case of safely recovering a train that has lost its position. In addition, when the train position is known but with a given position uncertainty, the accuracy of the path may vary significantly in the turnout area. Therefore, an online extraction of the corresponding track path of the train of interest (also called ego path), i.e., the corresponding track of the autonomous train of interest, is used. This is performed using a set of autonomous sensors on the train, which are much cheaper than surveying equipment, but have lower density and lower accuracy.

[0005] Other path extraction techniques based on online onboard data processing rely on data that may not be available in actual operations (e.g., back reflections from the rail head / top of the track), cannot utilize highly constrained track information, and / or do not adequately combine multiple sensors to manage performance and integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The embodiments can be best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be noted that, in accordance with standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of various features may be arbitrarily increased or reduced for clarity of discussion.

[0007] Figure 1 is a block diagram of an overall path extraction architecture for CMPE according to at least one embodiment;

[0008] Figure 2 is a block diagram showing further details of active and passive sensors and a surface model generator according to at least one embodiment;

[0009] Figure 3 is a plan view of a vehicle on track according to at least one embodiment;

[0010] Figure 4 is a view of sensor results according to at least one embodiment;

[0011] Figure 5 is a block diagram of a track header region detector according to at least one embodiment;

[0012] Fig. 6A -B illustrates a block diagram of Redundant Multi-Sensor Path Extraction (RMPE) according to at least one embodiment;

[0013] Figure 7 is a flow chart of a method for performing path extraction for a vehicle on a guideway according to at least one embodiment;

[0014] Figure 8 is a schematic diagram of a system for performing path extraction according to at least one embodiment. DETAILED DESCRIPTION

[0015] The embodiments described herein provide many different examples for realizing the different features of the provided subject matter. Specific examples such as components, values, operations, materials, arrangements, etc. are described below to simplify the embodiments described herein. Of course, these are merely examples and are not intended to be restrictive. Other components, values, operations, materials, arrangements, etc. may be considered. For example, in the following description, the formation of a first feature above or on a second feature may include an embodiment in which the first feature and the second feature are formed in direct contact, and may also include an embodiment in which an additional feature may be formed between the first feature and the second feature, so that the first and second features may not be in direct contact. In addition, the description of the embodiments herein repeats reference numerals and / or letters in various examples. This repetition is for the purpose of simplicity and clarity, and does not itself specify the relationship between the various embodiments and / or configurations discussed.

[0016] Additionally, spatially relative terms, such as "below," "beneath," "below," "above," "upper," etc., may be used herein for convenience of description to describe the relationship of one element or feature to another element or feature, as shown in the figures. Spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation shown in the figures. The device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein may likewise be interpreted accordingly.

[0017] Safety critical applications are applications where a failure or malfunction could result in death or serious injury to personnel, loss or severe damage to equipment or property, or environmental harm. Safety critical applications or systems are rated at Safety Integrity Level (SIL) 4. For a system rated at Safety Integrity Level (SIL) 4, the system provides verifiable on-demand reliability, as well as techniques and measures to detect and respond to failures that could compromise the safety nature of the system. SIL 4 is based on the International Electrotechnical Commission (IEC) IEC61508 standard and EN 50126 and 50129 standards. For a SIL 4 system, the probability of a failure per hour is 10 -8 to 10 -9 A safety system that does not meet the safety integrity level standard is called non-SIL. The embodiments described herein are implemented according to safety integrity level (SIL) 4.

[0018] The technical field of one or more embodiments includes path / track extraction in the rail industry, and one or more technical problems are to construct candidate paths for train movement using mobile autonomous sensors on a train; i.e., without assuming any information from a positioning system or reference map / infrastructure database. The candidate paths are constructed with a high degree of integrity and can be used for safety-critical navigation solutions. The mobile autonomous sensors are installed on the train and include a set of commercial off-the-shelf (COTS) sensors, where the sensors are referred to herein as passive 2D sensors (e.g., image frame-based sensors such as cameras), active 3D point cloud sensors (e.g., lidar or radar), or passive 3D point cloud sensors (e.g., stereo camera pairs). In at least one embodiment, the methods describe a combination of passive 2D sensors and the outputs of active 3D sensors and passive 3D sensors (as defined above) to achieve track path extraction. Thus, the sensors may include one or more mobile lidars (3D sensors), radars / imaging radars (3D sensors), and / or inertial measurement units (IMUs) with cameras (visible color or monochrome and / or infrared; 2D sensors). Embodiments described herein include methods and architectures for (a) extracting path information from 2D sensors and 3D sensors and (b) combining path information from 2D sensors and 3D sensors to achieve performance and integrity.

[0019] At least one embodiment combines data from multiple onboard autonomous sensors (e.g., one or more mobile lidars, cameras, radars, and / or IMUs) to extract online candidate train paths for the train with high confidence and completeness. At least one embodiment takes full advantage of sensor diversity by using complementary passive 2D and active 3D sensing (complementary multi-sensor path extraction (CMPE)) and flexibly adapts to multiple types of specific sensing technologies (redundant multi-sensor path extraction (RMPE)); for example, multiple passive 2D sensors (e.g., visible light and LWIR cameras, or two visible light cameras) and / or multiple active 3D sensors (e.g., lidar and imaging radar).

[0020] In at least one embodiment, the CMPE includes a fused single sensor pipeline for path / track extraction: a passive 2D sensor (e.g., camera) pipeline is fused with an active 3D sensor (e.g., lidar) pipeline, and (optionally) IMU data can be used to cross-check past paths with vehicle inertial data. Diversity checks and continuous sensor fusion between single sensor pipeline results are performed to detect potential errors due to limitations of single sensor technology, thereby ensuring high integrity of the results of the overall path extraction function. Optionally, in at least one embodiment, IMU data stored on a sliding time window / buffer is used to cross-check curvature and slope information of the train's current self-path. This improves the integrity of path extraction as described in the embodiments of the present invention because IMU sensors are not affected by the scene like cameras, lidar, and radar sensors. RMPE combines multiple CMPE chains with similar sensor consistency checks to improve integrity and performance.

[0021] At least one embodiment described herein uses inexpensive COTS autonomous sensors and does not require knowledge of the train's location. Thus, at least one embodiment described herein can be used in situations where the train's location is unavailable (recovering the train in the event of a lost location) or is not accurate enough.

[0022] At least one embodiment includes a method and architecture for CMPE and RMPE, including:

[0023] 1) Passive 2D sensor path extraction - Independently extract paths (CMPE and RMPE) using passive 2D sensor data.

[0024] 2) Active 3D sensor path extraction—independently extracting paths (CMPE and RMPE) using active 3D sensor data.

[0025] 3) Path fusion - combining 2D path information with 3D path information and merging them (CMPE and RMPE).

[0026] 4) Similar sensor merging - combining multiple 2D paths together, or combining multiple 3D paths together (RMPE).

[0027] 5) Supervision and cross-checking, including one or more of the following:

[0028] a) 3D path diversity check (CMPE and RMPE);

[0029] b) 3D path rationality check (CMPE and RMPE);

[0030] c) 3D path history diversity check (CMPE and RMPE);

[0031] d) 3D path consistency check (CMPE and RMPE); and

[0032] e) 3D path similarity sensor consistency check (RMPE).

[0033] At least one embodiment is in the field of path / rail / track / spline / guideway extraction or generation in the railway industry. At least one feature of at least one embodiment is the use of COTS autonomous sensors (e.g., mobile lidar, camera, radar, and IMU) on the train, rather than survey sensors, for online generation of candidate track self-paths for the train without knowing the train's location in the railway network.

[0034] Path extraction is a function used for autonomous trains. Path information is used as the basis for various vehicle situational awareness functions, such as: (1) forward object detection, detecting obstacles along the ego path that impede the safe operation of the train; (2) signal recognition, detecting ego signals (i.e., signals corresponding to the train of interest) and identifying the vehicle's position based on the ego path information; and (3) localization, localizing the vehicle by understanding the vehicle's surroundings including the path information. Therefore, accurate and complete path information is used to support the continuous functions of autonomous driving.

[0035] In other path extraction methods, the path is created through offline processing, where semi-automatic or manual methods are applied to data collected from various measurement methods (such as LiDAR, total station and GNSS / IMU) to extract the path. During the train operation, the extracted path is retrieved with the help of known vehicle positions. However, other path extraction methods include at least one or more of the following problems:

[0036] The precise position of the train is used to retrieve the appropriate routing information, which makes these methods unsuitable for situations where the train position is lost (safety recovery use case of a lost train), or where the train position has large uncertainty;

[0037] Expensive offline measurements using special equipment and additional measurement processes; or

[0038] For data collected through offline measurements, offline semi-automatic or manual work is used to process a large amount of measurement data and extract paths from the measurement data.

[0039] Online path extraction methods using onboard train autonomous sensors also have problems. Several other methods for online path extraction have been reported, which use data obtained from sensors installed on the train to detect the path. However, other path extraction based on online path extraction focuses on specific sensor data (e.g., camera-based or lidar-based data) and has one or more of the following problems:

[0040] The output / quality of the extracted paths depends on the characteristics of the sensor used.

[0041] ο The 2D paths extracted by other camera-based path extraction cannot be directly used for autonomous driving applications.

[0042] οOther LiDAR-based path extraction uses points calibrated from the track as evidence of the track to extract or identify the path. A specific installation of LiDAR is used to collect the track points (e.g., the LiDAR sensor is mounted facing the ground).

[0043] No other path extraction technology can improve the accuracy and completeness of the path extracted from a specific sensor.

[0044] According to at least one embodiment, the online route extraction method uses multiple sensors to improve the accuracy and completeness of the extracted route by integrating or fusing the routes obtained from different sensors. In addition, the online route extraction method is generic to support the use of COTS autonomous sensors, which may be limited in installation to meet the needs of other autonomous services.

[0045] Other path extraction methods exhibit one or more of the following problems:

[0046] Low robustness due to reliance on a single sensor modality: For example, lidar data is used to extract paths. Camera images are used for visible or invisible spectrum. However, heavy reliance on a single sensor brings the problem of low robustness.

[0047] Limited to specific types of LiDARs and features: For example, LiDAR-based path extraction methods are limited to specific types of LiDARs with specific point cloud patterns (e.g., rotating LiDARs with high density of points). This makes the methods less applicable to COTS mobile LiDARs, especially those with low density. Moreover, LiDAR sensors are used in more than one service, for example, for path extraction and in the signal recognition service for detecting gantries or poles. LiDARs are mounted almost parallel to the ground to be able to detect poles / gantry at longer distances. But this in turn increases the angle of incidence of the track points, and therefore, methods that rely on detecting specific track point patterns will not work properly.

[0048] Focus on generating segmentation masks for paths instead of polynomial trajectories: To handle safety-critical navigation situations, it is crucial to represent the path as a polynomial trajectory so that the polynomial can be used to predict the path that exists beyond the detectable distance. However, other camera-based path extraction methods focus on generating the path region, which is a segmentation mask. The segmentation mask indicates the pixels of the path in the input image. However, such a mask does not include any information about the polynomial trajectory of the path. In other path extraction methods, additional heavy processing is used to obtain the polynomial trajectory from the segmentation mask.

[0049] Susceptible to errors in track segmentation results: For example, camera-based path extraction methods rely heavily on utilizing track segmentation results to generate the final result, i.e., the path region mask. The quality of the track segmentation results has a great impact on the final result. However, track segmentation is prone to contain errors, especially for distant tracks, because distant tracks appear as a few pixels in the image. Therefore, the performance will be degraded even if there are small errors in the track segmentation.

[0050] Heavy reliance on prior information of scene geometry: To extract the path, camera-based path extraction works heavily rely on prior information of scene geometry, such as the relative position between the camera and the railway track and the location of the railway track. However, even a slight deviation between the prior information and the actual position can lead to performance degradation. Therefore, methods that heavily rely on prior information work in limited cases.

[0051] Description of Embodiments

[0052] Thus, at least one embodiment includes an online path extraction system using multiple sensors.

[0053] At least one embodiment combines two or more of the following elements:

[0054] A high-integrity, highly accurate path extraction method that improves the accuracy and integrity of the extracted path by combining paths obtained from multiple sensors with their own characteristics.

[0055] A state-of-the-art machine learning detector (a path extraction network combined with a semantic segmentation network), trained using a comprehensive dataset of sensor data acquired under different weather conditions.

[0056] A fully automatic LiDAR path detector based on Kalman filter to extract track paths without LiDAR sensor installation restrictions by using the blank areas of railway tracks and track points as observations for track tracking.

[0057] An accurate ground surface model generated from the LiDAR data, used to recover the 2D path obtained from the camera into a 3D path, with flexibility in scaling the process depending on the specific application and the accuracy to be achieved (by including optional inputs and by using alternative methods).

[0058] A fusion module that improves detection completeness and accuracy by examining diversity and integrating paths obtained from multiple sensors.

[0059] Flexible support for multiple different sensors of the same type, allowing intra-sensor consistency checks for paths and merging of multiple paths from the same sensor type.

[0060] Support for strong partitioning of critical (state machine / deterministic) merging and supervisory functions in applications that use critical processing. For example, critical functions and critical partitions can be defined in accordance with CENELEC standards (e.g., CENELECT EN

[0061] 50126 and 50129) were developed for track.

[0062] advantage

[0063] Robustness: At least one embodiment disclosed herein improves the robustness of the detector by fusing multiple sensor data. Different sensors have different characteristics that affect the quality of the detected path. For example, when an image is acquired at night, a camera sensor cannot detect a path, while a lidar can detect a path regardless of weather conditions and lighting conditions. On the other hand, the density of lidars decreases with increasing distance, resulting in a decrease in ground points (or track points). At least one embodiment disclosed herein compensates for the shortcomings of different sensors through a fusion process combined with logical operations. This design allows at least one embodiment to robustly detect a path regardless of different weather and lighting conditions.

[0064] High Accuracy: At least one embodiment disclosed herein improves the accuracy and completeness of the extracted path by integrating passive 2D sensors and active 3D sensors. Prior to training operations, at least one embodiment accurately integrates the paths detected from different sensor data through a diversity check and integration process in a fusion module through accurate multi-sensor calibration. In addition, at least one embodiment is designed to increase the accuracy of sensor-level path extraction. The active 3D path extractor uses track points and blank areas (observed on the side of or on the track due to high incidence angles) as track evidence to increase the accuracy of the extracted path.

[0065] Fully automated: At least one embodiment is fully automated, eliminating tedious pre-processing and initialization processes. The relationship between multiple sensors is represented by a reference frame (bogie frame) that provides a fixed relationship between the train and the train path. The reference frame automatically provides the initial track position of the path extracted by the individual sensors. In addition, the detector is designed to input raw sensor data without any pre-processing processes such as sub-sampling and data editing, and output the path, so that the detector does not involve manual work.

[0066] Scalability: In at least one embodiment, the detector is modularized to increase scalability. The embodiments described herein include two sensor pipelines (a lidar pipeline and a camera pipeline) and a fusion pipeline. The sensor pipelines input sensor data and output independent paths, while the fusion pipeline collects the outputs of the sensor pipelines in a fully automatic manner and outputs accurate paths. Therefore, new sensor pipelines can be added to the designed system.

[0067] Figure 1 is a block diagram of an overall path extraction architecture 100 for CMPE according to at least one embodiment.

[0068] According to at least one embodiment, a path extraction system receives input data frames from multiple sensors. A data frame refers to the data and metadata output by a sensor at a given moment, corresponding to the measurement and processing operations performed by the sensor in a short period of time before that moment. For example, a single camera image or a single lidar point cloud is considered a single data frame.

[0069] exist Figure 1, an overall path extraction architecture 100 includes an active 3D sensor path extraction module 110, a ground model generation module 120, a passive 2D path extraction module 130, and a fusion module 140. The overall path extraction architecture (or simply "system") 100 for CMPE receives data frames from a sensor model. The system 100 receives data frames from sensors that correspond to measurements performed at a given moment. The system includes multiple sensor pipelines (two sensor processing pipelines in at least one embodiment - active 3D sensor pipeline input 112 and passive 2D sensor pipeline input 132 - but can be expanded if redundant sensor types are available; see reference below) Fig. 6A -B described RMPE). Active 3D sensor or passive 3D sensor (e.g., lidar, radar) input 122 is provided to the 3D sensor ground surface model generator 120. The system 100 also includes path fusion 140 and supervision 150, and the supervision 150 includes a 3D path diversity check 151, a 3D path rationality check 152, a 3D path history diversity check 153, and a 3D path consistency check 154. The sensor pipeline is processed in parallel, and subsequent path fusion 140 and supervision 150 modules are executed when the output of the sensor pipeline is available. The sensors of the system 100 begin collecting data when powered on. In response to being triggered, the path extraction software (e.g., active 3D sensor path extraction module 110, ground model generation module 120, or passive 2D path extraction module 130) requests data from sensors, sensor buffers, sensor gateways, etc., and begins processing path extraction.

[0070] The active 3D sensor path extractor 110 receives active 3D sensor input 112 (such as LiDAR or RADAR) and constraints 114 (such as track gauge, radius of curvature (ROC), slope, etc.). With currently available technology, CMPE is preferably implemented using LiDAR (as an active 3D sensor) and a visible spectrum camera (as a passive 2D sensor). As sensing technology becomes cheaper, other active 3D sensors (e.g., imaging radar) and passive 2D sensors (e.g., LWIR) can be incorporated into the embodiments described herein. Industrial applications of at least one embodiment include automatic path extraction, switch determination systems, driver assistance systems, and / or object detection systems for vehicles or robots traveling along a known fixed route.

[0071] Other LiDAR-based path extraction methods use high-density survey LiDAR data (using airborne laser scanning (ALS), mobile laser scanning (MLS), or unmanned aircraft systems (UAS)) in which the track head points can be clearly observed. Other algorithms extract the track head points as evidence of the railway track by applying various classical clustering algorithms (such as region growing, RANSAC, K-means clustering, and EM-GMM clustering) based on the railway track geometry, and then model the 3D track path by connecting the track head points or tracking the track head points. However, when the low-density LiDAR sensor is mounted horizontally on the vehicle (the sensor mounting method in mobile autonomous applications), the track points are difficult to observe due to the large incident angle and track material. Therefore, other path extraction methods cannot be used to extract the track path because other path extraction methods are designed to extract the path assuming that the track points can be well detected from the LiDAR data.

[0072] The active 3D sensor path extractor 110 takes into account the physical properties of the active sensor signal reflecting from the rail top. LiDAR and radar sensor signals do not reflect strongly back from smooth metal, and the absence of diffuse scattering results in low signal returns in the back reflection direction and few sample points from the rail top / rail head / rail head. Therefore, the standard method of using the signal returned from the rail top / rail head is not sufficient in actual operation, and one or more embodiments of the method proposed herein address this problem.

[0073] The active 3D sensor path extractor 110 relies on extracting a specific 3D point pattern where two empty lines (i.e., no 3D point returns) are obtained between dense areas of back-reflection points, indicating the positions of the left and right rails of the track. As described above, this phenomenon occurs due to the shallow incidence angle of the sensor (e.g., greater than 45 degrees from the vertical) and the nature of the metal track, for example, the incident signal will be reflected off the sensor, resulting in no return from the track. The active 3D sensor path extractor 110 extracts the track in the absence of back-reflection from the rail head, which is different from the standard method used in LiDAR measurement path / track extraction. While the LiDAR measurement method relies on extracting the rail head points to extract the track points, it is not possible for LiDARs with significantly lower point density than the measurement LiDAR. Similarly, the active 3D sensor path extractor 110 extracts the track based on reflections from the ground and the absence of reflections from the track due to the incidence angle. In addition, in the case of Doppler measurement sensors (such as radars), the direction of the velocity vector provides a secondary cross-check for estimating the track curvature.

[0074] The passive 2D sensor path extractor 130 first relies on a multi-task deep neural network model for track point prediction. Then, candidate paths are created from the track points. The passive 2D sensor path extractor 130 relies on two steps: (1) creating track segments from track points through point clustering, and (2) creating candidate paths from track segments by linking track segments. The multi-task deep neural network model combines a semantic segmentation network with a regression branch to predict track points in an image. In particular, the regression branch is designed to produce pixel-level outputs including centrality (i.e., how close a pixel is to the center of the track) and horizontal distance to the left and right tracks. The regression branch and its combination with the semantic segmentation network is one of the novel aspects of the embodiments described herein.

[0075] The 3D sensor ground surface model generator 120 receives the passive 2D sensor path extracted by the passive 2D sensor path extraction module 130 based on the passive 2D sensor pipeline input 132, and receives the active or passive 3D sensor input 122 to generate the 3D sensor ground model. Using the 2D-3D calibration parameters 124 stored in the database, the passive 2D path in the image space (represented by a set of vertices) is converted into a line vector in the local frame of the corresponding 3D sensor (for generating the surface model).

[0076] The passive 3D path generator 126 recovers the missing one dimension of the passive 2D sensor path extracted by the passive 2D sensor path extractor 130 by integrating the surface model generated by the 3D sensor ground surface model generator 120. The passive 3D path generator 126 receives constraints 128, such as track gauge, radius of curvature (ROC), slope, etc. The 3D path is recovered by intersecting the line vector and plane generated by the 3D sensor ground surface model generator 120. The supervisor 150 generates an alarm when the time difference between the surface model and the 2D sensor frame data is greater than a user-defined threshold, such as 1 second, such as the time difference between a LiDAR ground point and a camera.

[0077] Supervision 150 includes a 3D path diversity check 151. The 3D path diversity check 151 receives the active 3D sensor path generated by the active 3D sensor path extractor 110 and the passive 3D path generated by the passive 3D path generator 126. The 3D path diversity check 151 compares the previous fused path (generated in the previous time instant, corresponding to the time window of length p) with the current path (generated by the sensor pipeline and assumed to include additional path data, such as path data at a long distance from the vehicle). The 3D path diversity check 151 creates a buffer along the previous fused path and overlaps the buffer with the vertices of the current path. The buffer tolerance interval at a point is defined relative to a predetermined deviation and relative to the expected error in the path from the passive 2D sensor path extractor 130 and the active 3D sensor path extractor 110. The ratio in the buffer is calculated based on the vertices in the previous fused path and the vertices of the current path, and it is determined whether the current path passes the diversity check based on the ratio being above a predetermined value. Note that supervision will issue an alarm for low values ​​of the ratio. The buffer threshold according to at least one embodiment is 30cm. The buffer ratio is the number of vertices in the buffer (generated by the previous fused path) (current path) / total number of vertices (current path) * 100. In response to the buffer ratio being greater than 90%, the diversity check passes. However, those skilled in the art recognize that the 90% buffer ratio value can be set to another predetermined value. In response to the buffer ratio being less than a predetermined value, such as 50%, 75%, or the like, the diversity check fails.

[0078] The diverse sensor 3D path merging (“path fusion”) 140 receives the active 3D sensor path from the active 3D sensor path extractor 110, the input from the 3D path diversity check 151, and the passive 3D path from the passive 3D path generator 126. When the 3D path diversity check 151 succeeds, the diverse sensor 3D path merging (“path fusion”) 140 merges the previous path inertial measurement value z with the passive 3D path generator 126. -p…-1 142 is integrated with the current path. The vertices of the current path in the buffer mentioned in the 3D path diversity check 151 are collected and the spline parameters are calculated using these vertices. Diversified sensor 3D path merging ("path fusion") 140 can be configured for performance (i.e., the union of two paths or parts of two paths can be used; i.e., if the current path is significantly shorter than the previous path, some vertices of the previous path can be used), or it can be configured for completeness (i.e., the intersection of two paths).

[0079] After estimating the new spline parameters, the vertices are sampled at regular intervals to generate the final accurate path. The diverse sensor 3D path merging ("path fusion") 140 does not rely on the ordered outputs from the active 3D sensor path extractor 110 and the passive 3D path generator 126 as input. The diverse sensor 3D path merging ("path fusion") 140 maintains the latest fused path and integrates the latest fused path with the new input. The diverse sensor 3D path merging ("path fusion") 140 outputs a highly accurate and highly complete fused path. High accuracy means that the path generated by one sensor data is confirmed by another sensor data, thereby improving the accuracy. High completeness refers to a path generated by one sensor, which partially lacks a part of the complete path due to the characteristics of the sensor. The missing part of the path can be completed by fusing other sensor data by the diverse sensor 3D path merging ("path fusion") 140.

[0080] The 3D path plausibility check 152 receives the output from the diverse sensor 3D path merging ("path fusion") 140 and the constraints 160 (e.g., track gauge, ROC, slope, etc.). The 3D path plausibility check 152 breaks down the current path (the look-ahead path) into sections (of configurable size) and confirms (where possible, based on available information from the network) the following 3D path plausibility parameters (in order of priority):

[0081] The path is smooth, without kinks or interruptions;

[0082] The path has the correct track gauge;

[0083] The path has an acceptable radius of curvature (ROC) (larger than the expected minimum);

[0084] The path has an acceptable slope (within the expected range);

[0085] The path has acceptable hills and sags (within expected ranges);

[0086] The path has an acceptable track transition curve (within expected guidelines); and

[0087] • The path has acceptable superelevation / crosslevel (within expected ranges), e.g. as defined by a standard.

[0088] Depending on the processing power and performance of the onboard sensors, one, more, or all of these checks may be performed (with higher levels of checks enabled by greater available power and higher performing sensors).

[0089] 3D path history diversity check 153 receives input from 3D path rationality check 152 and inertial sensor input 170. 3D path history diversity check 153 compares the past traversed paths (from past windows of size k) with the past inertial measurements z in the window of size k. -k…-1 172 is used to compare the determined path. In the comparison of the two 3D paths, the check is similar to the 3D path diversity check 151, with a predetermined deviation at the point where IMU noise is taken into account. As described above, the threshold value according to at least one embodiment is 30cm. In addition, the 3D path history diversity check 153 takes into account the branch points passed by comparing the two paths, and is able to subsequently use this information to determine the branch passed (i.e., the branch corresponding to the successful diversity check). The 3D path history diversity check 153 confirms that the path passed in the past is credible and that the path passed in the past matches at least one fused 3D path extracted in the past after the current vehicle position. The 3D path history diversity check 153 also identifies which of the potential two paths (at the branch) is the path that has been passed (and therefore is the correct) path passed in the past.

[0090] 3D path consistency check 154 receives the output from 3D path history diversity check 153 and constraints 162 (e.g., track gauge, ROC, slope, etc.). 3D path consistency check 154 also receives the output from objects z that have been fused in the past. -m…-1 174. The 3D path consistency check 154 uses the past and current (forward) paths to verify that there are no continuity breaks or "kinks" and confirms that the checks outlined in the 3D path rationality check 152 hold on the historical and current path information. The 3D path consistency check 154 provides a fused 3D path 180 as output.

[0091] Figure 2 is a block diagram 200 showing further details of active and passive sensors and a surface model generator according to at least one embodiment.

[0092] exist Figure 2, the active 3D sensor path extractor 210 includes initialization 212, prediction 214, track head region detection 216, and update state 218. Initialization 212 receives the active 3D sensor input 202 and automatically determines the starting point (left or right railway track) and direction of the track in the frame data. A pair of track windows of user-defined (parameterized) length and width are used to capture the physical relationship between the tracks and constrain subsequent track extraction during the path tracking process. In at least one embodiment, when the track gauge is 1.5m, the track window can be, for example, 1m by 0.4m. However, those skilled in the art recognize that the embodiments described herein are not meant to be limited to these example dimensions, and the embodiments described herein can use other dimensions. The initial position of the track is determined by a search area, which is generated by considering the minimum distance to the ground point (calculated using the sensor mounting configuration) and the maximum curvature. Candidate positions for a pair of track windows are created at regular intervals along the cross section.

[0093] Prediction 214 receives the candidate position from initialization 210 to predict the next state vector including position (x, y and z) and orientation based on the previous state vector in a Kalman filter framework.

[0094] The track head region detection 216 receives the next state vector from the prediction 212. The track head region detection 216 determines the observation of the track head in the Kalman filter framework. The track head region detection 216 is applied to a candidate position of a pair of track windows to find the initial track region. When the head track region of the track is found, the candidate position is considered the initial track region. Note that multiple initial track regions can be detected for multiple track regions. In addition, the track head region detection 216 provides the direction of the track.

[0095] Update state 218 provides an updated state vector predicted by prediction 212 using observations obtained from track head region detection 216 in a Kalman filter framework.

[0096] The 3D sensor surface model generator 220 includes a ground filter 222 and a surface model generator 224. The ground filter 222 receives an active 3D sensor or passive 3D sensor input 204 (e.g., a lidar, an imaging radar, or a stereo camera) and divides the lidar points into ground points and non-ground points based on a grid-based filtering method constrained by a slope. After generating a grid and assigning the lidar points to the grid, the 3D sensor ground filter 222 collects the lowest points in the grid. Ground seed points are selected by comparing the k nearest lowest points in terms of distance and slope. Ground points are then detected by collecting points below a certain height from the ground seed points. The remaining points are considered non-ground points. The surface model generator 224 receives a 2D-3D calibration 226 and creates a surface model from the active 3D sensor or passive 3D sensor input 204, assuming that the ground is modeled by a plane within a frame of data. For example, the plane parameters are calculated by using a least squares method applied to a subset of the input 3D sensor data. In order to select a subset, a combination of one or more methods can be used:

[0097] 1) Unconstrained method: Use ground points to fit a plane.

[0098] 2) Loosely constrained method: This method does not use all ground points, but uses points that fall within the region that are generated by considering the maximum curvature of the track design.

[0099] 3) Path Constrained Method: This method uses a subset of ground points corresponding to the available passive 2D path information 206 (+ / - a configurable buffer).

[0100] Regardless of the method used to select the plane fit subset, the plane parameters are updated whenever new 3D sensor frame data is input. The specific method to be used depends on the operating environment (less complex track geometry favors less constrained methods; open and / or hilly terrain may favor more constrained methods), the processing power available in a particular platform configuration (less available power favors less constrained methods), the integrity requirements (higher integrity requirements favor more constrained methods), or the required extraction performance (e.g., extraction range, where shorter range requirements favor less constrained methods). The output of the surface model generator 224 is a 3D surface model 228.

[0101] The passive 2D sensor path extraction 230 receives the passive 2D sensor input 208 at the semantic segmentation network 232. In the context of camera-based path extraction, the semantic segmentation network 232 receives frames of data from a camera. The semantic segmentation network 232 implements a deep neural network with an encoder-decoder structure. The semantic segmentation network 232 is combined with a regression branch 234 trained using a synthetic dataset. The output of the semantic segmentation network 232 is a feature map, which is used by the regression branch 234.

[0102] The regression branch 234 creates pixel-level information for the track from the combination of feature maps provided by the semantic segmentation network 232. For a pixel in a given input image, this information includes: (i) the degree to which it is the center point of the track, (ii) the horizontal distance to the left track of the track, and (iii) the horizontal distance to the right track of the track. The regression branch 234 implements three sub-branches that perform convolution operations to create three types of information.

[0103] Track segment generator 236 receives pixel-level information for tracks from regression branch 234. The input image is segmented into sub-regions, i.e., non-overlapping windows, where the width of the window is equal to the width of the image and has a certain height. Then, within the window, track center points (pixels with higher degree values ​​as track center points) are clustered into track segments by clustering spatially adjacent track center points into a group. The output of track segment generator 236 is a list of track segments in the window.

[0104] The self-path generator 238 receives track segments from the track segment generator 236. Based on the track segments in the windows, a tree is constructed to represent topological information about the path. Starting with the track segment located around the center of the bottommost window as the start node, spatially adjacent track segments are clustered on two adjacent windows. When a track segment in a window is spatially close to two different track segments in the upper window, the track segment is regarded as the intersection of the path, i.e., the bifurcation point at the switch. The generated tree has the following three types of nodes: (1) a start node, which indicates the point where the path starts; (2) an end node, which indicates the point where the path ends; and (3) a switch node, which indicates the bifurcation point in the path. The edges between the nodes include trajectories between the nodes, where the trajectory includes the track center point and the corresponding left and right track points. Based on the tree, a possible self-path is obtained by simply passing from the end node to the start node. The output of the self-path generator 238 is a possible self-path, i.e., a passive 2D path 240, where the self-path is a trajectory including the track center point and the corresponding left and right track points of the path.

[0105] Figure 3 is a plan view of a vehicle on track 300 , according to at least one embodiment.

[0106] exist Figure 3, a vehicle 310 is shown on a track 312. The vehicle 310 includes one or more sensors 314. The sensors 314 observe two track windows 320, 322. The sensors 314 cover a track region having a curvature constraint 330. The track 312 is located in the track region having the curvature constraint 330. The track windows 320, 322 are aligned with track 1 340 and track 2 342 of the track 312, respectively. The track windows 320, 322 are defined by a length 350 and a width 352. The track windows 320, 322 are separated so that the distance between the approximate centers of the track windows 320, 322 is aligned with the spacing 360 between track 1 340 and track 2 342 of the track 312. The one or more sensors 314 determine the track head point by investigating the statistics of the points belonging to the track windows 320, 322.

[0107] Figure 4 is an illustration of sensor results 400 according to at least one embodiment.

[0108] exist Figure 4 , a reflection point 410 from one or more sensors is shown. Due to the low density of airborne laser radar data and installation limitations, blank areas 420, 422, 424, 426 of the rail head are observed in the rail head area 430, 432, 434, 436. The blank areas 420, 422, 424, 426 represent railway tracks 440, 442, 444, 446. The blank areas 420, 422, 424, 426 are caused by low diffuse scattering due to high incident angles and the geometric relationship between the phase of the reflected wave relative to the roughness of the reflecting surface. Unlike other path extraction methods, at least one embodiment disclosed herein uses rail head points and blank areas 420, 422, 424, 426 observed in the rail head area, which are observed in the rail head area due to the low density of airborne laser radar data and installation limitations.

[0109] Figure 5 is a block diagram of a track header region detector 500 according to at least one embodiment.

[0110] exist Figure 5 In the embodiment, the track head region detector 500 checks whether there is a track head point by investigating the statistical data of the points within the track window 510. The detection of the track head point is performed by analyzing the height distribution and / or continuity of the points on the track profile.

[0111] The points within the track window 510 are provided to a statistical analyzer 520. When the statistical analyzer 520 indicates the presence of a track head point based on the height distribution and / or continuity of the points on the track profile, the statistical analyzer 520 divides the points belonging to the track window into a track head, a track waist (web), and a track bed (bed) (including ground points) according to a distribution segmentation or clustering method, such as expectation maximization-Gaussian mixture model (EM-GMM) clustering 540. A line extractor 542 generates lines representing clusters by fitting the track head points to the lines using random sampling consensus (RANSAC). RANSAC is an iterative method for estimating the parameters of a mathematical model from a set of observation data including outliers when the outliers have no effect on the estimated values. Therefore, the RANSAC of the line extractor 542 can also be interpreted as an outlier detector. The center and direction analyzer 550 generates an observation 560. The observation 560 includes the center and direction of the line.

[0112] When the statistical analyzer 520 indicates that there is no rail head point, the rail head region detector 500 attempts to find a blank area caused by a high angle of incidence. An occupancy grid 530 is generated using the points belonging to the track window, and non-occupied pixels are clustered using a connected component clustering algorithm 532. The connected component clustering algorithm 532 provides clusters for rail heads to the railway track region selector 534. The railway track region selector 534 detects rail heads by analyzing the linearity of the clusters. The line extractor 536 is similar to the case of rail heads and generates lines representing the clusters using the RANSAC algorithm. The center and direction analyzer 550 produces observations 560. The observations 560 include the center and direction of the line.

[0113] Fig. 6A -B is a block diagram of a redundant multi-sensor path extraction (RMPE) 600 according to at least one embodiment.

[0114] The RMPE system 600 is used when there are multiple redundant sensors of a given type in the architecture (e.g., two active 3D sensors such as LiDAR and radar, and two passive 2D sensors such as LWIR and a visible light camera). Fig. 6A -B is the configuration flow chart of this processing architecture.

[0115] exist Fig. 6A In FIG. 6 , the passive 2D sensor path extractor 610 includes two passive 2D sensor path extractors 612 and 614 . A passive 2D input 1 616 is provided to the passive 2D sensor path extractor 612 , and a passive 2D input 2 618 is provided to the passive 2D sensor path extractor 614 .

[0116] The 3D ground surface model generator 620 includes 3D ground surface model generators 622 , 624 . Active or passive 3D input 626 is provided to the 3D ground surface model generators 622 , 624 . The 3D ground model generators 622 , 624 also receive 2D-3D calibration parameters 628 .

[0117] The redundant multi-sensor path extraction (RMPE) 600 further includes an active 3D sensor path extractor 630 and a passive 3D path generator 640. The active 3D sensor path extractor 630 includes an active 3D sensor path extractor 631, an active 3D sensor path extractor 632, and a 3D path similarity sensor consistency check 633.

[0118] The active 3D sensor path extractor 631 receives active 3D sensor (e.g., LiDAR, radar) measurements as active 3D input 1 634 and constraints 635. The active 3D sensor path extractor 632 receives active 3D sensor (e.g., LiDAR, radar) measurements as active 3D input 2 636 and constraints 635. The active 3D sensor path extractor 631 and the active 3D sensor path extractor 632 take into account the physical properties of the active sensor signal reflecting from the rail top. The LiDAR and radar sensor signals are not strongly reflected back from smooth metal, and the signal return in the reverse reflection direction is low due to the lack of diffuse scattering, and there are few sample points at the rail top / rail head / rail head. The active 3D sensor path extractor 631 and the active 3D sensor path extractor 632 extract a specific 3D point pattern, in which two empty lines (i.e., no 3D point returns) are determined between the dense areas of reverse reflection points, wherein the two empty lines represent the positions of the left and right rails of the track. Active 3D sensor path extractor 631 and active 3D sensor path extractor 632 extract the track based on reflections from the ground and the absence of reflections from the track due to the angle of incidence. In addition, in the case of Doppler measurement sensors such as radar, the direction of the velocity vector provides a secondary cross-check for the estimated track curvature.

[0119] The 3D path similarity sensor consistency check 633 compares the outputs of two or more active 3D paths. The 3D path similarity sensor consistency check 633 is similar to the consistency check described for the 3D path diversity check 660 described below, with tolerances appropriately defined for the various sensor path extraction tolerances.

[0120] The active 3D sensor path extractor 631 generates an active 3D sensor path as an output A, the active 3D sensor path extractor 632 generates an active 3D sensor path as an output B, and the 3D path similarity sensor consistency check 633 generates a similarity indication between the active 3D sensor path from the active 3D sensor path extractor 631 and the active 3D sensor path from the active 3D sensor path extractor 632 as an output C. The 3D path similarity sensor consistency check 633 generates a consistency flag 1 in response to the two outputs being consistent, and generates a consistency flag 0 in response to the two outputs being inconsistent.

[0121] The passive 3D path generator 640 includes a passive 3D path generator 641 , a passive 3D path generator 642 , and a 3D path similarity sensor consistency check 643 .

[0122] The passive 3D path generator 641 receives the output of the passive 2D sensor path extractor 612, the output of the 3D sensor ground surface model generator 622, and the 2D-3D calibration parameters 644. The passive 3D path generator 642 receives the output of the passive 2D sensor path extractor 614, the output of the 3D sensor ground surface model generator 624, and the 2D-3D calibration parameters 644.

[0123] The passive 3D path generator 641 and the passive 3D path generator 642 respectively restore the missing one dimension of the passive 2D sensor path obtained by the passive 2D sensor path extractor 612 from the passive 2D sensor input 1 616 and the missing one dimension of the passive 2D sensor path obtained by the passive 2D sensor path extractor 614 from the passive 2D sensor input 2 618. The passive 3D path generator 641 and the passive 3D path generator 642 generate the passive 3D path by integrating the surface models generated by the 3D sensor ground surface model generator 622 and the 3D sensor ground surface model generator 624, respectively. By using the 2D-3D calibration parameters 644 stored in the database, the passive 2D sensor path in the image space (represented by a set of vertices) is converted by the passive 2D sensor path extractors 612 and 614 to line vectors in the local frame of the corresponding 3D sensor (for generating the surface model). The 3D path is restored by intersecting the line vectors generated by the 3D sensor ground surface model generators 622 and 624, respectively, with the plane. Note that the system has supervisory logic that issues an alarm when the time difference between the 3D sensor ground model and the 2D sensor frame data from the 3D sensor ground surface model generators 622, 624, respectively, is greater than a user-defined threshold, such as 1 second, according to at least one embodiment.

[0124] 3D path similarity sensor consistency check 643 receives the output of passive 3D path generator 641 and the output of passive 3D path generator 642. 3D path similarity sensor consistency check 643 compares two or more passive 3D paths respectively to evaluate similarity and issues an alarm when the paths exceed a predetermined tolerance. For example, according to at least one embodiment, the predetermined tolerance is 15 cm.

[0125] Passive 3D path generator 641 generates a passive 3D sensor path as output D, passive 3D path generator 642 generates a passive 3D sensor path as output E, and 3D path similarity sensor consistency check 643 generates an indication of similarity between the passive 3D sensor path from passive 3D sensor path extractor 641 and the passive 3D sensor path from passive 3D sensor path extractor 642 as output F. Inertial sensor input 645 is provided at output G.

[0126] refer to Figure 6B , the 3D path similarity sensor merging 650 includes an active 3D path merging 652 and a passive 3D path merging 654. The active 3D path merging 652 receives the active 3D sensor path from the active 3D sensor path extractor 631 at input A, receives the active 3D sensor path from the active 3D sensor path extractor 632 at input B, and receives input C from the 3D path similarity sensor consistency check 633. The passive 3D path merging 654 receives the passive 3D path from the passive 3D path generator 641 at input D, receives the passive 3D path from the passive 3D path generator 642 at input E, and receives input F from the 3D path similarity sensor consistency check 643. The active 3D path merging 652 integrates the active 3D sensor path from the active 3D sensor path extractor 631 and the active 3D sensor path from the active 3D sensor path extractor 632. The passive 3D path merge 654 integrates the passive 3D path from the passive 3D path generator 641 and the passive 3D path from the passive 3D path generator 642 .

[0127] Active 3D path merging 652 and passive 3D path merging 654 provide an integrated active 3D path and an integrated passive 3D path as inputs to a 3D path diversity check 660. The 3D path diversity check 660 compares a previous fused path (generated at a previous time instant, corresponding to a time window of length p) with a current path (generated by the sensor pipeline and assumed to include additional path data, e.g., path data at a long distance from the vehicle). The 3D path diversity check 660 creates a buffer along the previous fused path and overlaps the buffer with the vertices of the current path. The buffer tolerance interval at a point is defined relative to a predetermined deviation and relative to an expected error in the passive 2D sensor paths from the passive 2D sensor path extractors 612, 614, respectively, and the active 3D sensor paths from the active 3D sensor path extractors 632, 633, respectively. Based on the vertices in the previous fused path and the vertices in the current path, a ratio in the buffer is calculated, and based on the ratio being above a predetermined value, it is determined in the 3D path diversity check 660 whether the current path passes the diversity check. Note that there is supervision that will raise an alarm for low values ​​of the ratio.

[0128] Active 3D path merging 652, passive 3D path merging 654, and 3D diversity check 660 provide input to diverse sensor 3D path merging ("path fusion") 670. Diversified sensor 3D path merging ("path fusion") 670 also receives the previous path inertial measurements z -p…-1 672.

[0129] The functionality of the diverse sensor 3D path merging ("path fusion") 670 is similar to the 3D path similar sensor merging 650, including active 3D path merging 652 for active path merging and passive 3D path merging 654 for passive path merging. The diverse sensor 3D path merging ("path fusion") 670 can be configured for performance (i.e., outputting the union of two paths within a tolerance range) or completeness (i.e., outputting the intersection of two paths within a tolerance range). This functionality allows for flexible adjustment based on use cases and / or sensor selection.

[0130] When the 3D path diversity check 660 succeeds, the diverse sensor 3D path merging (“path fusion”) 670 combines the previous path inertial measurements z -p…-1672 is integrated with the current path. The vertices of the current path in the buffer mentioned in the 3D path diversity check 660 are collected and the spline parameters are calculated using these vertices. Note that the diverse sensor 3D path merging ("path fusion") 670 can be configured for performance (i.e., the union of the two or part of the two can be used; i.e., in the case where the current path is significantly shorter than the previous path, part of the vertices of the previous path can be used), or it can be configured for completeness (i.e., the intersection of the two paths). Once the new spline parameters are estimated, the vertices are sampled at regular intervals to generate the final accurate path. The diverse sensor 3D path merging ("path fusion") 670 does not rely on the ordered output from the sensor pipeline as input. The diverse sensor 3D path merging ("path fusion") 670 maintains the latest fused path and integrates the latest fused path with the new input. The diverse sensor 3D path merging ("path fusion") 670 outputs a highly accurate and highly complete fused path.

[0131] Diverse sensor 3D path merging ("path fusion") 670 provides the fused 3D path to 3D path rationality check 680. 3D path rationality check 680 also receives constraints 682. 3D path rationality check 680 subdivides the current path (look-ahead path) into sections (of configurable size) and confirms (where possible, based on available information from the network) the following 3D path rationality parameters (in order of priority):

[0132] The path is smooth, without kinks or interruptions;

[0133] The path has the correct track gauge;

[0134] The path has an acceptable radius of curvature (ROC) (larger than the expected minimum);

[0135] The path has an acceptable slope (within the expected range);

[0136] The path has acceptable slopes and depressions (within expected ranges);

[0137] The path has an acceptable track transition curve (within expected guidelines); and

[0138] • The path has acceptable superelevation / crossing levels (within expected ranges), e.g. as defined by the standard.

[0139] Depending on the processing power and performance of the onboard sensors, one, more, or all of these checks may be performed (with higher levels of checks enabled by greater available power and higher performing sensors).

[0140] 3D path rationality check 680 provides input to 3D path history diversity check 684. 3D path history diversity check 684 compares the past traversed paths (from a past window of size k) with past inertial measurements z according to input G from inertial sensor input 645. -k...-1 and z in a window of size k -k...-1 686. In the comparison of the two 3D paths, the 3D path history diversity check 684 is performed similarly to the 3D path diversity check 660, with a predetermined deviation at the point where IMU noise is taken into account. In addition, the 3D path history diversity check 684 takes into account the branching points that have been passed by comparing the two paths, and then uses this information to determine the branch that has been passed (i.e., the branch that corresponds to a successful diversity check). The 3D path history diversity check 684 confirms that the past path passed can be trusted and matches at least one of the fused 3D paths 694 extracted in the past after the current vehicle position. The 3D path history diversity check 684 also identifies (at a branch) which of the potential two paths is the past path that has been passed (and therefore is the correct one).

[0141] 3D path history diversity check 684 provides input to 3D path consistency check 688. 3D path consistency check 688 also receives input from constraints 690 and past inertial measurements z -m…-1 692 Input. 3D path consistency check 688 using past path inertial measurements z -m…-1 692 and the current (forward) path to verify that there are no continuity breaks or "kinks" and to confirm that the checks outlined in the 3D path rationality check 680 are maintained on the historical and current path information.

[0142] The 3D path consistency check 688 provides a fused 3D path 694 at the output. The fused 3D path 694 is provided as path z -p…-1 672, z -k…-1 686 and z -m…-1 692.

[0143] Figure 7 is a flow chart 700 of a method for performing path extraction for a vehicle on a guideway according to at least one embodiment.

[0144] exist Figure 7 In the process, the process starts S702, and two or more sensor inputs are received from two or more sensors, the two or more sensor inputs including at least one active 3D sensor input from at least one active 3D sensor and at least one passive 2D sensor input from at least one passive 2D sensor S710. Fig. 6A, the passive 2D sensor path extractor 610 includes two passive 2D sensor path extractors 612, 614. Passive 2D input 1 616 is provided to the passive 2D sensor path extractor 612, and passive 2D input 2 618 is provided to the passive 2D sensor path extractor 614. The active 3D sensor path extractor 631 receives active 3D sensor (e.g., lidar, radar) measurements as active 3D input 1 634 and constraints 635. The active 3D sensor path extractor 632 receives active 3D sensor (e.g., lidar, radar) measurements as active 3D input 2 636 and constraints 635.

[0145] Extracting at least one active 3D sensor path based on at least one active 3D sensor input and extracting at least one passive 2D sensor path based on at least one passive 2D sensor input S714. Fig. 6A , the active 3D sensor path extractor 631 receives active 3D sensor (e.g., lidar, radar) measurements as active 3D input 1 634 and constraints 635. The active 3D sensor path extractor 632 receives active 3D sensor (e.g., lidar, radar) measurements as active 3D input 2 636 and constraints 635. The active 3D sensor path extractor 631 and the active 3D sensor path extractor 632 take into account the physical properties of the active sensor signal reflecting from the rail top. The lidar and radar sensor signals are not strongly reflected back from smooth metal, and the signal return in the reverse reflection direction is low due to the lack of diffuse scattering, and there are few sample points at the rail top / rail head / rail head. The active 3D sensor path extractor 631 and the active 3D sensor path extractor 632 extract a specific 3D point pattern, in which two empty lines (i.e., no 3D point returns) are determined between the dense areas of reverse reflection points, wherein the two empty lines represent the positions of the left and right rails of the track. Active 3D sensor path extractor 631 and active 3D sensor path extractor 632 extract the track based on reflections from the ground and the absence of reflections from the track due to the angle of incidence. In addition, in the case of Doppler measurement sensors such as radar, the direction of the velocity vector provides a secondary cross-check for the estimated track curvature.

[0146] Generate at least one 3D sensor ground model based on at least one passive 2D sensor path S718. Fig. 6A The passive 3D path generator 641 and the passive 3D path generator 642 generate passive 3D paths by integrating the surface models generated by the 3D sensor ground surface model generator 622 and the 3D sensor ground surface modeling generator 624, respectively.

[0147] Generate at least one passive 3D path based on at least one passive 2D sensor path and at least one 3D sensor ground surface model S722. Fig. 6A , the passive 3D path generator 641 and the passive 3D path generator 642 respectively recover the missing one dimension of the passive 2D sensor path derived from the passive 2D sensor input 1 616 by the passive 2D sensor path extractor 612 and the missing one dimension of the passive 2D sensor path derived from the passive 2D sensor input 2 618 by the passive 2D sensor path extractor 614. By using the 2D-3D calibration parameters 644 stored in the database, the passive 2D sensor path in the image space (represented by a set of vertices) is converted by the passive 2D sensor path extractors 612, 614 to a line vector in the local frame of the corresponding 3D sensor (for generating a surface model). The 3D path is recovered by intersecting the line vectors generated by the 3D sensor ground surface model generators 622, 624, respectively, with the plane.

[0148] Perform a first supervisory check, the first supervisory check including at least one active 3D path similarity sensor consistency check for at least one active 3D sensor path and at least one passive 3D path similarity sensor consistency check for at least one passive 3D path to verify the integrity of at least one active 3D sensor path and at least one passive 3D path S726. Fig. 6A , 3D path similarity sensor consistency check 633 compares the outputs of two or more active 3D paths. 3D path similarity sensor consistency check 633 is similar to the consistency check described below for 3D path diversity check 660, with tolerances appropriately defined for various sensor path extraction tolerances. 3D path similarity sensor consistency check 643 receives the output of passive 3D path generator 641 and the output of passive 3D path generator 642. 3D path similarity sensor consistency check 643 compares two or more passive 3D paths, respectively, to assess similarity and raise an alarm if the paths are outside of a predetermined tolerance.

[0149] A second supervisory check S730 including a 3D path diversity check is performed in the path extraction pipeline. Figure 6B, the active 3D path merge 652 and the passive 3D path merge 654 provide an integrated active 3D path and an integrated passive 3D path as inputs to the 3D path diversity check 660. The 3D path diversity check 660 compares the previous fused path (generated in the previous time instant, corresponding to a time window of length p) with the current path (generated by the sensor pipeline and assumed to include additional path data, for example, path data at a long distance from the vehicle). The 3D path diversity check 660 creates a buffer along the previous fused path and overlaps the buffer with the vertices of the current path. The buffer tolerance interval at a point is defined relative to a predetermined deviation and relative to an expected error in the passive 2D sensor path from the passive 2D sensor path extractors 612, 614, respectively, and the active 3D sensor path from the active 3D sensor path extractors 632, 633, respectively. The ratio in the buffer is calculated based on the vertices in the previous fused path and the vertices of the current path, and based on the ratio being above a predetermined value, it is determined in the 3D path diversity check 660 whether the current path passes the diversity check.

[0150] Fusing at least one passive 3D path and at least one active 3D sensor path to generate a merged 3D path S734. Figure 6B , the functionality of the diverse sensor 3D path merging ("path fusion") 670 is similar to the 3D path similarity sensor merging 650, including active 3D path merging 652 for active path merging and passive 3D path merging 654 for passive path merging. The diverse sensor 3D path merging ("path fusion") 670 can be configured for performance (i.e., outputting the union of two paths within a tolerance range) or completeness (i.e., outputting the intersection of two paths within a tolerance range). This functionality allows for flexible adjustment based on use cases and / or sensor selection. When the 3D path diversity check 660 succeeds, the diverse sensor 3D path merging ("path fusion") 670 merges the previous path inertial measurement value z -p…-1672 is integrated with the current path. The vertices of the current path in the buffer mentioned in the 3D path diversity check 660 are collected and the spline parameters are calculated using these vertices. Note that the diverse sensor 3D path merging ("path fusion") 670 can be configured for performance (i.e., the union of two paths or parts of two paths can be used; i.e., in the case where the current path is significantly shorter than the previous path, some vertices of the previous path can be used), or it can be configured for completeness (i.e., the intersection of the two paths). Once the new spline parameters are estimated, the vertices are sampled at regular intervals to generate the final accurate path. The diverse sensor 3D path merging ("path fusion") 670 does not rely on the ordered output from the sensor pipeline as input. The diverse sensor 3D path merging ("path fusion") 670 maintains the latest fused path and integrates the latest fused path with the new input. The diverse sensor 3D path merging ("path fusion") 670 outputs a highly accurate and highly complete fused path.

[0151] After the merged 3D path is generated, a third supervisory check S738 including a 3D path rationality check is performed in the path extraction pipeline. Figure 6B , the diverse sensor 3D path merging ("path fusion") 670 provides the fused 3D path to the 3D path rationality check 680. The 3D path rationality check 680 also receives constraints 682. The 3D path rationality check 680 subdivides the current path (the look-ahead path) into sections (of configurable size) and confirms (where possible, based on available information from the network) the following 3D path rationality parameters (in order of priority):

[0152] The path is smooth, without kinks or interruptions;

[0153] The path has the correct track gauge;

[0154] The path has an acceptable radius of curvature (ROC) (larger than the expected minimum);

[0155] The path has an acceptable slope (within the expected range);

[0156] The path has acceptable slopes and depressions (within expected ranges);

[0157] The path has an acceptable track transition curve (within expected guidelines); and

[0158] • The path has acceptable superelevation / crossing levels (within expected ranges), e.g. as defined by the standard.

[0159] Depending on the processing power and performance of the onboard sensors, one, more, or all of these checks may be performed (with higher levels of checks enabled by greater available power and higher performing sensors).

[0160] After the 3D path rationality check, a fourth supervised check including the 3D path history diversity check is performed in the path extraction pipeline. Figure 6B , 3D path rationality check 680 provides input to 3D path history diversity check 684. 3D path history diversity check 684 compares the past traversed paths (from a past window of size k) with past inertial measurements z according to input G from inertial sensor input 645. -k...-1 and z in a window of size k -k...-1 686. In the comparison of the two 3D paths, the 3D path history diversity check 684 is performed similarly to the 3D path diversity check 660, with a predetermined deviation at the point where IMU noise is taken into account. In addition, the 3D path history diversity check 684 takes into account the branching points that have been passed by comparing the two paths, and then uses this information to determine the branch that has been passed (i.e., the branch that corresponds to a successful diversity check). The 3D path history diversity check 684 confirms that the past path that has been passed can be trusted and matches at least one of the fused 3D paths 694 extracted in the past after the current vehicle position. The 3D path history diversity check 684 also identifies (at a branch) which of the potential two paths is the past path that has been passed (and therefore is the correct one).

[0161] After the 3D path history diversity check, a fifth supervised check including 3D path consistency check is performed in the path extraction pipeline. Figure 6B , 3D path history diversity check 684 provides input to 3D path consistency check 688. 3D path consistency check 688 also receives input from constraints 690 and past inertial measurements z -m…-1 692 Input. 3D path consistency check 688 using past path inertial measurements z -m…-1 692 and the current (forward) path to verify that there are no continuity breaks or "kinks" and to confirm that the checks outlined in the 3D path rationality check 680 are maintained on the historical and current path information.

[0162] Then, the process ends S750.

[0163] In at least one embodiment, a path extraction method for a vehicle on a guideway includes: receiving two or more sensor inputs from two or more sensors, the two or more sensor inputs including at least one active 3D sensor input from at least one active 3D sensor and at least one passive 2D sensor input from at least one passive 2D sensor; extracting at least one active 3D sensor path based on the at least one active 3D sensor input and extracting at least one passive 2D sensor path based on the at least one passive 2D sensor input; generating at least one 3D sensor ground surface model based on the at least one passive 2D sensor path; generating at least one passive 3D path based on the at least one passive 2D sensor path and the at least one 3D sensor ground surface model; fusing at least one passive 3D path and at least one active 3D sensor path to produce a merged 3D path; and performing at least one supervisory check in the path extraction pipeline to provide integrity of the merged 3D path.

[0164] Figure 8 is a schematic diagram of a system 800 for performing path extraction according to at least one embodiment.

[0165] exist Figure 8 8, system 800 includes a hardware processor 802 and a non-transitory computer-readable storage medium 804, which is encoded with (i.e., stored with) a computer program code 806 (i.e., a set of executable instructions). The processor 802 is electrically coupled to the computer-readable storage medium 804 via a bus 808. The processor 802 is also electrically coupled to an I / O interface 810 via the bus 808. A network interface 812 is also electrically connected to the processor 802 via the bus 808. The network interface 812 is connected to a network 814, so that the processor 802 and the computer-readable storage medium 804 can be connected to external elements via the network 814. The processor 802 is configured to execute the computer program code 806 encoded in the computer-readable storage medium 804, so that the system 800 can be used to perform some or all of the processes or methods according to one or more embodiments described above.

[0166] In some embodiments, processor 802 is a central processing unit (CPU), a multiprocessor, a distributed processing system, an application specific integrated circuit (ASIC), and / or a suitable processing unit.

[0167] In some embodiments, the computer-readable storage medium 804 is an electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor system (or device or apparatus). For example, the computer-readable storage medium 804 includes semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), rigid disk, and / or optical disk. In some embodiments using optical disks, the computer-readable storage medium 804 includes a compact disk read-only memory (CD-ROM), a compact disk read / write (CD-R / W), and / or a digital video disk (DVD).

[0168] In some embodiments, the storage medium 804 stores computer program code 806, which is configured to cause the system 800 to perform a method as described herein. In some embodiments, the storage medium 804 also stores information for performing the method and information generated during the performance of the method, such as data and / or parameters and / or information 816 and / or a set of executable instructions 806 to perform a process or method according to one or more embodiments described above.

[0169] System 800 includes an I / O interface 810. I / O interface 810 is coupled to external circuits. In some embodiments, I / O interface 810 includes a keyboard, a keypad, a mouse, a trackball, a touchpad, and / or cursor direction keys for transmitting information and commands to processor 802.

[0170] The system 800 also includes a network interface 812 coupled to the processor 802. The network interface 812 allows the system 800 to communicate with a network 814, which connects one or more other computer systems. The network interface 812 includes a wireless network interface such as BLUETOOTH, WIFI, WIMAX, GPRS, or WCDMA; or a wired network interface such as ETHERNET, USB, or IEEE-1394. In some embodiments, the method is implemented in two or more systems 800, and information is exchanged between different systems 800 via the network 814.

[0171] System 800 is configured to receive information through I / O interface 810. This information is transferred to processor 802 via bus 808.

[0172] Thus, in at least one embodiment, the processor 802 executes instructions 806 stored on one or more non-transitory computer-readable storage media 804 to receive two or more sensor inputs from two or more sensors, the two or more sensor inputs including at least one active 3D sensor input from at least one active 3D sensor and at least one passive 2D sensor input from at least one passive 2D sensor. The processor 802 extracts at least one active 3D sensor path based on the at least one active 3D sensor input, and extracts at least one passive 2D sensor path based on the at least one passive 2D sensor input. The processor 802 generates at least one 3D sensor ground surface model based on the at least one passive 2D sensor path. The processor 802 generates at least one passive 3D path based on the at least one passive 2D sensor path and the at least one 3D sensor ground surface model. The processor 802 performs a first supervisory check including at least one active 3D path similarity sensor consistency check for at least one active 3D sensor path and at least one passive 3D path similarity sensor consistency check for at least one passive 3D path to verify the integrity of at least one active 3D sensor path and at least one passive 3D path. The processor 802 performs a second supervisory check in the path extraction pipeline, including a 3D path diversity check. The processor 802 fuses at least one passive 3D path and at least one active 3D sensor path to produce a merged 3D path. After generating the merged 3D path, the processor 802 performs a third supervisory check in the path extraction pipeline, including a 3D path rationality check. After the 3D path rationality check, the processor 802 performs a fourth supervisory check in the path extraction pipeline, including a 3D path history diversity check. After the 3D path history diversity check, the processor 802 performs a fifth supervisory check in the path extraction pipeline, including a 3D path consistency check.

[0173] Thus, at least one embodiment provides high integrity, accurate train paths and turnout locations using multi-sensor data. In at least one embodiment, the problem domain is constrained to determining track segments from a single sensor pipeline. In at least one embodiment, track detection is performed over several frames. In at least one embodiment, one or more supervisions (and fusions) are implemented to mitigate the risk of sensor failures and algorithmic errors, thereby providing high integrity of the estimated path.

[0174] The embodiments described herein include advantages over other path extraction methods by providing a multi-sensor approach for estimating a path that provides robustness by compensating for the shortcomings of different sensors. The embodiments described herein enable a robust detection path regardless of varying weather and lighting conditions. In addition, at least one embodiment increases the accuracy of the extracted path by incorporating a path estimate that incorporates a fused path that takes into account multi-sensor calibration transformations. At least one embodiment also provides a fully automated process that uses low-resolution COTS sensors to determine the train path instead of high-performance sensors used during measurements.

[0175] Other path extraction methods rely on offline processing. When all point data in the scene are collected, one or more embodiments of the method can extract tracks by initializing the starting points of the tracks, clustering, and modeling the tracks. Track extraction in other methods relies on traditional computer vision techniques that involve temporarily setting parameters depending on a given scene. Other path extraction methods are susceptible to scene changes, including changes in venue, lighting, and weather, and are therefore not suitable for real-world situations where scenes change frequently. In addition, tracks in the real world include several types of complex paths, such as merging paths and forked paths. However, other path extraction methods operate in limited situations, for example, single-track situations.

[0176] Complementary multi-sensor path extraction (CMPE) according to at least one embodiment can use low-density lidar data and be used to provide a ground surface model, which is used to recover a 3D path from a 2D path generated by passive 2D data. Online path extraction is used to gradually extract railway tracks including those close to and far from the sensor using a Kalman filter. Unlike other methods, the embodiments described herein take into account the scanning pattern of the railway track and its adjacent ground observed in the low-density lidar sensor data, and detect the track head area instead of the track head point. During the update process of the Kalman filter, the center of the detected track head area is used as an observation value to extract the track path using a low-density lidar, where the track head point is not observed. In addition, the CMPE according to at least one embodiment can extract ground points using a slope-constrained grid-based ground filtering method and generate a surface model, assuming that the ground can be modeled as a plane in a local area (in a loose constraint method). The generated surface model is used to recover the 3D information of the 2D path generated by the passive 2D data. Therefore, the 3D path generated by the active 3D sensor and the passive 2D sensor can be directly fused in the fusion pipeline, which simplifies the fusion process and supports key partitions in high-criticality applications. Consideration of the partitioning of mixed SIL components is governed by CENELEC standards (eg, CENELEC EN 50126 and 50129).

[0177] Redundant Multi-Sensor Path Extraction (RMPE) according to at least one embodiment includes additional similar sensor consistency and merging functionality, which allows future implementations to use multiple similar sensor types to maximize performance and / or integrity. To accommodate the use of multiple similar sensor types, merging can be configured to maximize performance or integrity by using the union or intersection (respectively) of valid waypoints (where valid means that a particular waypoint has passed the similar sensor consistency check).

[0178] The features of several embodiments are summarized above so that those skilled in the art can better understand the aspects of the embodiments described herein. It should be understood by those skilled in the art that the embodiments disclosed herein can be used as a basis for designing or modifying other processes and structures to achieve the same purposes and / or obtain the same advantages of the embodiments described herein. It should also be recognized by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the embodiments described herein, and various changes, substitutions and modifications can be made herein without departing from the spirit and scope of the embodiments described herein.

Claims

1. A path extraction method for a vehicle on a guide rail, comprising: receiving two or more sensor inputs from two or more sensors, the two or more sensor inputs comprising at least one active 3D sensor input from at least one active 3D sensor and at least one passive 2D sensor input from at least one passive 2D sensor; extracting at least one active 3D sensor path based on the at least one active 3D sensor input and extracting at least one passive 2D sensor path based on the at least one passive 2D sensor input; generating at least one 3D sensor ground surface model based on the at least one passive 2D sensor path; generating at least one passive 3D path based on the at least one passive 2D sensor path and the at least one 3D sensor ground surface model; fusing the at least one passive 3D path and the at least one active 3D sensor path to produce a combined 3D path; as well as At least one supervisory check is performed in the path extraction pipeline to provide completeness to the merged 3D path.

2. The method according to claim 1, wherein: The performing of the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: performing a 3D path diversity check on the path extraction pipeline before fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, the 3D path diversity check comprising: receiving the at least one active 3D sensor path and the at least one passive 3D path; performing the 3D path diversity check based on the at least one active 3D sensor path and the at least one passive 3D path to generate a 3D path diversity check status indicating one of a success or a failure of the 3D path diversity check; and The merged 3D path is generated based on the 3D path diversity check status indicating one of a success or a failure of the 3D path diversity check.

3. The method according to claim 1, wherein: The performing of the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: after the fusing of the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing a 3D path rationality check in the path extraction pipeline, the 3D path rationality check comprising: receiving the merged 3D path; and The 3D path rationality check is performed on the merged 3D path by determining whether the merged 3D path satisfies one or more 3D path rationality parameters.

4. The method according to claim 3, wherein: Determining whether the merged 3D path satisfies the one or more 3D path rationality parameters includes determining whether the merged 3D path satisfies at least one of the following: being smooth, having no kinks or interruptions, having a correct track gauge, having an acceptable radius of curvature (ROC), having an acceptable slope within a predetermined slope range, having acceptable slopes and depressions within a predetermined slope and depression range, having an acceptable track transition curve, or having an acceptable superelevation / crossing level within a predetermined superelevation / crossing level range.

5. The method according to claim 1, wherein: The performing at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: after fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing a 3D path history diversity check in the path extraction pipeline, the 3D path history diversity check comprising: Receive past paths from a past window of size k; Receive past inertial measurements in a window of size k; determining a credibility of the past traversed path based on the past traversed path from a past window of size k and the past inertial measurements in the window of size k; and Based on the confidence level, it is determined whether the past path matches the merged 3D path.

6. The method according to claim 1, wherein: The performing of the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: after fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing a 3D path consistency check in the path extraction pipeline, the 3D path consistency check comprising: receiving the merged 3D path; receiving the path traversed in the past; and Verify that the merged 3D path does not include continuity breaks.

7. The method according to claim 1, wherein: The performing of the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: after the fusing of the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing at least one active 3D path similarity sensor consistency check on the at least one active 3D sensor path and checking whether the at least one active 3D sensor path is within a predetermined active 3D sensor path tolerance, and performing at least one passive 3D path similarity sensor consistency check on the at least one passive 3D path and checking whether the at least one passive 3D path is within a predetermined passive 3D path tolerance.

8. A path extractor comprising: a memory storing computer-readable instructions; as well as a processor coupled to the memory, wherein the processor is configured to execute the computer-readable instructions to: receiving two or more sensor inputs from two or more sensors, the two or more sensor inputs comprising at least one active 3D sensor input from at least one active 3D sensor and at least one passive 2D sensor input from at least one passive 2D sensor; extracting at least one active 3D sensor path based on the at least one active 3D sensor input and extracting at least one passive 2D sensor path based on the at least one passive 2D sensor input; generating at least one 3D sensor ground surface model based on the at least one passive 2D sensor path; generating at least one passive 3D path based on the at least one passive 2D sensor path and the at least one 3D sensor ground surface model; fusing the at least one passive 3D path and the at least one active 3D sensor path to produce a combined 3D path; and At least one supervisory check is performed in the path extraction pipeline to provide completeness to the merged 3D path.

9. The path extractor according to claim 8, wherein: The processor is further configured to perform the at least one supervisory check in the path extraction pipeline to provide completeness to the merged 3D path by: Prior to fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing a 3D path diversity check in the path extraction pipeline, wherein the processor performs the 3D path diversity check by: receiving the at least one active 3D sensor path and the at least one passive 3D path; performing the 3D path diversity check based on the at least one active 3D sensor path and the at least one passive 3D path to generate a 3D path diversity check status indicating one of a success or a failure of the 3D path diversity check; and The merged 3D path is generated based on the 3D path diversity check status indicating one of a success or a failure of the 3D path diversity check.

10. The path extractor according to claim 8, wherein: The processor is further configured to perform the at least one supervisory check in the path extraction pipeline to provide completeness to the merged 3D path by: After fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing a 3D path rationality check in the path extraction pipeline, wherein the processor performs the 3D path rationality check by: receiving the merged 3D path; and The 3D path rationality check is performed on the merged 3D path by determining whether the merged 3D path satisfies one or more 3D path rationality parameters.

11. The path extractor according to claim 10, wherein: The processor is also configured to determine whether the merged 3D path satisfies the one or more 3D path rationality parameters by determining whether the merged 3D path satisfies at least one of the following conditions: being smooth, having no kink interruptions, having a correct track gauge, having an acceptable radius of curvature (ROC), having an acceptable grade within a predetermined grade range, having acceptable slopes and depressions within a predetermined slope and depression range, having an acceptable track transition curve, or having an acceptable superelevation / crossing level within a predetermined superelevation / crossing level range.

12. The path extractor according to claim 8, wherein: The processor is further configured to perform the at least one supervisory check in the path extraction pipeline to provide completeness to the merged 3D path by: After fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing a 3D path history diversity check in the path extraction pipeline, the processor performing the 3D path history diversity check by: Receive past paths from a past window of size k; receiving past inertial measurements in a past window of size k; determining a credibility of the past traversed path based on the past traversed path from a past window of size k and the past inertial measurements in the past window of size k; as well as Based on the confidence level, it is determined whether the past path matches the merged 3D path.

13. The path extractor according to claim 8, wherein: The processor is further configured to perform the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path, the at least one supervisory check comprising: performing a 3D path consistency check in the path extraction pipeline after fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, wherein the processor performs the 3D path consistency check by: receiving the merged 3D path; receiving the path traversed in the past; and Verify that the merged 3D path does not include continuity breaks.

14. The path extractor according to claim 8, wherein: The processor is further configured to, after fusing the at least one passive 3D path and the at least one active 3D sensor path to produce the merged 3D path, perform the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path by: performing at least one active 3D path similarity sensor consistency check on the at least one active 3D sensor path to evaluate similarity and whether the at least one active 3D sensor path is within a predetermined active 3D sensor path tolerance, and At least one passive 3D path similarity sensor consistency check is performed on the at least one passive 3D path and checks whether the at least one passive 3D path is within a predetermined passive 3D path tolerance.

15. A non-transitory computer-readable medium having computer-readable instructions stored thereon, the computer-readable instructions, when executed by a processor, causing the processor to perform operations comprising: receiving two or more sensor inputs from two or more sensors, the two or more sensor inputs comprising at least one active 3D sensor input from at least one active 3D sensor and at least one passive 2D sensor input from at least one passive 2D sensor; extracting at least one active 3D sensor path based on the at least one active 3D sensor input and extracting at least one passive 2D sensor path based on the at least one passive 2D sensor input; generating at least one 3D sensor ground surface model based on the at least one passive 2D sensor path; generating at least one passive 3D path based on the at least one passive 2D sensor path and the at least one 3D sensor ground surface model; fusing the at least one passive 3D path and the at least one active 3D sensor path to produce a combined 3D path; as well as At least one supervisory check is performed in the path extraction pipeline to provide completeness to the merged 3D path.

16. The non-transitory computer readable medium of claim 15, wherein: The performing of the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: performing a 3D path diversity check on the path extraction pipeline before fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, the 3D path diversity check comprising: receiving the at least one active 3D sensor path and the at least one passive 3D path; performing the 3D path diversity check based on the at least one active 3D sensor path and the at least one passive 3D path to generate a 3D path diversity check status indicating one of a success or a failure of the 3D path diversity check; and The merged 3D path is generated based on the 3D path diversity check status indicating one of a success or a failure of the 3D path diversity check.

17. The non-transitory computer readable medium of claim 15, wherein: The performing of the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: after the fusing of the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing a 3D path rationality check in the path extraction pipeline, the 3D path rationality check comprising: receiving the merged 3D path; and performing the 3D path rationality check on the merged 3D path by determining whether the merged 3D path satisfies one or more 3D path rationality parameters, Wherein, determining whether the merged 3D path satisfies the one or more 3D path rationality parameters includes determining whether the merged 3D path satisfies at least one of the following: being smooth, having no kinks or interruptions, having a correct track gauge, having an acceptable radius of curvature (ROC), having an acceptable slope within a predetermined slope range, having acceptable slopes and depressions within a predetermined slope and depression range, having an acceptable track transition curve, or having an acceptable superelevation / crossing level within a predetermined superelevation / crossing level range.

18. The non-transitory computer readable medium of claim 15, wherein: The performing of the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: performing a 3D path history diversity check in the path extraction pipeline after fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, the 3D path history diversity check comprising: Receive past paths from a past window of size k; receiving past inertial measurements in a past window of size k; determining a credibility of the past traversed path based on the past traversed path from a past window of size k and the past inertial measurements in the past window of size k; and Based on the confidence level, it is determined whether the past path matches the merged 3D path.

19. The non-transitory computer readable medium of claim 15, wherein: The performing of the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: after fusing the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing a 3D path consistency check in the path extraction pipeline, the 3D path consistency check comprising: receiving the merged 3D path; receiving the path traversed in the past; and Verify that the merged 3D path does not include continuity breaks.

20. The non-transitory computer readable medium of claim 15, wherein: The performing of the at least one supervisory check in the path extraction pipeline to provide integrity to the merged 3D path comprises: after the fusing of the at least one passive 3D path and the at least one active 3D sensor path to generate the merged 3D path, performing at least one active 3D path similarity sensor consistency check on the at least one active 3D sensor path to evaluate similarity and whether the at least one active 3D sensor path is within a predetermined active 3D sensor path tolerance, and performing at least one passive 3D path similarity sensor consistency check on the at least one passive 3D path and checking whether the at least one passive 3D path is within a predetermined passive 3D path tolerance.

Images