Dynamic time-to-collision determination of a rail vehicle

Abstract

The invention provides, amongst other aspects, a method for dynamically determining a time-to- collision, TTC, related quantity of a rail vehicle with an object, the rail vehicle comprising an imaging sensor and a laser-based ranging sensor, the method comprising obtaining image data from the imaging sensor and point cloud data from the laser-based ranging sensor; detecting the object in the image data and the point cloud data based on first training data relating to a plurality of objects; extracting information from the image data and the point cloud data, wherein the step of extracting comprises determining a rail track condition in the image data and the point cloud data based on second training data relating to a plurality of rail track conditions; and determining the TTC-related quantity of the rail vehicle with the object based on the extracted information from the image data and the point cloud data.

Description

Dynamic time-to-collision determination of a rail vehicleField of the invention

[0001] The present invention relates to the technical domain of rail transportation and more particularly to techniques for dynamically determining time-to-collision of a rail vehicle with an object.Background art

[0002] The increasing complexity of modern rail systems, including higher speeds and more frequent freight operations, amplifies the need for advanced safety measures to prevent accidents. One such risk is the potential for collisions between a rail vehicle and an object on the rail track, which can have devastating consequences.

[0003] Existing solutions for detecting potential collisions in rail transportation often rely on fixed sensors, such as track-mounted cameras or radar systems, which can be limited in their ability to detect obstacles at varying distances or angles. While some prior art has explored the use of computer vision technology for monitoring rail tracks and detecting potential hazards, these solutions may not provide accurate and timely information about collisions.

[0004] Other solutions, such as US20100268466A1 and DE102007009771 A1 , focus on detecting potential collisions between two rail vehicles. These systems typically rely on sensors mounted on the leading vehicle to detect the presence of another vehicle ahead and provide warnings or initiate braking. While these solutions can mitigate some risks, they do not address the more complex challenge of detecting and responding to collisions with objects on the track, such as trespassers, wildlife, or other external obstacles. The problem is further compounded by the dynamic nature of rail track conditions, which can significantly impact the braking distance of a train. Thus, current solutions do not account for unexpected hazards, thereby, leaving operators without an accurate and dynamic solution to prevent collisions.

[0005] These challenges highlight the need for improved methods that can overcome these limitations.

[0006] The present invention aims at addressing issues, such as the issues mentioned above.Summary of the invention

[0007] According to a first aspect, the present invention provides a method for dynamically determining a time-to-collision, TTC, related quantity of a rail vehicle with an object, the rail vehicle comprising an imaging sensor and a laser-based ranging sensor, the method comprising:- obtaining image data from the imaging sensor and / or point cloud data from the laser-based ranging sensor;- detecting the object in the image data and / or the point cloud data based on first training data relating to a plurality of objects;- extracting information from the image data and / or the point cloud data, wherein the step of extracting comprises determining a rail track condition in the image data and / or the point cloud data based on second training data relating to a plurality of rail track conditions; and- determining the TTC-related quantity of the rail vehicle with the object based on the extracted information from the image data and / or the point cloud data.

[0008] While the prior art may detect objects on the track or provide basic collision avoidance warnings, without taking into any dynamic changes in rail track conditions, the method according to the present invention may advantageously allow to extract relevant information about rail track conditions, preferably friction-related conditions, such as water, sand, leaf mulch, debris, etc., and uses this information to more effectively determine the TTC-related quantity. By accounting for these changing conditions, the method can better anticipate potential collisions and provide improved and timely warnings and / or more effectively and efficiently initiate braking to prevent accidents. This invention has the potential to significantly improve the safety of rail transportation by providing a more comprehensive and dynamic solution for collision avoidance.

[0009] According to a second aspect, the present invention provides a device comprising means for carrying out the method according to the invention.

[0010] According to a third aspect, the present invention provides a system comprising the device according to the invention and at least one of an imaging sensor and laser-based ranging sensor, wherein the device is configured to perform the steps of:- obtaining image data from the imaging sensor and / or point cloud data from the laser-based ranging sensor;- detecting the object in the image data and / or the point cloud data based on first training data relating to a plurality of objects;- extracting information from the image data and / or the point cloud data, wherein the step of extracting comprises determining a rail track condition in the image data and / or the point cloud data based on second training data relating to a plurality of rail track conditions; and- determining the TTC-related quantity of the rail vehicle with the object based on the extracted information from the image data and / or the point cloud data.

[0011] Such a system may advantageously provide a complete solution to rail vehicles which do not already have an imaging sensor and / or a laser-based ranging sensor. Furthermore, the system may provide a more portable solution and may further provide an edge computing solution.

[0012] According to a fourth aspect, the present invention provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the invention. The computer program product may comprise at least one readable medium in which computer-readable program code portions are saved, which program code portions comprise instructions for carrying out said method.

[0013] According to a fifth aspect, the present invention provides a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the invention.

[0014] Preferred embodiments and their advantages are provided in the description and the dependent claims.Brief description of the drawings

[0015] The present invention will be discussed in more detail below, with reference to the attached drawings.

[0016] Fig. 1-5 show example embodiments of a method according to the invention.Description of embodiments

[0017] The following descriptions depict only example embodiments and are not considered limiting in scope. Any reference herein to the disclosure is not intended to restrict or limit the disclosure to exact features of any one or more of the exemplary embodiments disclosed in the present specification.

[0018] Furthermore, the terms first, second, third and the like in the description and in the claims are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. The terms are interchangeable under appropriate circumstances and the embodiments of the invention can operate in other sequences than described or illustrated herein.

[0019] Furthermore, the various embodiments, although referred to as “preferred” are to be construed as exemplary manners in which the invention may be implemented rather than as limiting the scope of the invention.

[0020] The term “comprising”, used in the claims, should not be interpreted as being restricted to the elements or steps listed thereafter; it does not exclude other elements or steps. It needs to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Thus, the scope of the expression “a device comprising A and B” should not be limited to devices consisting only of components A and B, rather with respect to the present invention, the only enumerated components of the device are A and B, and further the claim should be interpreted as including equivalents of those components.

[0021] In embodiments, the method according to the present invention is a computer-implemented method.

[0022] The term “dynamically”, such as dynamically determining, may refer to a process where an outcome or result is not predetermined, but rather determined as a response to changing circumstances. This may involve continuously monitoring and / or analyzing data. Dynamic determination may involve redetermining, such as adjusting (e.g., continuous learning and updating) and generating outputs that reflect the evolving situation. Dynamically determining may refer to real-time determining, such as within a time constraint measured in milliseconds, e.g., less than 500 ms, less than 250 ms, less than 100 ms, less than 50 ms, less than 25 ms, less than 20 ms, etc.

[0023] A "time-to-collision (TTC)-related quantity" may refer to a value which estimates the remaining time until a collision or near-collision event occurs, based on information extracted from sensor data. The TTC- related quantity may be a value that provides a measure of time, typically expressed in units such as seconds or milliseconds. This value may represent an estimated remaining time until a collision or near- collision event occurs, taking into account factors like object distances, speeds, and acceleration rates. The goal of this calculation is to provide an early warning system for potential collisions, allowing for timely interventions to prevent accidents. The TTC-relates quantity may be referred to as the TTC.

[0024] The term “rail vehicle” may refer to any self-propelled or unpowered entity that operates on rails, including a locomotive, a freight car (e.g., hopper, boxcar, tank car, flatcar, gondola car), a passenger vehicle (e.g., commuter train, high-speed train, subway / metro, tram, etc.), maintenance and inspection vehicles (e.g., cranes, graders, backhoes, etc.), etc.

[0025] The term “object” may refer to an entity or substance which is not part of the rail track, which may exclude rails, ties ballast, and other structural elements. Examples of objects include, a person, a bicycle, a (rail) vehicle, an animal, equipment, machinery, etc. In embodiments, the step of detecting the object comprises detecting the object in at least one, preferably both, of the image data and the point cloud data based on first training data relating to a plurality objects. In embodiments, the step of detecting the object (further) comprises detecting the object in at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data based on first training data relating to a plurality objects. In embodiments, the step of detecting the object comprises detecting the object in the fused dataset based on first training data relating to a plurality objects. The first training data may comprise historically obtained data (e.g., image data and / or point cloud data, electromagnetic wave data, echo data, etc.) from the respective sensor(s), which are labelled as featuring or including one or more objects. The step of detecting an object may comprise applying a machine learning, ML, model which has been trained on the first training data. The first ML model may comprise a regression model or a classification model. The machine learning model may comprise a deep learning, DL, model (e.g., neural networks, NN, convolutional NN, CNN, region-based CNN, etc.) or a classic ML model (e.g., support vector machine classification, logistic regression, etc.), or a hybrid of one or more DL models and / or one or more classic ML models.

[0026] The term “imaging sensor’ may refer to a sensor or sensor module configured for capturing (and optionally processing) image data, which may comprise (a sequence of individual) image frames. The imaging sensor may be capable of detecting various types of electromagnetic radiation, such as visible light, infrared, ultraviolet or other forms of radiation. In embodiments, the image data comprise image frames, and optionally metadata relating to the image frames. In embodiments, the image data consist of image frames. In embodiments, the image frames comprise respective metadata. In embodiments, the image frames consist of image pixels. In embodiments, the imaging sensor comprises at least one imaging sensor, preferably two or more imaging sensors. In embodiments, respective image data obtained from the two or more imaging sensors are processed to create a single set of imaging data, such as merged imaging data.

[0027] The term “laser-based ranging sensor” may refer to a sensor or sensor module which uses a laser source to transmit and detect electromagnetic radiation. The laser-based ranging sensor may be configured to emit modulated or pulsed laser light which interacts with objects int eh environment and returns as reflected signals. The laser-based ranging sensor may process these reflected signals to calculate the time- of-flight, ToF, or phase-shift, PS, between transmitted and received laser pulses. The resulting data can be used to create a three-dimensional representation of the environment or object being scanned. In embodiments, the laser-based ranging sensor is configured for capturing or scanning an environment and optionally provide a point cloud of the captured / scanned environment. The point cloud may be generated by the laser-based ranging sensor. In embodiments, the laser-based ranging sensor comprise one or more light detection and ranging, LiDAR, sensors. The one or more LiDAR sensors may comprise one or more of: solid-state LiDAR sensor(s) (e.g., flash LiDAR(s)), quasi solid-state LiDAR(s), mechanical or non solid- state LiDAR(s), scanning LiDAR(s), etc.

[0028] The term “rail track condition” may refer to a state or characteristics of a rail track, including its (top) surface. The rail track may refer to a portion of the rail track within a field-of-view of the sensor module orthe any one or combination of the sensors. The rail track condition can have significant impact on the braking distance of the rail vehicle travelling on the rail track, by increasing the braking distance, making collision avoidance more difficult. In embodiments, the rail track condition comprises a moisture level and / or percentage of the rail track covered by a material and / or the type of material covering the rail track. In embodiments, the determining a rail track condition comprises determining that the moisture level exceeds a predetermined moisture level, such as determining water pooling on the rail tracks when an amount of water is determined to be >1 mm, preferably >1.5 mm, >2 mm or >2.5 mm on the rail track surface or in the ballast and / or determining relative humidity or moisture level >80%, preferably >85%, >90% or >95%. In embodiments, the determining a rail track condition comprises determining that the rail track is covered by a material by more than a predetermined percentage, such as more than 5 %, preferably more than more than more than 10%, or more than 15%, or more than 20%, or more than 25%, or more than 30% of the rail track in the sensor data (e.g., image frame, point cloud, echo data, etc.) is covered by a material. In embodiments, the determining a rail track condition comprises determining that a type of material covering at least partially the rail track, such as snow, ice, leaves, vegetation, dust, sand, etc. The inventors have shown that the combination of determining the type of material and the percentage of rail track covered by said material can drastically improve the determining of the TTC-related quantity to reduce collisions, such as if 5-15% of the rail track is covered by snow or ice can be dangerous as it reduces traction and increases risk of slipping, or 10-30% of the rail track is covered with leaves or vegetation can lead to reduced friction and increased risk of slipping especially on curves or inclines. The inventors have also shown that a combination of determining the type of material, the percentage of rail track covered by said material, and the moisture level can for particular situations allow to better determine the rail track condition, such as high moisture level of >90% and 10-20% of the rail track is covered by sand, dust or leaves, such that when such material is combined with moisture can increase the risk of slipping and collisions. In embodiments, the method comprises dynamically determining a rail track condition, which may comprise dynamically determining at least one of a moisture level, a percentage of the rail track being covered by a material, and a type of said material.

[0029] The term “weather condition” may refer to a specific atmospheric state characterized by certain meteorological parameters, such as temperature, humidity, precipitation, cloudiness, wind (e.g., speed and / or direction), and solar radiation. In embodiments, the step of extracting information comprises determining a weather condition in the sensor data based on third training data relating to a plurality of weather conditions. The weather condition may comprise at least one of: a temperature value, a (relative) humidity value, a precipitation type (e.g., rainfall, snowfall, sleet or freezing rain, etc.), a precipitation intensity (determined in mm or cm), a percentage of cloud cover, a wind speed, a wind direction, a thunderstorm, and a solar radiation (in W / m3). In embodiments, the step of determining the weather condition comprises determining a temperature less than or equal to a temperature threshold, such as less or equal to 5 degrees Celsius, less or equal to 0 degrees Celsius, less or equal to -5 degrees Celsius, less or equal to -10 degrees Celsius, etc. In embodiments, the step of determining the weather condition comprises determining a relative humidity level exceeds a predetermined relative humidity level, such as >80% RH, >85% RH, >90% RH, >95% RH. In embodiments, the step of determining the weather condition comprises determining precipitation exceeding a predetermined precipitation intensity threshold, such asfor rainfall, more than 0.5 mm / h, more than 1 mm / h, more than 2 mm / h, more than 4 mm / h, more than 6 mm / h, more than 8 mm / h, more than 10 mm / h, more than 15 mm / h, more than 20 mm / h, etc., such as for snowfall, more than more than 0.5 cm / h, more than 1 cm / h, more than 2 cm / h, more than 4 cm / h, more than 6 cm / h, more than 8 cm / h, more than 10 cm / h, more than 15 cm / h, more than 20 cm / h, etc., such as for hail, more than 0.5 mm, more than 1 mm, more than 2 mm, more than 4 mm, more than 6 mm, more than 8 mm, more than 10 mm, more than 15 mm, more than 20 mm, etc. In embodiments, the step of determining the weather condition comprises determining wind speeds exceeding a predetermined wind speed threshold, such as more than 10 km / h, more than 20 km / h, more than 30 km / h, more than 40 km / h, more than 50 km / h, more than 60 km / h, more than 70 km / h, more than 80 km / h etc.

[0030] In embodiments, at least one sensor, preferably a plurality of sensors are mounted to the front end of the rail vehicle. In embodiments, the at least one sensor, preferably the plurality of sensors are pointed in the direction of motion of the rail vehicle. In embodiments, the plurality of sensors has at least partially overlapping field-of-views, FoVs. Alternatively, the plurality of sensors has end-to-end FoVs. It is preferred that the total or combined FoV of the plurality of sensors has an angle of at least 120 degrees, preferably at least 150 degrees, more preferably at least 180 degrees, most preferably 210 degrees. At each time point (e.g., time stamp), the image frames captured by a plurality of image sensors may be merged. At each time point (e.g., time stamp), the point clouds captured by a plurality of laser-based ranging sensors may be merged. Said merging of the image frames and / or the point clouds may comprise any one or combination of: Kalman filter-based registration, Random sample consensus, RSC, Mutual informationbased registration, Feature-based alignment, Deep learning-based registration, Iterative closest point, ICP, and other optimization algorithms (e.g., Gradient descent-based registration, Non-linear least squares registration, etc.). The Kalman filter-based registration may relate to uses a Kalman filter to estimate the camera poses (position and orientation) and then registers the images using an affine transformation. The RSC may relate to using a random sample of pixels from each image frame and computes the similarity between the samples. The mutual information-based registration may relate to using mutual information to measure the similarity between the image frames and / or the point clouds and then registers the images and / or the point clouds using a maximum likelihood estimation. The RSC and the mutual information-based registration result in the final merged image and / or the final merged point cloud being based on the most similar pixel pairs and / or similar point pairs, respectively. The feature-based alignment may relate to aligning the images by matching features such as corners, edges, or blobs. Once the images are aligned, they can be combined using an averaging technique. The deep learning-based registration may relate to mapping the image frames and / or point clouds with their corresponding registers using deep learning. The ICP may relate to alternating between finding the best registration transformation and verifying that the resulting image is close to the original image and / or to the original point cloud.

[0031] In embodiments, the method comprises obtaining electromagnetic radio wave data from a radar sensor and / or electromagnetic microwave data from a microwave sensor and / or echo data from an ultrasonic sensor.

[0032] In embodiments, at each time stamp the image data and the point cloud data are fused or merged together, e.g. into a fused dataset. In embodiments, the step of extracting relates to or comprises extracting information from the fused data(set). In embodiments, the image data and / or the point cloud data are fusedwith at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data, e.g. into the fused dataset. The method may combine any two or more sensors to advantageously extract information from the data obtained therefrom to more accurately determine the TTC-related quantity. The inventors have found that laser-based ranging sensors, particularly LiDAR, can be advantageously combined with other sensors, since it does not require external light, so it works both during the day and at night, it may allow engineers to obtain shapes of arbitrary objects without the need for visual cues like camera or object material and shape as much as radar, it may provide reliable estimation of short and long- range distances, and changes in distance and direction over time. This can help identify collisions earlier than pure vision technology would and can allow engineers to distinguish targets of interest from noise due to the ability to distinguish height, a key distinction from classical 2D radars. The laser-based ranging sensors, particularly LiDAR, may offer high-definition 3D data, at a rate of millions of points per second, optionally with detection ranges of up to and beyond 100 meters, and optionally a position error of less than any one of: less than 20 cm, 10 cm, less than 8 cm, less than 6 cm, less than 4 cm, less than 2 cm, etc. This can enable the creation of a real time, precise, 3D depiction of the surrounding environment. For example, by combining data from a laser-based ranging sensor with at least an image sensor, the inventors have shown that false positives, such as false detections of objects and / or false determination of weather conditions and / or false determination of rail conditions, are reduced, and longer and wider range detections can be achieved. For example, by combining an imaging sensor, which can provide high-definition images at a low cost but may require light or may be sensitive to glare, with a laser-based ranging sensor, which can provide (3D) spatial coverage while having a longer detection range but may be sensitive to bad weather and may suffer from LiDAR ghosts and LiDAR blooming, and with a radar sensor, which works well in all weather conditions but may have low spatial resolution, the effects of their combined strengths may outweigh their respective weaknesses , and thereby improve the determining of TTC-related quantity.

[0033] In embodiments, sensor data is obtained from any one or combination of the imaging sensor, the laser-based ranging sensor, the radar sensor, the microwave sensor, and the ultrasonic sensor. The sensor data may comprise any one or combination of the image data, the point cloud data, the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data.

[0034] In embodiments, the step of extracting information comprises determining a rail track condition in at least one, preferably both, of the image data and point cloud data based on second training data relating to a plurality of rail track conditions. In embodiments, the step of extracting information (further) comprises determining a rail track condition in at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data based on second training data relating to a plurality of rail track conditions. In embodiments, the step of extracting information comprises determining a rail track condition in the fused dataset based on second training data relating to a plurality rail track conditions. The second training data may comprise historically obtained data (e.g., image data and / or point cloud data, electromagnetic wave data, etc.) from the respective sensor(s), which are labelled as featuring or including one or more rail track conditions. In embodiments, the method comprises dynamically determining a rail track condition in any one or combination of the sensor data. The step of determining a rail track condition may comprise applying a second machine learning, ML, model which has been trained on the second training data. The second ML model may comprise a regression model or a classification model. Themachine learning model may comprise a deep learning, DL, model (e.g., neural networks, NN, convolutional NN, CNN, region-based CNN, etc.) or a classic ML model (e.g., support vector machine classification, logistic regression, etc.), or a hybrid of one or more DL models and / or one or more classic ML models.

[0035] In embodiments, the step of extracting information comprises determining a weather condition in at least one, preferably both, of the image data and point cloud data based on third training data relating to a plurality of weather conditions. In embodiments, the step of extracting information (further) comprises determining a weather condition in at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data based on third training data relating to a plurality of weather conditions. In embodiments, the step of extracting information comprises determining a weather condition in the fused dataset based on third training data relating to a plurality weather conditions. The third training data may comprise historically obtained data (e.g., image data and / or point cloud data, electromagnetic wave data, etc.) from the respective sensor(s), which are labelled as featuring or including one or more weather conditions. In embodiments, the method comprises dynamically determining a weather condition in any one or combination of the sensor data. The step of determining a weather condition may comprise applying a third machine learning, ML, model which has been trained on the third training data. The third ML model may comprise a regression model or a classification model. The machine learning model may comprise a deep learning, DL, model (e.g., neural networks, NN, convolutional NN, CNN, region-based CNN, etc.) or a classic ML model (e.g., support vector machine classification, logistic regression, etc.), or a hybrid of one or more DL models and / or one or more classic ML models.

[0036] In embodiments, the step of determining a rail track condition and / or weather condition are / is based on fourth training data which comprises the second and third training data. The second and third training data may be merged into the fourth training data. The step of determining a rail track condition and / or weather condition may comprise applying the second and / or third ML model which has been trained on the fourth training data.

[0037] In embodiments, the step of determining a rail track condition is based on the determined weather condition. By determining the weather condition and basing the determination of the rail track condition and / or the TTC-related quantity, the method can advantageously allow for even more improved collision avoidance. Particular weather conditions, such as rain or snow, can more accurately allow to determine the rail track condition. Furthermore, a particular weather condition in combination with a particular rail track condition, such as dusty rail track and rain, which can further reduce the friction between the rail track and the rail vehicle wheels. Therefore, by determining both a weather condition and a rail track condition, the TTC-related quantity can be more effectively determined.

[0038] In embodiments, the method comprises determining a target velocity of the rail vehicle for avoiding a collision of the rail vehicle with the object. The target velocity may be (dynamically) determined based on the determined TTC-related quantity. The method therefore advantageously allows for more adaptive braking based on the determined target velocity, thereby, reducing unnecessary sudden or harsh braking.

[0039] In embodiments, the method comprises generating a first alert upon detecting that the TTC-related quantity is lower than a predetermined first time threshold. In embodiments, the first alert comprises at least one of the TTC-related quantity and target velocity of the rail vehicle. In embodiments, the method comprises generating a second alert upon detecting that the TTC-related quantity is lower than apredetermined second time threshold which is lower than the predetermined first time threshold. In embodiments, at least one of the first alert and second alert comprises at least one of the TTC-related quantity and target velocity of the rail vehicle. While some prior art methods detect a risk of collision too late and / or alert a rail vehicle conductor too late, which can cause devastating consequences, other prior art methods alert the conductor with a basic alert too early, requiring the conductor to completely stop the rail vehicle, e.g. until the risk of collision is minimized or prevented. The latter can significantly delay rail vehicles and / or cause harm to passengers and / or freight due to a rapid decrease in velocity of the rail vehicle. The method according to the present invention advantageously provide a more effective and efficient first alert to warn the conductor, thereby allowing the braking to be initiated to more effectively and efficiently reduce the velocity of the rail vehicle to the target value which is determined as an optimal value, preferably different than standstill. The method may advantageously provide a more effective and efficient second alert which instructs the conductor to stop the rail vehicle, e.g., as a backstop, thereby, further improving collision avoidance.

[0040] In embodiments, the method comprises controlling the braking device comprised in the rail vehicle to initiate braking upon detecting that the TTC-related quantity is lower than the predetermined first time threshold. The method may therefore advantageously allow for more reliable control of the braking to reduce the velocity of the rail vehicle to the target value, which is determined as an optimal value, preferably different than standstill.

[0041] In embodiments, the method comprises controlling the braking device comprised in the rail vehicle to initiate braking upon detecting that the TTC-related quantity is lower than a predetermined second time threshold which is lower than the predetermined first time threshold.

[0042] In embodiments, at least one of the first time threshold and second time threshold is determined based on the determined weather condition and / or the determined rail track condition. The at least one of the first time threshold and second time threshold may be dynamically determined based on the determined weather condition and / or the determined rail track condition. The method may therefore advantageously allow for more reliable control of the braking, e.g., as a backstop, preferably to stop the rail vehicle, thereby, further improving collision avoidance.

[0043] In embodiments, the method comprises obtaining metadata relating to or attributed to at least one, preferably both, of the image data and point cloud data, wherein the step of (dynamically) determining the TTC-related quantity is based on the metadata. In embodiments, the at least one, preferably both, of the image data and point cloud data comprises the metadata. In embodiments, the method comprises obtaining metadata (further) relating to or attributed to at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data, wherein the step of (dynamically) determining the TTC-related quantity is based on the metadata. In embodiments, the at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data comprises the metadata. In embodiments, the fused dataset comprises the metadata.

[0044] In embodiments, the metadata comprises time stamps and / or global positioning system, GPS, coordinates related to at least one, preferably both, of the image data and point cloud data. In embodiments, the metadata comprises time stamps and / or global positioning system, GPS, coordinates (further) related to at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echodata. In embodiments, the metadata comprises time stamps and / or global positioning system, GPS, coordinates related to the fused dataset. For instance, each image frame in the image data comprises a respective time stamp. The respective time stamp of an image frame and of a point cloud may be the same, i.e., when the image frame and the point cloud are captured at substantially the same time. Thus, the obtained data (e.g., image data and / or point cloud data, electromagnetic wave data, etc.) captured at substantially the same time may be attributed the same time stamp.

[0045] In embodiments, the metadata is obtained from at least one external module, such as the GPS coordinates are obtained from a GPS module and the time stamps are obtained from a timer or clock. The at least one external module may be external to at least one of the sensors (e.g., the imaging sensor, the laser-based ranging sensor, the radar sensor, the ultrasonic sensor, etc.), such that a common respective time stamp and / or common respective GPS coordinates are / is attributed to the respective data (e.g., image data and / or point cloud data, electromagnetic wave data, echo data, etc.) captured by the respective at least one of the sensors, or such that each of the captured and / or obtained data is attributed a respective time stamp and / or respective GPS coordinates. The external module may be comprised in a sensing module comprising the at least one of the sensors, such that a common respective time stamp and / or common respective GPS coordinates are / is attributed to each of the captured and / or obtained data. The external module may be comprised in at least one of the sensors, such that a respective time stamp and / or respective GPS coordinates are / is attributed to each of the captured and / or obtained data.

[0046] In embodiments, the method comprises obtaining or determining meteorological information based on the metadata, particularly based on the GPS coordinates and the time stamps, such as based on the GPS coordinates relating to the time stamps (e.g., at the time of obtaining at least one of the image data, point cloud data, electromagnetic wave data, echo data, etc.). In embodiments, the method comprises obtaining or determining meteorological information based on a predetermined distance from the GPS coordinates away from the rail vehicle in the direction of the rail vehicle direction of motion, such as at least 1 m away from the rail vehicle in the direction of the rail vehicle direction of motion, preferably at at least 2 m, at least 4 m, at least 5 m, at least 10 m, at least 15 m, at least 20 m, at least 25 m, at least 30 m, at least 35 m, at least 40 m, at least 45 m, at least 50 m, etc. In embodiments, the method comprises dynamically determining meteorological information based on the metadata, particularly based on the GPS coordinates and the time stamps, and optionally based on the predetermined distance from the GPS coordinates.

[0047] In embodiments, the step of determining a weather condition is based on the meteorological information determined based on the GPS coordinates and optionally the time stamps. For example, the weather condition is determined in the image data and the point cloud data based on meteorological information at the GPS coordinates at the time when the image data and the point cloud data are captured / obtained, and thus, in cases where the it is difficult to determine the weather condition only from the image data and the point cloud (e.g., due to difficult light conditions), the meteorological information can help improve determining the weather condition at the GPS coordinates of the sensing module or at at least one of the imaging sensor and the laser-based ranging sensor or at a predetermined distance from the GPS coordinates. In embodiments, the step of dynamically determining a weather condition is based on the meteorological information dynamically determined based on the GPS coordinates and optionally the time stamps.

[0048] In embodiments, the step of determining a rail track condition is based on the meteorological information determined based on the GPS coordinates within a predetermined timeline from a respective time stamp relating / associated to the respective sensor data, such as in the past 24 hours of the time stamps, or in the past 12 hours, in the past 8 hours, in the past 6 hours, in the past 4 hours, in the past 2 hours, in the past 1 hour, in the past 45 minutes, in the past 30 minutes, in the past 15 minutes, in the past 10 minutes, in the past 5 minutes, in the past 1 minute, etc. of the time stamps. In embodiments, the step of determining a rail track condition is based on meteorological information determined based on a predetermined distance from the GPS coordinates away from the rail vehicle in the direction of the rail vehicle direction of motion. In embodiments, the step of determining a rail track condition is based on the determined weather condition which is determined based on meteorological information. For example, at the GPS coordinates related / attributed to the image data and point cloud data, it is determined that it had snowed 24 hours in advance (i.e., 24 hours before time stamps) and that the temperatures in the past 24 hours has been below 0 degrees Celsius, therefore, the rail track condition is determined to be a portion of the rail track covered with black ice which may be invisible to the eye and / or the vision technology, and thereby the predetermined first threshold and / or the predetermined second threshold are / is increased to compensate for the frozen portion of the rail track which would delay the possibility to reduce the velocity of the rail vehicle. In embodiments, the step of dynamically determining a rail track condition is based on the meteorological information dynamically determined based on the GPS coordinates, such as within a predetermined timeline from a respective time stamp relating / associated to the respective sensor data and / or based on a predetermined distance from the GPS coordinates away from the rail vehicle in the direction of the rail vehicle direction of motion.

[0049] In embodiments, the step of extracting comprises determining a distance to the object and / or a relative velocity of the rail vehicle with respect to the object. Therefore, the step of determining the TTC- related quantity is based on the determined distance and the determined relative velocity. In embodiments, the step of extracting comprises dynamically determining a distance to the object and / or a relative velocity of the rail vehicle with respect to the object.

[0050] In embodiments, the step of extracting comprises determining a velocity of the rail vehicle based on the metadata, particularly based on the GPS coordinates and the time stamps. In embodiments, the step of determining a velocity of the rail vehicle comprises determining a travelled distance by the velocity based on the GPS coordinates, such as based on a difference between the (consecutive) GPS coordinates, wherein the velocity of the rail vehicle is based on the determined travelled distance and the time stamps, such as based on a difference between the (consecutive) time stamps. In embodiments, the step of extracting comprises dynamically determining a velocity of the rail vehicle based on the metadata, particularly based on the GPS coordinates and the time stamps. In embodiments, the step of dynamically determining a velocity of the rail vehicle comprises dynamically determining a travelled distance by the velocity based on the GPS coordinates, wherein the velocity of the rail vehicle is based on the dynamically determined travelled distance and the time stamps.

[0051] In embodiments, the step of extracting comprises determining a velocity of the rail vehicle based on at least one of the image data and the point cloud data. In embodiments, the step of determining a velocity of the rail vehicle comprises detecting a rail track, preferably detecting rail track features, such ascrossties or sleepers and / or pavements or sidewalks, in at least one of the image data and the point cloud data. In embodiments, the step of extracting comprises determining a velocity of the rail vehicle is (further) based on at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data. In embodiments, the step of extracting comprises determining a velocity of the rail vehicle is based on the fused dataset. In embodiments, the step of determining a velocity of the rail vehicle (further) comprises detecting a rail track, preferably detecting rail track features, such as crossties or sleepers and / or pavements or sidewalks, in at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data. In embodiments, the step of determining a velocity of the rail vehicle (further) comprises detecting a rail track, preferably detecting rail track features, such as crossties or sleepers and / or pavements or sidewalks, in the fused dataset. The rail track may have predetermined dimension(s), such as a predetermined rail track width or gauge, and / or such as a predetermined distance between the crossties. Additionally or alternatively, the step of detecting the rail track features comprises determining the rail track dimensions in in any one or combination of the sensor data. In embodiments, any of the steps of detecting a rail track, preferably rail track features, more preferably the dimensions of the rail track (features), is based on fifth training data relating to a plurality rail tracks and optionally a plurality of rail track features and / or dimensions thereof. The fifth training data may comprise historically obtained data (e.g., image data and / or point cloud data, electromagnetic wave data, echo data, etc.) from the respective sensor(s), which are labelled as featuring or including a rail track, preferably having one or more rail track features, more preferably having dimensions of the rail track (features). In embodiments, the step of extracting comprises dynamically determining a velocity of the rail vehicle based on any one or combination of the sensor data. In embodiments, the step of dynamically determining a velocity of the rail vehicle comprises detecting a rail track, preferably detecting rail track features, such as crossties or sleepers and / or pavements or sidewalks, in any one or combination of the sensor data. Additionally or alternatively, the step of detecting the rail track features comprises dynamically determining the rail track dimensions in any one or combination of the sensor data.

[0052] In embodiments, the step of detecting a rail track, preferably detecting rail track features and / or determining rail track (features) dimensions are based on sixth training data which comprise the second and fifth training data. The second and fifth training data may be merged into the sixth training data. The step of detecting a rail track, preferably detecting rail track features and / or determining rail track (features) dimensions may comprise applying the second and / or fifth ML model which has been trained on the sixth training data.

[0053] In embodiments, the step of determining a velocity of the rail vehicle is based on the detected rail track, preferably detected rail track features, more preferably the determined dimensions thereof. In embodiments, the step of determining a velocity of the rail vehicle comprises determining a travelled distance of the rail vehicle based on the detected rail track, preferably detected rail track features, more preferably the determined dimensions thereof, wherein the velocity of the rail vehicle may be determined based on the determined travelled distance of the rail vehicle and based on the timestamps, e.g. the timestamps related / attributed to the image data and the point cloud data based on which the travelled distance of the rail vehicle is determined. In embodiments, the step of determining a velocity of the rail vehicle comprises calculating a travelled time relating to the detected object as a difference between timestamps relating / attributed to the at least two (consecutive) captures (e.g., captured image frame, captured point cloud, captured echo signal, etc.), and calculating a mean distance travelled by points relating to the at least one of the detected rail track features in at least two (consecutive) point clouds and / or calculating a mean distance travelled by pixels relating to at least one of the detected rail track features in at least two (consecutive) image frames comprised in the image data. In embodiments, the step of determining a velocity of the rail vehicle (further) comprises calculating a mean distance travelled by waves relating to at least one of the detected rail track features in at least two (consecutive) electromagnetic radio waves, and / or at least two (consecutive) electromagnetic microwaves, and / or at least two (consecutive) pressure waves. For example, the travelled distance of the rail vehicle may be determined based on a change in position / location of a crosstie in (consecutive) image frames comprised in the image data and a change in position / location of points relating to the crosstie in the (consecutive) point cloud data, and a travelled time of the rail vehicle may be determined based on a difference between the respective timestamps related / attributed to the (consecutive) point clouds and in (consecutive) image frames comprised in the image data. The inventors have shown that by using the sensor data to determine the velocity of the rail vehicle, alternatively to the velocity determined based on the GPS coordinates and / or the obtained velocity, less modules, such as the GPS module, may be used, and thereby allowing to provide a system which is more cost efficient. However, determining the velocity of the rail vehicle based on the sensor data may be advantageous in addition to determining the velocity based on the GPS coordinates, particularly in cases where the latter may not be accurate, e.g. in tunnels where the connection to the GPS is limited.

[0054] In embodiments, the metadata comprises a (dynamic) behavior of the rail vehicle, such as rail vehicle conductor actions or autonomous rail vehicle actions. The behavior of the rail vehicle may comprise at least one of an obtained velocity, an obtained acceleration, and an obtained deceleration of the rail vehicle. In embodiments, the step of determining the TTC-related quantity is based on the metadata, particularly based on the (dynamic) behavior of the rail vehicle, such as based on the obtained velocity of the rail vehicle.

[0055] In embodiments, the step of extracting comprises determining a velocity of the object based on the detected object in the image data and / or the point cloud data. In embodiments, the step of determining a velocity of the object comprises detecting the rail track, preferably detecting rail track features, such as crossties or sleepers and / or pavements or sidewalks, in the image data and / or the point cloud data. In embodiments, the step of extracting comprises determining a velocity of the object (further) based on at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data. In embodiments, the step of extracting comprises determining a velocity of the object based on the fused dataset. In embodiments, the step of determining a velocity of the object (further) comprises detecting the rail track, preferably detecting rail track features, such as crossties or sleepers and / or pavements or sidewalks, in at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data. In embodiments, the step of determining a velocity of the object comprises detecting the rail track, preferably detecting rail track features, such as crossties or sleepers and / or pavements or sidewalks, in the fused dataset. The rail track may have predetermined dimension, such as a predetermined rail track width or gauge, and / or such as a predetermined distance between the crossties. Additionally or alternatively, the step of detecting the rail track features comprises determining the rail track dimensions inin any one or combination of the sensor data. In embodiments, any of the steps of detecting a rail track, preferably rail track features, more preferably the dimensions of the rail track (features), is based on the fifth training data. In embodiments, the step of determining a velocity of the object is based on the detected rail track, preferably detected rail track features. In embodiments, the step of determining a velocity of the object comprises determining a travelled distance of the object based on the detected rail track, preferably detected rail track features, wherein the velocity of the object may be determined based on the determined travelled distance of the object and based on the timestamps, e.g. the timestamps related / attributed to the image data and the point cloud data based on which the travelled distance of the object is determined. In embodiments, the step of extracting comprises dynamically determining a velocity of the object based on the detected object in any one or combination of the sensor data. In embodiments, the step of dynamically determining a velocity of the object comprises detecting the rail track, preferably detecting rail track features, such as crossties or sleepers and / or pavements or sidewalks, in any one or combination of the sensor data.

[0056] In embodiments, the step of extracting comprises determining a velocity of the object based on the obtained or determined velocity of the rail vehicle and based on the determined relative velocity between the rail vehicle and the object. In embodiments, the step of extracting comprises dynamically determining a velocity of the object based on the obtained or determined velocity of the rail vehicle and based on the determined relative velocity between the rail vehicle and the object. The (dynamically) determined velocity of the object may be a portion of the velocity of the object in the direction of the rail track.

[0057] In embodiments, the step of extracting comprises determining a direction of motion of the object based on the detected object in the image data and / or the point cloud data. In embodiments, the step of determining a direction of motion of the object comprises detecting the rail track, preferably detecting rail track features, such as rail track gauge and / or two or more rails (e.g., running rail, guard rail, etc.) and / or two or more pavements or sidewalks, in the image data and / or the point cloud data. In embodiments, the step of extracting comprises determining a direction of motion of the object (further) based on at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data. In embodiments, the step of determining a direction of motion of the object (further) comprises detecting the rail track, preferably detecting rail track features, such as rail track gauge and / or two or more rails (e.g., running rail, guard rail, etc.) and / or two or more pavements or sidewalks, in at least one of the electromagnetic radio wave data, the electromagnetic microwave data, and the echo data. In embodiments, the step of extracting comprises determining a direction of motion of the object based on the fused dataset. In embodiments, the step of determining a direction of motion of the object comprises detecting the rail track, preferably detecting rail track features, such as rail track gauge and / or two or more rails (e.g., running rail, guard rail, etc.) and / or two or more pavements or sidewalks, in the fused dataset. In embodiments, the step of extracting comprises dynamically determining a direction of motion of the object based on the detected object in any one or combination of the sensor data. In embodiments, the step of dynamically determining a direction of motion of the object comprises detecting the rail track, preferably detecting rail track features.

[0058] In embodiments, the step of determining a direction of motion of the object comprises determining a distance between the detected object and at least one of the detected two or more rails. In embodiments,the step of determining a direction of motion of the object comprises determining at least two respective distances between the detected object and at least one of the detected two or more rails in at least two (consecutive) image frames comprised in the image data and / or determining at least two respective distances between at least one point related to the detected object and to at least one of the detected two or more rails in at least two (consecutive) point clouds comprised in the point cloud data. In embodiments, the step of determining a direction of motion of the object (further) comprises determining at least two respective distances between waves related to the detected object and to at least one of the detected two or more rails in at least two (consecutive) electromagnetic radio waves comprised in the electromagnetic radio wave data, and / or at least two (consecutive) electromagnetic microwaves comprised in the electromagnetic microwave data, and / or at least two (consecutive) pressure waves comprised in the echo data. In embodiments, the step of determining a direction of motion of the object comprises determining a distance between the detected object and at least one of the detected two or more rails. In embodiments, the step of dynamically determining a direction of motion of the object comprises dynamically determining at least two respective distances between the detected object and at least one of the detected two or more rails in at least two (consecutive) image frames comprised in any one or combination of the sensor data.

[0059] In embodiments, any of the steps of detecting a rail track, preferably rail track features, more preferably the dimensions of the rail track (features), comprises applying a fifth ML model which has been trained on the fifth training data. The fifth ML model may comprise a regression model or a classification model. The machine learning model may comprise a deep learning, DL, model (e.g., neural networks, NN, convolutional NN, CNN, region-based CNN, etc.) or a classic ML model (e.g., support vector machine classification, logistic regression, etc.), or a hybrid of one or more DL models and / or one or more classic ML models.

[0060] In embodiments, the direction of motion of the object is (dynamically) determined to be in the direction of the rail track upon determining that a difference between the determined (consecutive) respective distances between the detected object and at least one of the detected two or more rails is lower than a predetermined distance threshold. Thus, if a change in the difference between the determined (consecutive) respective distances increases above the predetermined distance threshold, then it is determined that the direction of motion of the object is at least partially at an angle with the direction of the rail track, e.g., a person crossing the rail track would be determined as having a direction of motion substantially perpendicular to the direction of the rail track.

[0061] In embodiments, the step of (dynamically) determining a direction of motion of the object and the step of determining a velocity of the object in the direction of motion of the object comprise computing an optical flow of the detected object in at least two (consecutive) image frames comprised in the image data. The determined direction of motion of the object and the determined velocity of the object may be comprised in or represented by a displacement vector field. In embodiments, the step of determining a direction of motion of the object comprises calculating a mean distance travelled by points relating to the detected object between at least two (consecutive) point clouds comprised in the point cloud data. In embodiments, the step of determining a direction of motion of the object (further) comprises calculating a mean distance travelled by waves relating to the detected object in at least two (consecutive) electromagnetic radio waves,and / or at least two (consecutive) electromagnetic microwaves, and / or at least two (consecutive) pressure waves.

[0062] In embodiments, the step of (dynamically) determining a velocity of the object in the direction of motion of the object comprises calculating a travelled time relating to the detected object as a difference in time stamps between the at least two (consecutive) point clouds comprised in the point cloud data and / or the at least two (consecutive) image frames comprised in the image data, and is based on the calculated mean distance travelled by the points relating to the detected object in the at least two (consecutive) point clouds comprised in the point cloud data and / or calculating a mean distance travelled by the detected object in at least two (consecutive) image frames comprised in the image data. In embodiments, the step of determining a velocity of the object in the direction of motion of the object (further) comprises calculating a travelled time relating to the detected object as a difference in time stamps between the at least two (consecutive) electromagnetic radio waves, and / or at least two (consecutive) electromagnetic microwaves, and / or at least two (consecutive) pressure waves, and is based on the calculated mean distance travelled by the waves relating to the detected object in the at least two (consecutive) electromagnetic radio waves, and / or at least two (consecutive) electromagnetic microwaves, and / or at least two (consecutive) pressure waves.

[0063] In embodiments, the step of extracting comprises predicting a target location of the object on the rail track based on the (dynamically) determined direction of motion and the determined velocity of the object in the direction of motion of the object, wherein the step of determining the TTC-related quantity comprises predicting a TTC-related quantity based on the predicted target location of the object on the rail track. In embodiments, the step of extracting comprises predicting a target location of the object off of the rail track based on the determined direction of motion and / or the determined velocity of the object. In embodiments, the step of extracting comprises determining a target distance to the target location of the object on the rail track based on the determined direction of motion and velocity of the object, wherein the predicting a TTC-related quantity is based on the determined target distance and optionally on a determined or obtained velocity of the rail vehicle. Often, objects are moving and in a direction at least partially in a direction perpendicular to the rail track (e.g., at an angle different from 0 degrees with the direction of motion of the rail vehicle or with the rail track, such as at 45 degrees). Therefore, the object is detected near the rail tracks, but not thereon. Therefore, by using preemptive measures, the method can predict the target location of the object on the rail track and / or the target distance to the object on the rail track, and thereby allowing to more effectively determine the TTC-related quantity.

[0064] According to a further aspect, the present invention provides a device comprising means for carrying out the method for dynamically determining a TTC-related quantity of a rail vehicle with an object according to the invention.

[0065] In embodiments, the means comprised in the device concern at least one (micro) processor. In embodiments, the device is an edge computing device or a (cloud) server.

[0066] In embodiments, the device comprises a display module for displaying the first and / or second alert.

[0067] According to a further aspect, the present invention provides a system comprising the device according to the invention and at least one of an imaging sensor and laser-based ranging sensor.

[0068] In embodiments, the system comprises the display module configured to receive the first and / or second alert generated by the device and display the received first and / or second alert, wherein the device is configured for carrying out the method for dynamically determining a TTC-related quantity of a rail vehicle with an object according to the invention.

[0069] In embodiments, the system comprises a sensing module comprising at least one, preferably both, of the imaging sensor and laser-based ranging sensor. In embodiments, the system comprises at least one of a GPS module and a clock / timer. In embodiments, the sensing module comprises at least one of the GPS module and clock / timer.

[0070] According to a further aspect, the present invention provides a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method for dynamically determining a TTC-related quantity of a rail vehicle with an object according to the invention.

[0071] In embodiments, the computer-readable medium is a non-transient computer readable medium containing a computer executable software which when executed on a computer system performs the method as defined herein before by the embodiments of the present disclosure. A non-transient computer readable medium may include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a non-transient computer readable medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, device or module.

[0072] In a further aspect of the invention, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method for dynamically determining a TTC-related quantity of a rail vehicle with an object. The computer program product may comprise at least one readable medium in which computer-readable program code portions are saved, which program code portions comprise instructions for carrying out said method.Examples

[0073] Example embodiments of the invention will be described with reference to Fig. 1 and 2, which are not intended to limit the scope of the invention in any way.Example 1 : example of the method according to the invention

[0074] This example is described with reference to Fig. 1 and 2. Fig. 1 illustrates a top view of a rail vehicle (1 ) on a rail track (3). Fig. 2 illustrates a side view of the rail vehicle (1 ) on the rail track (3). In this example according to embodiments of the invention, there is provided a method for dynamically determining a time- to-col I ision , TTC, related quantity of a rail vehicle (1 ) with an object (2), the rail vehicle (1 ) comprising at the front end of the rail vehicle (1 ) a sensor module (10) comprising an imaging sensor and a laser-based ranging sensor, the method comprising:- obtaining image data (12) from the imaging sensor and point cloud data (11 ) from the laser-based ranging sensor;- detecting the object (2) nearby or on the rail track (3) in the image data (12) and the point cloud data (11 ) based on first training data relating to a plurality of objects;- extracting information from the image data (12) and the point cloud data (11 ); and- determining the TTC-related quantity of the rail vehicle (1 ) with the object (2) based on the extracted information from the image data (12) and the point cloud data (11 ).

[0075] In this example, the step of extracting comprises determining a distance (6) to the object (2) and a relative velocity of the rail vehicle (1 ) with respect to the object (2), wherein the step of determining the TTC-related quantity of the rail vehicle (1 ) with the object (2) is based on the determined distance (6) to the object (2) and the determined relative velocity of the rail vehicle (1 ) with respect to the object (2).

[0076] Furthermore, in this example the image sensor and the laser-based ranging sensor have partially overlapping FoVs, as can be seen from Fig. 1 . From both Fig. 1 and 2, the laser-based sensor has a longer FoV or detection range, whereas the imaging sensor has a shorter but wider FoV or detection range.

[0077] By using both types of sensors to cobmine the image data and he point cloud data, an improved (3D) spatial coverage for a longer detection range can be determined, and thereby improving the determining of TTC-related quantity.Example 2: example of the method according to the invention

[0078] This example is described with reference to Fig. 4 which illustrates a side view of the rail vehicle (1 ) on the rail track (3). In this example we consider the method described with reference to Fig. 1 and 2. In this example according to embodiments of the invention, the step of extracting comprises determining a rail track condition (16) in the image data (12) and the point cloud data (11 ) based on second training data relating to a plurality of rail track conditions.

[0079] Furthermore, the step of extracting information comprises determining a weather condition (15) in the image data (12) and the point cloud data (11 ) based on third training data relating to a plurality of weather conditions, wherein the step of determining a rail track condition (16) is based on the determined weather condition (15).

[0080] A particular weather condition in combination with a particular rail track condition, in this example rainfall (e.g., more than 10mm / h) in combination with water pooling (e.g., > 2mm) on the surface of the rail tracks, as dusty rail track and rain, can further reduce the friction between the rail track and the rail vehicle wheels. Therefore, by determining both the weather condition (15) and the rail track condition (16), the TTC-related quantity can be more effectively determined.

[0081] As shown in Fig. 4, the portion of the rail track for which the rail track condition is determined is away from the front of the rail vehicle (1 ) and / or from the sensory module (10) and in the FoV of the at least one sensor in the sensory module (10).Example 3: example of a method according to the invention

[0082] This example is described with reference to Fig. 5 which illustrates a side view of the rail vehicle (1 ) on the rail track (3). In this example we consider the method of any of Fig. 1 or Fig. 2. In this example according to embodiments of the invention, the step of extracting comprises determining a distance (6) to the object (2) and a relative velocity of the rail vehicle (1 ) with respect to the object (2).

[0083] In this example, the step of extracting comprises determining a direction of motion (8) and a velocity of the object (2) in the direction of motion (8) based on the image data (12) and the point cloud data (11 ), and predicting a target location (2’) of the object (2) on a rail track (3) based on the determined direction of motion (8) and velocity of the object (2), wherein the step of determining the TTC-related quantity comprises predicting the TTC-related quantity based on the predicted target location (2’) of the object (2) on the rail track (3).

[0084] In this example, the step of extracting comprises determining a target distance (6’) to the target location (2’) of the object (2) on the rail track (3) based on the determined direction of motion (8) and velocity of the object (2), wherein the predicting a TTC-related quantity is based on the determined target distance (6’), and preferably on a determined or obtained velocity of the rail vehicle (1 ).

[0085] As can be seen in Fig. 5, the object (2) is a moving person detected nearby the rail track (3) and walking in a direction of motion (8) towards the rail track (3) at a determined velocity. Thus, the method can predict the TTC-related quantity before the person arrives at the rail track (3), or even at the target location (2’).Example 4: example of the method according to the invention

[0086] This example is described with reference to Fig. 3 which illustrates a side view of the rail vehicle (1 ) on the rail track (3). In this example we consider the method of any of Fig. 1 , 2, 4 and 5. In this example according to embodiments of the invention, the step of extracting comprises determining a velocity of the rail vehicle (1 ) based on two consecutive image frames and point clouds (5, 5’). In embodiments, the step of determining a velocity of the rail vehicle (1 ) comprises detecting rail track features, particularly crossties in the two consecutive image frames and point clouds (5, 5’).

[0087] The distance between the crossties is predetermined. However, the distance therebetween may be determined in at least one of the image frames and point clouds based on training data relating to a plurality rail tracks and optionally a plurality of rail track features and / or dimensions thereof. In this example, the travelled distance of the rail vehicle (1 ) is determined by the change in location of the at least one crosstie in the two consecutive image frame and point clouds (5, 5’), wherein a first distance (7) to the crosstie in the first image frame and point cloud (5) is determined and a second distance (7’) to the crosstie in the second image frame and point cloud (5’) is determined. Therefore, the travelled distance of the rail vehicle (1 ) is determined based on the difference between the first and second distances (7, 7’). Thus, the velocity of the rail vehicle is determined based on the determined travelled distance of the rail vehicle (1 ) and on a difference between a second time stamp related / attributed to the second image frame and point cloud (5’) and a first time stamp related / attributed to the first image frame and point cloud (5).

[0088] By using the sensor data to determine the velocity of the rail vehicle (1 ), the method ensures that the velocity can be reliably determined, particularly in cases where other means to determine the velocity of the rail vehicle (1 ) may be limited, such as in tunnels.

[0089] (End of Example 4)

[0090] Although the present invention has been described above with reference to certain embodiments thereof, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader scope of the present invention, as defined by the appended claims.

Claims

Claims1. A method for dynamically determining a time-to-collision, TTC, related quantity of a rail vehicle (1 ) with an object (2), the rail vehicle (1 ) comprising an imaging sensor and a laserbased ranging sensor, the method comprising:- obtaining image data (12) from the imaging sensor and point cloud data (11 ) from the laser-based ranging sensor;- detecting the object (2) in the image data (12) and the point cloud data (11 ) based on first training data relating to a plurality of objects;- extracting information from the image data (12) and the point cloud data (11 ), wherein the step of extracting comprises determining a rail track condition (16) in the image data (12) and the point cloud data (11 ) based on second training data relating to a plurality of rail track conditions; and- determining the TTC-related quantity of the rail vehicle (1 ) with the object (2) based on the extracted information from the image data (12) and the point cloud data (11 ), wherein the determining is further based on metadata relating to a dynamic behaviour of the rail vehicle, the metadata comprising at least one of an obtained velocity, an obtained acceleration, and an obtained deceleration of the rail vehicle.

2. The method according to claim 1 , wherein the step of extracting information comprises determining a weather condition (15) in the image data (12) and the point cloud data (11 ) based on third training data relating to a plurality of weather conditions, wherein the step of determining a rail track condition (16) is based on the determined weather condition (15).

3. The method according to claim 1 or claim 2, further comprising obtaining metadata relating to the image data (12) and the point cloud data (11 ), wherein the metadata comprises time stamps and global positioning system, GPS, coordinates, wherein the step of determining the TTC-related quantity is based on the metadata.

4. The method according to claim 3, wherein the step of determining a weather condition (15) is based on meteorological information determined based on the GPS coordinates and the time stamps.

5. The method according to claim 4, wherein the step of determining a rail track condition (16) is based on the determined weather condition (15) within the past 24 hours, preferably within the past 12 hours, of the time stamps.

6. The method according to any of claims 1-5, further comprising determining a target velocity of the rail vehicle (1 ) for avoiding a collision of the rail vehicle (1 ) with the object (2).

7. The method according to any of claims 1-6, further comprising generating an alert upon detecting that the TTC-related quantity is lower than a predetermined first time threshold,wherein the alert comprises at least one of the TTC-related quantity and target velocity of the rail vehicle (1 ).

8. The method according to any of claims 1-7, further comprising controlling a braking device comprised in the rail vehicle (1 ) to initiate braking upon detecting that the TTC-related quantity is lower than a predetermined second time threshold which is lower than the predetermined first time threshold.

9. The method according to any of claims 1-8, wherein the step of extracting comprises determining a distance (6) to the object (2) and a relative velocity of the rail vehicle (1 ) with respect to the object (2).

10. The method according to claim 9, wherein the step of extracting comprises determining a direction of motion (8) and a velocity of the object (2) in the direction of motion (8) based on the image data (12) and the point cloud data (11 ), and predicting a target location (2’) of the object (2) on a rail track (3) based on the determined direction of motion (8) and / or velocity of the object (2), wherein the step of determining the TTC-related quantity comprises predicting the TTC-related quantity based on the predicted target location (2’) of the object (2) on the rail track (3).

11. The method according to any of claims 1-10, wherein at least one of the first time threshold and second time threshold is determined based on the determined weather condition (15) and / or the determined rail track condition (16).

12. A device comprising means for carrying out the method of any one of claims 1-11.

13. A system comprising: a device according to claim 12, an imaging sensor, and a laser-based ranging sensor, wherein the device is configured to perform the steps of:- obtaining image data (12) from the imaging sensor and point cloud data (11 ) from the laser-based ranging sensor;- detecting the object in the image data (12) and the point cloud data (11 ) based on first training data relating to a plurality of objects;- extracting information from the image data (12) and the point cloud data (11 ), wherein the step of extracting comprises determining a rail track condition (16) in the image data (12) and the point cloud data (11 ) based on second training data relating to a plurality of rail track conditions; and- determining the TTC-related quantity of the rail vehicle (1 ) with the object (2) based on the extracted information from the image data (12) and the point cloud data (11 ),wherein the determining is further based on metadata relating to a dynamic behaviour of the rail vehicle, the metadata comprising at least one of an obtained velocity, an obtained acceleration, and an obtained deceleration of the rail vehicle.

14. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1-11.

15. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1-11.

Images