SURVEILLANCE PROCEDURES AND SYSTEM

Description

[0001] The present invention relates to a method and a system for monitoring a vehicle's environment while a vehicle is traveling on a route, and to a vehicle, in particular a rail vehicle.

[0002] Modern vehicles, especially rail vehicles, are typically equipped with sensors used to monitor their surroundings. These sensors can detect other road users or lanes, for example, and the information gathered can be used by driver assistance systems to control the vehicle, potentially providing additional support. A particular challenge lies in processing the sheer volume of data generated by monitoring the vehicle's environment. For instance, it can be difficult to analyze complex driving situations involving numerous static objects and / or other road users, and to determine an appropriate response from the assistance system.

[0003] Documents DE 10 2018 206 593 A1 and WO 2018 / 104460 A1 concern methods for detecting objects on a railway platform. Objects on the platform are detected by sensors, and a hazard assessment is carried out. US 2017 / 161568 A1 also concerns a method for object detection in the vicinity of railway vehicles.

[0004] It is an object of the present invention to improve, and in particular to simplify, the analysis of driving situations.

[0005] This task is solved by a method and a system for monitoring the vehicle environment during a journey of a vehicle on a route, and by a vehicle according to the independent claims.

[0006] Preferred embodiments are the subject of the dependent claims and the following description.

[0007] A first aspect of the invention relates to a method, particularly a computer-implemented method, for monitoring a vehicle's surroundings while a vehicle, especially a rail vehicle, is traveling on a route, particularly a predetermined one. The method comprises the following steps: (i) sensorial detection of a scene in the vehicle's surroundings and generation of corresponding sensor data; (ii) generation of a virtual scene representation of the detected scene based on the sensor data and quality-assured route information stored in a database.

[0008] A scene within the meaning of the invention preferably refers to a driving situation of the vehicle. In other words, a scene can represent a "snapshot" of the vehicle's surroundings at a specific point in time. Thus, a scene is expediently characterized by objects from the vehicle's environment and their arrangement relative to each other and / or to the vehicle at a specific point in time.

[0009] A virtual scene representation within the meaning of the invention preferably refers to a digital image of a (real) scene, for example, from the vehicle's surroundings. Based on a virtual scene representation, an assistance system can analyze the vehicle's surroundings and control the vehicle, or at least individual vehicle functions, based on this analysis, and / or provide the driver with at least some guidance on vehicle control.

[0010] Route information within the meaning of the invention preferably refers to additional information about possible scenes along the route. This route information can, for example, be additional information about sensor-detectable objects or a specific area within the currently detected scene. The route information can be at least partially route-specific. For instance, it is conceivable that the route information identifies particular objects, object constellations, route segments, and / or the like that occur along the route, and / or their effects on each other and / or on the vehicle.

[0011] A database within the meaning of the invention is preferably understood to be a map, particularly a digital one, which may contain, for example, route-specific information. The information contained in the database, particularly route-specific information, is expediently linked to geographical information. It is particularly preferred if the database also contains generic information about the vehicle's surroundings, for example, about road users that may be present in the vehicle's vicinity. Expediently, in addition to route information about the course of roads or paths, the database contains additional information about the area surrounding the roads, for example, about so-called landmarks. The landmarks can characterize, for example, traffic signs, traffic lights, streetlights, buildings, and / or other prominent objects in the vicinity of roads or paths with extremely high spatial resolution.

[0012] For the purposes of this invention, quality-assured information preferably refers to information that fulfills at least one predefined safety requirement. For example, quality-assured information may be correct with a predefined probability or exhibit a low error rate, such that a system relying on the information has only a low probability of exhibiting behavior that endangers health or life. An (assistance) system operated on the basis of the quality-assured information preferably fulfills a so-called Safety Integrity Level (SIL) in order to reduce the risk to persons, the environment, or processes posed by the system with a predefined reliability—for example, to less than one hazard every eleven years.

[0013] Quality-assured information can, for example, characterize the properties, hereinafter also referred to as features, of objects. Quality assurance of the information expediently includes, among other things, ensuring that at least one combination of the properties or features is specific to each of the objects. In particular, quality-assured information may stipulate that the unambiguous description of an object by at least one feature, especially a combination of features, is guaranteed with regard to specificity compared to other objects.

[0014] One aspect of the invention is based on the approach of adapting, in particular simplifying and / or supplementing, a virtual scene representation based on sensory perception of a vehicle's environment, which characterizes a real scene from the vehicle's environment, using external information. External information here means that the information is not provided by the vehicle or its sensors, but rather is taken, for example, from a database. Such an adapted virtual scene representation can then, for example, be analyzed and used as the basis for controlling at least one sub-function of the vehicle. Determining the scene by taking the information from the database into account enables not only a more reliable, but potentially also a faster scene analysis.

[0015] Accessing external information stored in the database, also known as route information, is particularly advantageous because it allows prior knowledge about the vehicle's surroundings along the route to be incorporated into the determination of the virtual scene representation—especially its adaptation. This approach leverages the fact that rail vehicles frequently travel the same route, meaning that objects that do not impair driving safety are either already known or can at least be identified. If the virtual scene representation is to be used, for example, as the basis for a hazard analysis to assess driving safety, these objects do not need to be included in the scene representation at all or can be marked as irrelevant to driving safety.This technical support allows even a driver who is not yet trained for this route to drive a new route with a comparable level of safety as a trained driver, thereby significantly increasing the flexibility of rail vehicle operations with regard to personnel planning.

[0016] To determine the virtual scene representation, a scene in the vehicle's vicinity, such as the track bed in front of a train or a platform as the train enters a station, can be captured by sensors, and the vehicle's position on its route determined. Route information corresponding to this position can then be loaded from a digital map. The virtual scene representation can be generated using the sensor data generated during the sensor capture of the vehicle's surroundings and the route information. It is particularly conceivable that the virtual scene representation determined based on the sensor data could be further adapted, i.e., modified, using the route information.

[0017] The route information should ideally be quality-assured, i.e., in particular precise, complete, up-to-date and available.

[0018] This means that the semantics of the information should be well-defined so that, for example, an algorithm based on the information can interpret it correctly. If the information contains object attributes for characterizing an object, the range of values ​​of the attributes, along with their possible values ​​and the semantics of each value, should be defined precisely and completely. Furthermore, the necessary parameters should be defined for each use of the information, such as in connection with a safety-related function. These parameters could be, for example, the quality attributes specified for identifying or determining the position of an object. The parameters should accurately reflect the correct situation on the route for every journey.Similarly, the information should be available for use during the journey with the quality required, for example, in a safety architecture.

[0019] The use of quality-assured route information thus allows not only particularly reliable scene adaptation, but also, if necessary, a verification of the vehicle's sensor functions by comparing the route information with information obtained from the sensor data. For example, properties of an object detected in the vehicle's environment, derived from the sensor data, can be compared with so-called feature vectors stored in the digital map, which characterize objects in the vehicle's environment.

[0020] The feature vectors preferably contain specific features of the detected objects as entries. The length of the vectors, i.e., the number of features they describe, expediently corresponds to a reliability measure. A large amount of information stored in the database allows for the highly reliable validation of objects detected from the vehicle's surroundings using sensor data.

[0021] In particular, by combining features of a feature vector, an object detected based on sensor data can be identified in the database with a high degree of certainty, i.e., associated with an object recorded in the database. This allows proof of the object's presence in the sensor data to be provided.

[0022] Advantageously, the feature vectors are overdetermined. This means that even a subset of the specific features contained in the feature vectors is sufficient to uniquely identify the object. This increases the robustness when comparing the sensor data with the corresponding feature vectors.

[0023] The length and properties of the feature vectors are preferably defined or predetermined by a hazard analysis and safety architecture. In particular, the features in the feature vector should be sufficiently specific to rule out a coincidental match between objects in the vehicle's environment and objects recorded in the database with a sufficiently high degree of certainty.

[0024] Route information for scene adaptation is conveniently extracted from such feature vectors. These feature vectors can, for example, contain information about the extent to which a particular object influences the vehicle's driving safety. Accordingly, certain objects in the scene representation can be given greater weight when controlling the vehicle, while other objects can be neglected.

[0025] For objects identified in the captured scene, the route information is used to check whether a predefined hazard criterion is met. Depending on the result of this check, the objects are considered when determining the virtual scene representation. For example, they may be marked in the scene representation with a "flag," i.e., a status indicator, or not included at all. Preferably, the objects can be considered in a driving safety assessment based on the scene representation, depending on the result of the check / marking. For example, it is conceivable that objects could be removed from the scene representation during the assessment depending on their marking status.This avoids increasing the (computational) effort in an analysis of the scene representation, especially with regard to driving safety, by unnecessarily considering inherently harmless objects, for example stationary objects such as buildings, vegetation and / or the like.

[0026] The route information should ideally include a hazard assessment – ​​with regard to the vehicle's movement – ​​of individual objects in the captured scene. For example, individual objects along the route, or even entire types of objects, can be rated as dangerous or harmless and / or have a hazard score. This type information should also be quality-assured. This allows for a particularly simple check of whether the specified hazard criterion is met and / or for a particularly simple consideration of the route information.

[0027] The route information contains type information specific to an object type, i.e., information that characterizes that object type. All objects of a given object type are considered when determining the virtual scene representation according to this type information, for example, by being marked. Ideally, all objects of a given object type are removed from the virtual scene representation based on this type information, for example, to facilitate scene analysis. Type information can be understood here as, in particular, generalized information about an object type, such as a vehicle or a specific vehicle class, a pedestrian, vegetation, buildings, traffic signals, and / or the like. The type information can also include, in particular, a universally applicable hazard assessment for a specific object type, i.e., a hazard assessment valid for all objects of that type, perhaps in the form of a hazard score.The vehicle type information can, for example, indicate that a pedestrian may be at particularly high risk from the vehicle. Alternatively or additionally, the vehicle type information can indicate that birds or an oncoming rail vehicle (on a different track) generally pose no danger. This vehicle type information thus takes into account the fact that certain objects in a recorded scene do not need to be considered a priori with regard to driving safety.

[0028] Preferred embodiments of the invention and their further developments are described below, which, unless expressly excluded, can be combined with each other and with the aspects of the invention described below.

[0029] In a preferred embodiment, a consistency measure is determined during the generation of the virtual scene representation to assess the agreement between environmental information derived from the sensor data and the route information. This consistency measure can also be interpreted as a confidence measure that characterizes the reliability of object detection in the vehicle's environment, particularly the magnitude of discrepancies between the environmental information and the route information. In other words, the consistency measure indicates the specificity with which an object is detected based on the sensor data. The reliability of object detection can be assessed particularly easily using this consistency measure.

[0030] The degree of agreement can be high, for example, if the sensor data not only characterizes objects that are also characterized by the route information, but also if the characterization by the respective information essentially matches. The route information might contain not only information about an object's position, but also about other quality-assured features whose traceability and specificity can be verified by an expert. The route information might also include information about an object's shape, structure, extent, texture, color, and / or similar characteristics. The greater the proportion of this information that matches corresponding information from the sensor data, the higher the determined degree of agreement can be.Accordingly, the determined degree of agreement may be small if, although the information contained in the sensor data about the position and shape of a detected object essentially matches the corresponding route information, the information about the extent and texture does not.

[0031] It is advisable to check whether the degree of agreement reaches or exceeds a predefined agreement threshold. This threshold can serve as an indicator of whether a sufficient or reliable number of properties of a detected object match the route information. If so, reliable object detection can be assumed. This allows conclusions to be drawn, for example, about the reliability of the vehicle's position determination, the dependable and accurate functioning of the sensors for detecting and determining object positions, and, if applicable, the correct calibration of multiple sensors relative to each other. If this is not the case, it can be assumed that the object was not detected correctly.

[0032] The conformance threshold preferably depends on the security requirements or can be chosen accordingly. If the conformance threshold is set high and the conformance level is reached or exceeded, a high level of security can generally be associated with object detection. Conversely, if the conformance threshold is set low and the conformance level is reached or exceeded, only a low level of security can be associated with object detection.

[0033] Different route information may be valid for different circumstances under which the vehicle travels on the route. Therefore, in a further preferred embodiment, the route information is provided depending on at least one driving parameter, i.e., read from a database. A driving parameter can be, for example, the vehicle's position, speed, and / or direction of travel. Alternatively or additionally, a driving parameter can also be the weather, time, date, and / or the like. This allows for a differentiated assessment of the captured scene.

[0034] By considering route information dependent on driving parameters, the impact of an object on driving safety can be taken into account within a broader context. For example, the safety of people on a platform as a train passes through the station differs from that of people on a potentially physically separated path alongside the railway line. While the latter do not need to be included in the virtual scene representation, considering the people on the platform can be of paramount importance for driving safety.

[0035] However, if a person is on or near the open track without a protective barrier, especially at the same distance from the track as the person in the station, then they are considered more relevant to the safety of the journey, since this person should not actually be in that position. If this person is on the open track without a barrier in close proximity to a hospital where people with psychiatric illnesses are treated, the assessment of the person's short distance to the track can be even more significant. Accordingly, the assessment of the fact that a person is near the track can depend heavily on the context of the situation. The information for the safety assessment of the person's relative position to the track is preferably stored in the database with appropriate quality assurance.The safety assessment can thus be stored for the entire route during the planning of the journey, meaning that this assessment does not have to be determined by an algorithm during the journey.

[0036] In a further preferred embodiment, the determined virtual scene representation forms the basis for an assessment of driving safety. For example, an analysis of the virtual scene representation can be performed, and a hazard assessment can be carried out based on this analysis. Driving safety is preferably understood to mean safety both with regard to hazards to the vehicle from its environment, for example, from objects in the vehicle's vicinity, and with regard to hazards to the vehicle's environment, for example, to objects in the vehicle's vicinity, caused by the vehicle itself. Based on the virtual scene representation, the vehicle can be safely controlled through its environment, particularly along the specified route.

[0037] In a further preferred embodiment, the route information includes scene information specific to an area within the captured scene. In other words, the route information contains information that characterizes an area in the scene, preferably with regard to its impact on driving safety. Advantageously, the determination or (subsequent) adaptation of the virtual scene representation is then carried out based on the scene information. The scene information enables improved safety of the journey, even in potentially dangerous sections of the route.

[0038] For example, scene information can identify areas where objects should never be considered dangerous. This could include areas separated from the vehicle's route, such as the track bed, by a barrier, like a structural element. Scene information can be used to determine whether objects detected in the captured scene can be removed from the virtual scene representation or whether they should be included at all. Conversely, scene information can also highlight areas where there is a particularly high risk. For instance, during scene analysis, special attention can be paid to an area identified by scene information, such as a train platform.

[0039] In a further preferred embodiment, the route information includes interaction information about the interaction between at least two objects in the captured scene. For example, the route information can contain information about a barrier in the vehicle's surroundings that restricts the freedom of movement of other objects in the vehicle's surroundings. The database might contain information such as the fact that a footpath running parallel to the route is separated from it, for example, a railway track, by a fence, preventing pedestrians from entering the route. Similarly, the database could contain the position of a tree standing at a level crossing, whose shadow makes it difficult to detect people on the crossing. This makes it possible to consider the impact of specific objects on their surroundings in a differentiated manner, possibly even during the sensor-based capture of the scene.

[0040] Preferably, the determination, and in particular the (subsequent) adaptation, of the virtual scene representation is carried out taking into account the interaction information. In the case of the aforementioned examples, for instance, people on the footpath could be removed from the virtual scene representation or marked as harmless, or the scene representation in the shade of the tree could be captured and / or processed with other parameters or sensors.

[0041] In a further preferred embodiment, a safety zone is determined in the captured scene based on the route information, in particular the interaction information, within which the risk to the journey posed by an object in the scene is at least reduced. A safety zone can also be understood as a protected area in which objects cannot impair driving safety. A safety zone can be formed, for example, by a barrier such as a fence, a wall, a gate, a safety line on a platform, and / or the like. The safety zone is expediently located on the side of the barrier facing away from the route. Objects captured in the scene that are located within the safety zone can accordingly be removed from the scene representation or marked as harmless.Alternatively, at least their risk assessment can be adjusted, for example, a risk score assigned to them can be reduced, at least temporarily.

[0042] In a further preferred embodiment, the effect of the safety zone on the trajectory of an object from the scene is determined. This effect on the trajectory is expediently taken into account during the determination, and in particular the (subsequent) adaptation, of the virtual scene representation. Preferably, the effect of the safety zone on an object depends on the type information corresponding to the object, i.e., the object type. For example, bollards can slow down a cyclist, while a pedestrian can continue essentially unhindered, but a car will be stopped. The effects of a barrier on moving objects can thus be considered in a differentiated manner.

[0043] The effect on the trajectory can also depend on the type information corresponding to the barrier. Depending on the type of barrier, the scene adaptation can be based, for example, on a strong restriction of object movement, such as by a fence or a barrier, or on a weak restriction, such as by a safety line on a platform. It is useful to include information about the strength of the effect in the interaction information.

[0044] In a further preferred embodiment, the route information is validated while the vehicle is traveling along the route. Preferably, during the journey, it is specifically checked whether an object in the detected scene has the expected effects on other objects in the scene – for example, according to the interaction information. This allows, for instance, verification that a barrier or a safety zone defined by it is intact, i.e., whether it actually restricts the movement of other objects, particularly to the expected extent. Such validation of the route information allows for even greater safety for the vehicle's journey, and especially for future journeys of other vehicles on the same route.Alternatively or additionally, it is also conceivable that if validation fails, appropriate safety measures are initiated, such as correcting the route information and / or initiating an emergency stop.

[0045] In a further preferred embodiment, the route information is validated by comparing the temporal evolution of the detected scene with a prediction based on the route information. For example, a determined ("measured") trajectory of a cyclist detected in the vicinity of a barrier can be compared with a trajectory of the cyclist predicted, for instance, based on the interaction information. If there is a match, it can be assumed that the route information is correct.

[0046] In a further preferred embodiment, (i) additional object information specific to an object from the captured scene is received, and (ii) the virtual scene representation is determined, and in particular adapted, based on this additional object information. Such additional object information can be sent, for example, by objects from the scene, such as other road users, traffic signals, personnel working on the route, and / or the like. Considering this additional object information is particularly advantageous because it can be trusted with a high degree of reliability. In other words, the scene representation can be adapted based on highly reliable information.

[0047] It is preferred that the determined virtual scene representation be transmitted to other road users, for example, an oncoming train. Preferably, information about the reliability of the determination of the virtual scene representation is also transmitted during the transmission. This facilitates the scene analysis by the other road user.

[0048] In a further preferred embodiment, the additional object information includes state information that characterizes the state of an object in the captured scene. This state information preferably allows for an assessment of the current hazard posed by the vehicle to the vehicle and / or the object. Such additional information could, for example, relate to the open / closed state of a barrier: if the barrier is closed, the hazard is low; if the barrier is open, the hazard is high. The additional information could also include the fact that personnel working along the route have perceived and confirmed the approach of the vehicle.However, status information can also characterize a dynamic state, for example, the acceleration of a road user: if the road user brakes before a level crossing, the risk is low; however, if the road user accelerates or maintains its speed, the risk is high. Information about dynamic objects can also come from a trackside system that monitors a certain area and transmits the information about dynamic objects with quality assurance.

[0049] In another preferred embodiment, the route information includes lighting information. This lighting information can, for example, include information about the shadows cast by objects in the captured scene, glare effects, reflections, brightness gradients, and / or the like, which are to be expected, for instance, when the scene is sensor-acquired. Advantageously, lighting-related effects are taken into account when determining the virtual scene representation, and optionally even during the sensor-acquired scene. For example, shaded areas in the scene representation can be illuminated, or objects can be identified as reflections of another object and removed from the scene representation or not captured at all. Thus, even seemingly complex decision-making situations can be reliably resolved or at least simplified.

[0050] A second aspect of the invention relates to a system for monitoring a vehicle's surroundings while a vehicle, in particular a rail vehicle, is traveling on a route, especially a predetermined one. The system is configured to perform the method according to the first aspect of the invention and includes a sensor device configured to capture a scene and generate corresponding sensor data, and a database in which quality-assured route information is stored. Furthermore, a data processing device is provided, which is configured to generate a virtual scene representation based on the sensor data and the route information.

[0051] Such a system can be particularly advantageous in the case of a rail vehicle to ensure the safe movement of the rail vehicle on a predetermined route, i.e. along rails, and / or to avoid or at least reduce the danger to objects in the vicinity of the vehicle or, more generally, the vicinity of the route.

[0052] A vehicle, in particular a rail vehicle, according to a third aspect of the invention comprises a system according to the second aspect of the invention.

[0053] The preceding description of preferred embodiments of the invention contains numerous features, some of which are summarized in the individual dependent claims. However, these features can also be advantageously considered individually and combined into meaningful further combinations. In particular, these features can each be combined individually and in any suitable combination with the method according to the first aspect of the invention, the system according to the second aspect of the invention, and the vehicle according to the third aspect of the invention. Thus, method features can also be considered as properties of the corresponding device unit, and vice versa.

[0054] The properties, features, and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more readily understandable in connection with the following description of the exemplary embodiments, which are explained in more detail in conjunction with the drawings. The exemplary embodiments serve to illustrate the invention and do not limit it to the combination of features specified therein, including functional features. Furthermore, suitable features of each exemplary embodiment can also be considered explicitly in isolation, removed from one exemplary embodiment, incorporated into another exemplary embodiment to complement it, and combined with any of the claims.

[0055] They show: FIG 1 is an example of a scene from the environment of a vehicle; FIG 2 is an example of scene information specific to an area of ​​a scene; FIG 3 is an example of a system for monitoring a vehicle environment; and FIG 4 is an example of the use of quality-assured route information stored in a database.

[0056] FIG 1 Figure 1 shows an example of a scene from the environment 10 of a vehicle 1. Vehicle 1 is represented here as a rail vehicle traveling on a route 2 defined by a railway track. The scene is characterized by objects 11 from the environment 10 of vehicle 1 – or, more generally, from the environment of route 2. In this example, the environment 10 of vehicle 1 includes a tree 11a, an embankment 11b, a footpath 11c, birds 11d, a signal 11e, a bridge 11f, and people 11g.

[0057] The vehicle 1 is equipped with a sensor device 51 with which it can detect its environment 10, in particular the current scene, also referred to as the driving situation. The information obtained during detection about the vehicle's environment 10, for example about the objects 11 constituting the scene, can form the basis for controlling the vehicle 1 or at least individual vehicle functions. Such information can be used in particular by assistance systems to move the vehicle 1 safely along route 2, i.e., without endangering the vehicle 1 or objects 11 in the vehicle's environment 10.

[0058] The information obtained during data acquisition can also be understood as a virtual scene representation, since it characterizes the real scene. Such a virtual scene representation can be derived, for example, from sensor data generated by the sensor device 51.

[0059] Since the analysis of such a virtual scene representation, for example with regard to control measures to be taken, is generally very time-consuming and complex, it is expedient to use quality-assured route information stored in a database to determine, and in particular to adapt, the scene representation. This allows consideration of the fact that not all objects 11 in the vicinity 10 of vehicle 1 typically impair (or can impair) the driving safety of vehicle 1.

[0060] For example, stationary objects such as tree 11a, embankment 11b, footpath 11c, or bridge 11f do not pose a hazard to rail vehicle 1. Therefore, they require little or no attention during scene analysis. In other words, these stationary objects can be removed from the scene representation or at least disregarded during a hazard assessment based on the virtual scene representation.

[0061] In contrast, dynamic objects such as the birds 11d or the people 11g potentially pose a danger to vehicle 1 – or can be endangered by vehicle 1 – as they can move into the path of vehicle 1. These dynamic objects may therefore receive increased attention during scene analysis.

[0062] Similarly, objects 11 specifically designed for the operation of vehicle 1, such as the signaling system 11e, are also important for driving safety. Such objects 11, which can be designated as operational objects, should also be considered in the scene analysis.

[0063] Route information for the stationary objects 11, which are located in the vehicle's surroundings 10 during every journey of a vehicle 1 on route 2, or which are detected by the sensor device 51 during every journey of the vehicle 1 along route 2, can be directly stored in the database. In the virtual scene representation, it can be easily checked, based on the route information, whether an object 11 detected in the surroundings 10 is stationary or dynamic – for example, by checking whether its position relative to the vehicle 1 or to route 2 is characterized by the route information. The association of a stationary object 11 with a fixed position can be interpreted as a hazard criterion, based on which stationary and / or non-stationary objects 11 can be marked accordingly in the scene representation (i.e., marked with a corresponding "flag") or even completely removed from the scene representation.This marking makes it possible to determine, when assessing driving safety - and consequently also when generating appropriate control signals for controlling the vehicle 1 - whether and / or which of the objects 11 should be taken into account or can be disregarded.

[0064] It is conceivable that selected types of dynamic objects 11d, 11g in the virtual scene representation could also be marked as harmless or removed from the virtual scene representation. For example, it can be assumed that the birds 11d cannot impair the driving safety of vehicle 1. The route information can therefore contain corresponding type information specific to an object type. This type information thus represents generic information that can improve, and in particular simplify, the analysis of the virtual scene representation with regard to dynamic objects 11d, 11g.

[0065] But even potentially dangerous objects 11 such as persons 11g can be excluded from a hazard assessment under certain circumstances, as in connection with FIG 2 will be explained in more detail.

[0066] FIG 2 Figure 1 shows an example of scene information specific to area 12a and 12b of a scene. The scene is characterized by objects 11h and 11i in the vicinity 10 of a vehicle 1 on a route 2. These objects are two fences 11h and two barriers 11i, positioned along a track forming route 2, to ensure the safety of vehicle 1 – which in this case is a rail-bound vehicle.

[0067] Both the fences 11h and the barriers 11i are stationary objects that restrict the freedom of movement of dynamic objects. Such dynamic objects, for example, pedestrians, cyclists, or even cars located in one of the areas 12a or 12b, cannot enter the path of vehicle 1. Areas 12a and 12b can therefore also be referred to as safety areas 13 or protection zones. The stationary objects 11h and 11i, which define areas 12a and 12b or safety areas 13, can accordingly also be referred to as barriers.

[0068] Route information, which creates a virtual representation of the route. FIG 2 The route information, which can be adapted to the scene shown, preferably contains information regarding such barriers or areas 12a, 12b. At a minimum, however, it is advantageous if such information for restricting the freedom of movement of other, especially dynamic, objects can be derived from the route information. For example, the route information can contain interaction information that characterizes the effects of a (stationary) object, such as the fences 11h and / or the barriers 11i, i.e., in particular a barrier, on other objects. Accordingly, dynamic objects can be disregarded when assessing the driving safety of vehicle 1 based on a virtual scene representation if they are located in one of the areas 12a, 12b or safety zones 13.

[0069] It is also conceivable that objects from the (real-world) scene provide additional object information that supplements the route information, allowing the virtual scene representation to be determined or adapted in a more differentiated way. In the present example, the barriers 11i could, for instance, provide object information characterizing their open state using a communication device 14a. Vehicle 1 can receive this object information with a corresponding communication device 14b. Depending on this object information, a decision can then be made as to whether a dynamic object in the areas 12b assigned to the barriers 11i is currently prevented from moving into the path, i.e., onto route 2, or not.

[0070] FIG 3 Figure 50 shows an example of a system 50 for monitoring a vehicle's surroundings 10 while a vehicle, in particular a rail vehicle, is traveling on a route. The system 50 comprises a sensor device 51, a database 52, and a data processing device 53.

[0071] The sensor device 51 is configured to detect a scene from the vehicle's surroundings 10. For example, the sensor device 51 can detect objects 11 from the vehicle's surroundings 10. The sensor device 51 expediently generates corresponding sensor data, which can be processed, for example, by the data processing device 53. The sensor device 51 expediently includes an optical sensor device, such as one or more cameras, a radar device, a lidar device, and / or the like.

[0072] The data processing device 53 is configured to determine a virtual scene representation of the sensor-detected scene based on the sensor data. Advantageously, the data processing device 53 is configured to recognize the objects 11 in the vehicle's environment 10 based on the sensor data. Preferably, the data processing device 53 can also characterize the recognized objects 11, i.e., derive properties such as size, shape, texture, color, position relative to other objects and / or the vehicle, temperature, and / or the like from the sensor data. In other words, the data processing device 53 is preferably configured to determine a so-called virtual feature vector with a plurality of entries for each of the objects 11 in the detected scene, which characterizes the respective object 11 in detail in the scene representation.

[0073] Database 52 contains quality-assured route information. This route information preferably characterizes the route along which the vehicle travels. The route information can, for example, include information about stationary objects that appear in the vicinity of the vehicle or are detected by the sensor device 51 while the vehicle is moving along the route. Advantageously, database 52 contains a detailed feature vector for each object 11, which allows, for example, the unambiguous identification of the respective object 11 and / or an assessment of any potential hazard to the vehicle's journey.

[0074] Alternatively or additionally, the route information can also contain information about dynamic objects that might appear in the vicinity of the vehicle, such as generic information about other road users, living beings and / or the like.

[0075] The data processing device 53 is preferably configured to adapt the virtual scene representation based on the route information, for example, to simplify and / or supplement it. For example, based on the route information, the data processing device 53 can highlight objects 11 detected in the environment 10 in the virtual scene representation as significant for driving safety or mark them as insignificant.

[0076] It is also conceivable that the data processing device 53 receives additional object information from objects 11 in the real-world scene, which supplements the route information. Such object information could, for example, contain information about a state of object 11, such as the open state of a barrier, the acceleration of another road user, and / or the like. With this additional object information, the data processing device 53 can make a particularly reliable and detailed adjustment of the virtual scene representation.

[0077] System 50 can be installed in the vehicle whose environment 10 is to be monitored. Alternatively, at least part of System 50 can be located outside the vehicle. In particular, System 50 can be configured to be distributed between the vehicle and a land-based system. For example, Database 52 could be a central database where route information is stored on a server and can be accessed, for example, via a wireless communication link. Similarly, Data Processing Device 53 could be a central or at least external data processing device with which the vehicle can be connected via a wireless communication link.

[0078] FIG 4Figure 52 shows an example of the use of quality-assured route information stored in a database. Here, an object 11e in the vehicle's surroundings 10 is detected by a sensor device 51 on a vehicle 1 traveling along a route 2. From the sensor data generated, features of the object 11e – in this case, purely as an example, a traffic signal – can be derived using an object recognition algorithm, in particular an artificial intelligence. If necessary, background information about the vehicle's surroundings 10 can also be incorporated into the derivation. Some features can be determined, for example, by image processing such as edge detection, mathematical morphology, Fourier, Hough and / or Gabor transforms, wavelet transforms, and / or the like. Alternatively or additionally, features can be extracted from neural networks or recognized by optical character recognition (OCR).It is also possible to analyze 2.5D or 3D shapes of object 11e, possibly including substructures and / or information obtained from motion analysis about object 11e's joints, material types, size / extent, and / or estimated weight to determine features. It is also conceivable to determine features from normalized histograms of such quantities, for example, substructures of the object's shape. The features, e.g., size V'1, shape V'2, color V'3, position V'4 relative to route 2, and / or the like, together form a feature vector V'. The feature vector V' thus characterizes the captured object 11e.

[0079] In the present example, the route information also contains a feature vector V. The feature vector V indicates that object 11e is located at a specific point on route 2 and is characterized by features V1, V2, V3, V4, ... By comparing the feature vectors V' and V, it can be demonstrated, for example, that (exactly) object 11e was detected at the expected location. In particular, the verified quality of the route information can be transferred to or associated with the object detection result. The information obtained by detecting object 11e, with the quality determined in the security verification, can be used in further processing.

[0080] Furthermore, it is possible to demonstrate, with minimal effort, that the hardware and software are functioning correctly. In particular, it is possible to rule out the presence of errors, at least for a specified period, that could lead to system malfunctions. If a fault budget associated with a tolerated risk is provided, this can also compensate for a cluster of errors in another part of the system.

[0081] In the comparison W, it can be checked in particular whether an object type T or subtypes T1, ..., Tn of object 11e or an instance I of object 11e correspond to each other or at least to a predefined degree. The object type T and the subtypes T1, ..., Tn and the instance I preferably result from groups of features of the feature vectors V, V', as indicated by the parentheses.

[0082] An algorithm for performing the comparison W should be designed according to the requirements of the intended safety requirements and be based on mathematical and statistical theories such as probability theory, in particular conditional probabilities. Background knowledge about the vehicle's environment 10 can also be incorporated into the comparison W, for example, regarding the probability distribution of the feature values ​​in the captured scene, the dependence or independence of the features, the probability that features cannot be determined due to the current vehicle position, obstruction by another object, weather or other environmental influences, technical failures in data processing, and / or the like.

[0083] Although the invention has been further illustrated and described in detail by the preferred embodiments, the invention is not limited by the disclosed examples and other variations can be derived by the person skilled in the art without departing from the scope of protection of the invention according to the patent claims.

Claims

1. Method for monitoring a vehicle environment (10) when a vehicle (1), in particular rail vehicle, is travelling on a route (2), with the steps: - detecting a scene in the vehicle environment (10) by means of sensors and generating corresponding sensor data; and - generating a virtual scene representation of the detected scene on the basis of the sensor data and in a database (52) of stored, quality-assured route information, wherein, for objects (11, 11a-11i) identified in the detected scene, it is checked on the basis of the route information whether a predefined threat criterion is met, and the objects (11, 11a-11i) are taken into consideration as a function of a result of the check when ascertaining the virtual scene representation, wherein the route information contains an item of type information specific to an object type and all objects (11, 11a-11i) of an object type are taken into consideration when ascertaining the virtual scene representation according to the type information.

2. Method according to claim 1, wherein the route information is provided as a function of at least one travel parameter.

3. Method according to one of claims 1 or 2, wherein the ascertained virtual scene representation is based on an assessment of the travel safety.

4. Method according to one of the preceding claims, wherein the route information contains an item of scene information specific to a region (12a, 12b) in the detected scene and the adaptation of the virtual scene representation is carried out on the basis of the scene information.

5. Method according to one of the preceding claims, wherein the route information contains an item of interaction information relating to the interaction between at least two objects (11, 11a-11i) in the detected scene and the adaptation of the virtual scene representation is carried out while taking into consideration the interaction information.

6. Method according to one of the preceding claims, wherein the route information is used to ascertain a safety region (13) in the detected scene, in which the threat to the travel is at least lessened by an object (11, 11a-11i) from the scene.

7. Method according to claim 6, wherein the effect of the safety region (13) on a trajectory of an object (11, 11a-11i) from the scene is ascertained and is taken into consideration when adapting the virtual scene representation.

8. Method according to one of the preceding claims, wherein the route information is validated during the travel of the vehicle (1) on the travel route (2).

9. Method according to claim 8, wherein the route information is validated on the basis of a comparison of the temporal development of the detected scene with a prediction on the basis of the route information.

10. Method according to one of the preceding claims, having: - receiving additional object information specific to an object (11, 11a-11i) from the detected scene; and - ascertaining the virtual scene representation on the basis of the additional object information.

11. Method according to one of the preceding claims, wherein the route information contains an item of illumination information and effects caused by illumination are taken into consideration when adapting the virtual scene representation.

12. System (50) for monitoring a vehicle environment (10) when a vehicle (1), in particular rail vehicle, is travelling on a route (2), which system (50) is configured to carry out the method according to one of the preceding claims, with - a sensor apparatus (51), which is configured to detect a scene and generate corresponding sensor data, - a database (52), in which an item of quality-assured route information is stored; and - a data processing apparatus (53), which is configured to generate a virtual scene representation on the basis of the sensor data and the route information, wherein the data processing apparatus (53) is configured, for objects (11, 11a-11i) identified in the detected scene, to check on the basis of the route information whether a predefined threat criterion is met, and to take into consideration the objects (11, 11a-11i) as a function of a result of the check when ascertaining the virtual scene representation, wherein the route information contains an item of type information specific to an object type and all objects (11, 11a-11i) of an object type are taken into consideration when ascertaining the virtual scene representation according to the type information.

13. Vehicle (1), in particular rail vehicle, with a system (50) according to claim 12.