Method for detecting false images in sensor measurements

Abstract

A new method is presented for detecting spurious objects in sensor measurements around a vehicle. The present invention relates to a method for detecting spurious objects in sensor measurements around a vehicle. According to the present invention, map context information is obtained from a digital road map. Objects (O) with associated attributes are detected around the vehicle and mutual social context information of the objects (O) is generated. All available data about the traffic situation of the vehicle and the objects (O) are stored in a graph structure (GS) with nodes (K1-Km) and edges (E1-En), and relationship information (RI) is represented in the graph structure (GS) by the edges (E1-En). Taking into account the map context information and the social context information, anomalies and patterns are detected among the features of the graph structure (GS) and spurious objects are classified based on the detected anomalies and patterns.

Description

Technical Field

[0001] The present invention relates to a method for detecting ghost objects in sensor measurements around a vehicle.

Background Art

[0002] From the following Patent Document 1, a method for a vehicle radar system is known and has the steps described below. That is, - Using a vehicle radar system, detect two or more objects, - In a track database, start tracks for two or more objects, where each track stores data regarding two or more objects and is updated based on additional detections of the two or more objects. The tracks of the two or more objects are initially unclassified tracks in the track database. - Using a processor, select two tracks corresponding to two of the two or more objects in the track database as a candidate pair. - Apply criteria to the candidate pair using a processor to determine whether one of the two tracks of the candidate pair is a track of a ghost object resulting from multipath reflection and the other of the two tracks of the candidate pair is a track of a real object corresponding to the ghost object. Here, the ghost object records the real object at a wrong position. - Based on the determination that one of the two tracks of the candidate pair is a track of a ghost object and the other of the two tracks of the candidate pair is a track of a real object corresponding to the ghost object, use a processor to classify the candidate pair in the track database as a track of a real object and a ghost object pair. - Information from the track database is reported, and this report includes providing data only for the tracks of real objects based on the classification. Those information are used to control the operation of the vehicle. Patent Document 2 below describes a method for detecting the surroundings using radar, in which objects outside the radar's line of sight are detected. In this case, in one processing step, the raw radar data detected by the radar is preprocessed and adapted for subsequent processing steps. In subsequent processing steps, radar reflections are determined from the preprocessed raw radar data using a classification algorithm. Further subsequent processing steps generate a topology around the radar based on the radar reflections, and in yet another processing step, objects that are directly visible and objects located outside the radar's line of sight are distinguished in the topology around the radar based on the radar detections. Furthermore, Patent Document 3 below describes a method for identifying map attributes and map relationships of objects. In this case, dynamic information, static information including geographical spatial distance between objects, semantic information, and relationship information are displayed in a graph. Map attributes and map relationships are learned from the graph via a graph neural network, and the map attributes and map relationships to be learned are used to train the graph neural network, either by manually labeling the map or by automatically generating geometric information. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] German Patent Application Publication No. 10, 2020, 124, 236, Specification A1 [Patent Document 2] German Patent Application Publication No. 10 2021 001 452 Specification A1 [Patent Document 3] German Patent Application Publication No. 10 2021 005 084 Specification A1 [Overview of the project] [Problems that the invention aims to solve]

[0004] The problem that this invention aims to solve is to provide a novel method for detecting false images in sensor measurements around a vehicle. [Means for solving the problem]

[0005] This problem is solved by a method having the features described in claim 1, based on the present invention.

[0006] Advantageous embodiments of the present invention are subject to the dependent claims.

[0007] A method for detecting false images in sensor measurements around a vehicle , Contextual information is obtained from the digital road map, and objects with relevant attributes are detected around the vehicle. ru. According to the present invention, between objects The mutual social context information The generated social context information is derived from the relationships between objects present around the vehicle.The system generates a graph structure containing all available traffic data for vehicles and objects, with relational information represented by the edges. Anomalies and patterns are searched for within the graph structure's features, taking into account the map's contextual and social contextual information. Based on the detected anomalies and patterns, the system classifies the false objects. In sensor measurements using multiple sensors, object hypotheses determined from the sensor measurements of each individual sensor are considered, and based on at least one object hypothesis from at least one sensor measurement of the sensor, it is checked whether the detection of a false image object in a sensor measurement performed by at least one other sensor is valid or not.

[0008] Autonomous driving, especially highly automated or autonomous driving, requires precise detection of objects around the vehicle. For example, to comply with SAE J3016 standards Level 2, Level 3, Level 4, and Level 5 and to meet the high safety requirements for autonomous driving, sensor measurements using sensor modalities such as camera sensors, radar sensors, and / or lidar sensors are necessary. In this case, the sensor modalities often form redundant sensor systems with multiple similar and / or different sensor modalities. Sensor measurements of these sensor modalities are used individually or in an integrated manner.

[0009] To maximize safety, the delay time in a vehicle's reaction to an object must be minimized. For this reason, even slight sensor readings or individual sensor readings are designed to trigger a system reaction. Among sensor readings, there are not only true positive readings but also false positive readings, where an object is detected in a place where there is actually nothing. False positive readings can occur, for example, due to undesirable sensor reflections, false detections, and / or the detection of surrounding objects, such as a vehicle depicted in a poster advertisement. Typical examples include radar reflections from guardrails and traffic signs.

[0010] In autonomous vehicles, objects are generated from such false positive measurements. Because these objects do not actually exist, they are generally called false images. When an autonomous vehicle reacts to a false image, it can trigger dangerous situations such as emergency braking, potentially leading to a collision.

[0011] By using this method, it is possible to detect false object images particularly effectively and reliably. In this method, by using information on all objects in the traffic situation or traffic scene, it is possible to determine whether the attributes of a certain object show abnormalities compared to surrounding objects. In this method, by using context information to comprehensively represent the scene, context-based classification of objects can be achieved, and in this classification, for example, the underlying road network is used. Also, by using this method, it is possible to consider the temporal aspect and extract patterns over the temporal dimension.

[0012] Therefore, this method enables recording the surroundings of the vehicle with high sensitivity and then recognizing and deleting false object images, that is, false positive measurements. Thereby, the risk that real objects in the surroundings are not detected is reduced. That is, the probabilities of false negative measurements and false positive measurements are significantly reduced.

[0013] In a possible embodiment of this method, by providing an extension to learning-based classification, it becomes possible to extract complex patterns that cannot be defined manually. Furthermore, as a result, this method and the system that executes it can be extended according to the data.

[0014] Hereinafter, embodiments of the present invention will be described in detail based on the drawings.

Brief Explanation of Drawings

[0015] [Figure 1] It is a schematic view of the traffic situation seen from above. [Figure 2] It is a schematic view of the traffic situation seen from above according to FIG. 1 in which nodes of a graph structure are generated. [Figure 3] It is a schematic view of the nodes according to FIG. 2 of the graph structure and the edges connecting them. [Figure 4] It is a schematic view of a graph structure showing the relationship between a plurality of nodes and a plurality of nodes. [Figure 5]It is a schematic diagram showing the configuration of a graph structure and the processing of existing information in the graph structure by a graph-based artificial neural network.

Mode for Carrying Out the Invention

[0016] In any of the figures, the corresponding parts are denoted by the same reference numerals.

[0017] FIG. 1 is a view of the traffic situation at an intersection seen from above. In this intersection, there are objects O6 to O11 formed as lane segments derived from a digital road map. Further, in the area of this intersection, there are an object O3 formed as a traffic signal, an object O4 formed as a stop sign, an object O5 formed as a crosswalk, and two objects O1 and O2 formed as vehicles.

[0018] In order to perform autonomous driving, particularly highly autonomous driving or self-driving, it is necessary to accurately detect the objects O1 to O11 around the vehicle. For this purpose, an autonomous vehicle has a surrounding detection sensor system equipped with, for example, a lidar, a radar, a camera, and an ultrasonic sensor. At this time, each sensor modality may exist redundantly. When recording the surroundings, the sensor measurements of the sensor modalities are used individually or in an integrated form. Here, it is necessary to avoid false negative measurement results, that is, the situation where the actual objects O1 to O11 are not detected in the surroundings, and false positive measurement results, that is, the situation where false objects O1 to O11, so-called false image objects, are detected.

[0019] When recording the surroundings with low sensitivity, there is a risk that all the actual objects O1 to O11 are not detected in the surroundings, and as a result, there may be a dangerous situation in the autonomous driving of the vehicle.

[0020] On the contrary, when recording the surroundings with high sensitivity, there is a risk that false image objects that do not exist in the surroundings are detected in this surroundings. As a result, there may also be a dangerous situation in the autonomous driving of the vehicle.

[0021] In order to determine whether objects O1 to O11 actually exist, the method for detecting false images in sensor measurements uses contextual information of traffic conditions or traffic scenes. This makes it possible to first record the surroundings with high sensitivity, then detect false images from the recorded sensor measurements (measurement results), and remove the false images from those measurements (measurement results).

[0022] Here, important information is provided by the road network that forms the basis of traffic conditions, and this road network is obtained as map context information from a digital road map. In this case, the road network is represented by objects O6-O11, which are specifically formed as lane segments.

[0023] Further important information is provided by other vehicles in the vicinity of the vehicle, which are represented here, for example, by objects O1 and O2. Social context information is generated from the relationship between objects O1 and O2.

[0024] To achieve high sensitivity when recording the surroundings, this method stores all available traffic data in a graph structure GS, also known as a scene graph, which is shown in detail in Figure 5. In this graph structure GS, relational information is represented by edges E1 to En connecting nodes K1 to Km.

[0025] Figure 2 is a top view of the traffic scene from Figure 1, showing the generated nodes K1 to K11 in possible embodiments of the graph structure GS.

[0026] To create the graph structure GS, nodes K1 to Km are formed for all available dynamic information DI, static information SI, and semantic information SEI (each shown in detail in Figure 5).

[0027] In this case, static information SI includes, for example, objects O6 to O11 formed as lane segments, dynamic information DI includes, for example, objects O1 and O2 formed as vehicles and their tracks, and semantic information SEI includes, for example, object O4 formed as a stop sign, object O3 formed as an indicator light such as a traffic light, object O5 formed as a pedestrian crossing, and other traffic control equipment.

[0028] Here, for example, each lane segment (objects O6 to O11) is assigned nodes K6 to K11, the primary stop sign (object O4) is assigned node K4, the pedestrian crossing (object O5) is assigned node K5, the traffic light (object O3) is assigned node K3, and objects O1 and O2, which are formed as vehicles, are assigned nodes K1 and K2, respectively.

[0029] Figure 3 shows a possible embodiment of the graph structure GS, with nodes K1 to K11 and the edges E1 to E14 connecting them, as shown in Figure 2.

[0030] In this case, edges E1 to E14 represent the relationship information RI, which is also illustrated in detail in Figure 5, namely the dependencies and relationships between nodes K6 to K11.

[0031] For example, edge E1 represents, as relational information RI, that object O8 (node ​​K8), which is formed as a lane segment, is connected to object O6 (node ​​K6), which is also formed as a lane segment. For example, when passing through an intersection, the fact that object O8 is following object O6 is stored as a relational attribute.

[0032] For example, edge E2 indicates, as relational information RI, that object O10 (node ​​K10), which is formed as a lane segment, precedes object O8 (node ​​K8), which is also formed as a lane segment. For example, it is stored as a relational attribute that object O10 is the first preceding object O6.

[0033] For example, edge E3 represents, as relational information RI, that object O6 (node ​​K6), which is formed as a lane segment, and object O11 (node ​​K11), which is also formed as a lane segment, are connected. For example, it is stored as a relational attribute that object O6 is to the right of object O11.

[0034] For example, edge E4 represents, as relational information RI, that object O11 (node ​​K11), which is formed as a lane segment, and object O6 (node ​​K6), which is also formed as a lane segment, are connected. For example, it is stored as a relational attribute that object O11 is to the left of object O6.

[0035] For example, edge E5 indicates, as relational information RI, that the regulations of the traffic light (object O3, node K3) are applied to object O6 (node ​​K6), which is formed as a lane segment.

[0036] For example, edge E6 represents, as relational information RI, that object O11 (node ​​K11), which is formed as a lane segment, is located beneath object O1 (node ​​K1), which is formed as a vehicle. For example, the fact that a lane segment is located beneath a vehicle with a specific probability (e.g., 1.0) is stored as a relation-specific attribute.

[0037] For example, edge E7 represents, as relational information RI, that object O7 (node ​​K7), which is formed as a lane segment, and object O6 (node ​​K6), which is also formed as a lane segment, are connected. For example, when passing through an intersection, the fact that object O7 is following object O6 is stored as a relation-specific attribute.

[0038] For example, edge E8 represents, as relational information RI, that object O7 (node ​​K7), which is formed as a lane segment, is located beneath object O2 (node ​​K2), which is formed as a vehicle. For instance, the fact that a lane segment is located beneath a vehicle with a specific probability (e.g., 0.7) is stored as a relation-specific attribute.

[0039] For example, edge E9 represents, as relational information RI, that object O8 (node ​​K8), which is formed as a lane segment, is located beneath object O2 (node ​​K2), which is formed as a vehicle. For instance, the fact that a lane segment is located beneath a vehicle with a specific probability (e.g., 0.3) is stored as a relation-specific attribute.

[0040] For example, edge E10 represents, as relational information RI, that object O10 (node ​​K10), which is formed as a lane segment, and object O9 (node ​​K9), which is also formed as a lane segment, are connected. For example, the fact that object O10 is followed by object O9 is stored as a relational attribute.

[0041] For example, edge E11 indicates, as relational information RI, that the regulations of the stop sign (object O4, node K4) apply to object O9 (node ​​K9), which is formed as a lane segment.

[0042] For example, edge E12 represents, as relational information RI, that the pedestrian crossing (object O5, node K5) passes over object O9 (node ​​K9), which is formed as a lane segment, meaning that it overlaps with it.

[0043] For example, edges E13 and E14 represent, as relational information RI, that objects O1 and O2 (nodes K1 and K2) formed as a vehicle interact with each other. Relational specific attributes are, for example, - The geometric relationship between objects O1 and O2, for example, the positional difference between vehicles in meters. - The geometric relationship between objects O1 and O2, for example, the difference in yaw angles between vehicles, i.e., the difference in the orientation of the vehicles. - The kinematic relationship between objects O1 and O2, for example, the speed difference between vehicles. That is the case.

[0044] The aforementioned objects O1-O11, relational information RI, and attributes are just examples. The approach described here is not limited to the aforementioned object types, relational information RI, and attributes, and can be arbitrarily extended to include other possible object types, relational information RI, and attributes.

[0045] By considering contextual information, anomalies and patterns can be discovered (detected) within the graph structure GS, and these anomalies and patterns are used to classify false images. For example, attributes of objects O1-O11 detected (recorded) using their respective sensor measurements are used as features for classifying objects O1-O11. These attributes can consider, for example, the type of each object O1-O11, such as a vehicle, road sign, pedestrian, etc., and / or the type of each object O1-O11. For example, in the case of objects O1-O11 having the properties of a vehicle, possible object types include passenger cars, trucks, buses, motorcycles, etc. Furthermore, the attributes may include information representing the objects O1-O11, such as their kinematic state and / or positional information.

[0046] In other words, the sensor measurements (measurement results) for each of the objects O1 to O11 being classified include not only the objects O1 to O11 themselves, but also the unique attributes of each object that are useful for classification.

[0047] Furthermore, as a characteristic for classifying objects O1 to O11, the attributes of the relationships between objects O1 to O11 include, for example, - Geometric relationships between objects O1 and O11, and / or - The geometric relationship between objects O1 to O11 and their positions around them, and / or - The kinematic relationship between dynamic objects O1~O11, and / or - Semantic relationships between objects O1~O11, and / or - Static relationship between objects O1 and O11 You can use it.

[0048] For example, to classify objects O1 to O11, we can use attributes that describe the relationship between objects O1 to O11, and these attributes are, for example, -temporal information, - The distance between objects O1 to O11 and the vehicle at the time of initial detection of objects O1 to O11 and / or the first sensor measurement of a series of measurements, - Velocities of objects O1 to O11 in comparison with surrounding objects O1 to O11, - The positions of objects O1~O11 in comparison with the road network, and / or - Direction of objects O1~O11 in comparison with the direction of the lane segment below. It includes.

[0049] This means, for example, that the initial sensor measurement (measurement result) can be classified. However, multiple "initial" sensor measurements (measurement results) may be classified. Autonomous vehicles have a requirement regarding how quickly they must react to surrounding objects O1-O11. If multiple sensor measurements are taken by the same sensor within this time, multiple "initial" sensor measurements (measurement results) can also be used for classification. Therefore, since the sensor measurements (measurement results) from this time can also be used, the historical context of objects O1-O11 can be determined and useful patterns can be provided. This makes it possible, for example, to record the vehicle's behavior at this time and to consider it before the actual objects O1-O11 are detected.

[0050] Furthermore, in a conceivable embodiment, in sensor measurements using multiple sensors, i.e., redundant sensors, object hypotheses determined from the sensor measurements of each individual sensor are taken into consideration, and a check is provided to determine whether the detection of a false image object in a sensor measurement performed using at least one other sensor is valid or not, based on at least one object hypothesis from at least one sensor measurement of one sensor.

[0051] Figure 4 illustrates possible embodiments of a graph structure GS having multiple nodes K1 to Km and edges E1 to En representing the relationships between nodes K1 to Km.

[0052] In this case, node K1 is, for example, a pedestrian crossing, node K2 is a warning light, node K3 is a lane, node K4 is a road user, and node Km is a stop sign.

[0053] Edges E1 to En are formed between nodes K1 to Km, and these reflect the relationship between nodes K1 to Km, i.e., relational information RI.

[0054] For example, edge E1 represents, as relational information RI, that the indicator light (node ​​K2) indicates a pedestrian crossing (node ​​K1) or signals it.

[0055] For example, edge E2 represents, as relational information RI, that a road user (node ​​K4) is crossing a pedestrian crossing (node ​​K1).

[0056] For example, edge E3 represents, as relational information RI, that a pedestrian crossing (node ​​K1) passes over or overlaps with a lane (node ​​K3).

[0057] For example, edge E4 represents, as relational information RI, that a road user (node ​​K4) is on a lane (node ​​K3).

[0058] For example, edge E5 represents, as relational information RI, that the lane (node ​​K3) is located below the road user (node ​​K4).

[0059] For example, edge E6 represents, as relational information RI, that a lane (node ​​K3) is intersecting with another node K5~Km-1 (which is not shown in detail), or that nodes K5~Km-1 are intersecting with that lane.

[0060] For example, edge E7 represents, as relational information RI, that a lane (node ​​K3) is connected to another node K5~Km-1 (not shown in detail), or that nodes K5~Km-1 are connected to that lane.

[0061] For example, edge E8 represents, as relational information RI, the interaction between the road user (node ​​K4) and other nodes K5 to Km-1, which are not shown in detail.

[0062] For example, edge E9 represents, as relational information RI, that the regulations of the indicator light (node ​​K2) are applied to the lane (node ​​K3), that is, to control objects O1 to O11 that are on the lane.

[0063] For example, edge E10 represents, as relational information RI, that lane (node ​​K3) precedes another node K5~Km-1 (not shown in detail), or that node K5~Km-1 precedes that lane.

[0064] For example, edge En represents, as relational information RI, that the regulations of the stop sign (node ​​Km) apply to the lane (node ​​K3), meaning that objects O1-O11 in that lane will be stopped.

[0065] The illustrated graph structure GS represents only possible embodiments of such a graph structure and can be flexibly expanded or limited depending on the requirements and surrounding circumstances.

[0066] Figure 5 shows the configuration of the graph structure GS, and the processing of existing information within the graph structure GS by the graph-based artificial neural network N.

[0067] In methods for detecting false images in sensor measurements around a vehicle, the classification of false images is performed on a learning basis.

[0068] At this time, static information SI, dynamic information DI, semantic information SEI, and relational information RI are transferred to the graph structure GS as input information for traffic conditions.

[0069] Next, a learning-based method, specifically a graph-based artificial neural network N, is used, in which case the output information AI provides classification of individual objects O1-O11 as false images and information about the actual existing objects O1-O11.

Claims

1. A method for detecting false images in sensor measurements around a vehicle, wherein map context information is obtained from a digital road map, and objects (O1 to O11) having related attributes are detected around the vehicle, Social context information between the objects (O1 to O11) is generated from the relationships between the objects (O1 to O11) present around the vehicle. All usable data relating to the traffic conditions of the vehicles and objects (O1 to O11) is stored in a graph structure (GS) which includes nodes (K1 to Km) assigned to the objects (O1 to O11) and edges (E1 to En) representing relational information (RI) that represents the relationships between the nodes, and the relational information (RI) is represented in the graph structure (GS) by the edges (E1 to En). Considering the contextual information of the map and the social contextual information, anomalies and patterns are detected from the features of the graph structure (GS). Based on the detected anomalies and patterns, the false images are classified. In observational measurements using multiple observations, The object hypothesis determined from the sensor measurements of individual sensors is considered. Based on at least one object hypothesis from at least one sensor measurement of the sensor, it is checked whether the detection of a false image object in a sensor measurement performed using at least one other sensor is valid or not. A method characterized by the following:

2. The characteristics of the aforementioned graph structure (GS) are as follows: The geometric relationships between the aforementioned objects (O1 to O11), and / or The geometric relationships between the objects (O1 to O11) and their positions around them, and / or The kinematic relationship between the aforementioned objects (O1 to O11), and / or The semantic relationships between the aforementioned objects (O1 to O11), and / or Static relationship between the aforementioned objects (O1 to O11) is used The method according to claim 1, characterized by the features described above.

3. The characteristics of the aforementioned graph structure (GS) are as follows: Temporal information, and / or The distance from the vehicle to at least one object (O1 to O11) at the time of the initial detection of the object (O1 to O11) and / or the first sensor measurement of a series of measurements, and / or The velocity of at least one object (O1 to O11) in comparison with further surrounding objects (O1 to O11), and / or The position of at least one object (O1 to O11) in comparison with the road network determined from the context information of the map, and / or The direction of at least one object (O1 to O11) in comparison with the direction of the lane beneath the object (O1 to O11) is used The method according to claim 1, characterized by the features described above.

4. The aforementioned classification is performed on a learning basis using a graph-based artificial neural network (N). The method according to any one of claims 1 to 3, characterized in that

Images