Procedure and system for identifying critical scenarios

Abstract

Procedure for the identification of critical scenarios (SZ1, SZ2, ..., SZn) for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle, comprising the following procedural steps: - recording (S10) of sensor data (250) by at least one sensor such as a camera, a lidar system, a radar system and / or an ultrasound system of a sensor device (220) of at least one vehicle (200) when driving a plurality of routes, wherein each route comprises a plurality of route sections (S1, S2, ..., Sn) and each route section (Sj) comprises one or more scenarios (SZj) scenarios; - generating (S20) a data set (450) for each route section (Sj) by data fusion of the sensor data (250) of various sensors and additional data (350) such as map data; - extracting (S30) trajectories (T1, T2, ..., Tn) and scenario attributes (A1, A2, ..., An) from each data set (450) of a route section (Sj); - arranging (S40) the trajectories (T1, T2, ..., Tn) of a route section (Sj) in a digital map; - determining (S50) intersection points (P1, P2, ..., Pn) of at least two intersecting trajectories (Tj, Tk); - Assigning (S60) features to the respective intersection point (Pj), wherein the features comprise the associated scenario (SZj) and / or the respective scenario attributes (A1, A2, ..., An) of the scenario (SZj) and / or trajectory properties; - transforming (S70) the features of the intersection point (Pj) into a multidimensional intersection point vector (785); - training (S80) an embedding (750) with a neural network (770) by embedding the intersection vectors (785) in a multidimensional vector space (780), wherein the embedding of the intersection vectors (785) is carried out according to the degree of their similarity, so that intersections (P1, P2, ..., Pn) with similar features form clusters; - determining (S90) clusters in the embedding (750) by means of clustering algorithms, wherein cluster formations of intersection vectors (785) indicate critical scenarios; - Identifying (S100) critical scenarios (SZ1, SZ2, ..., SZn) from the specific clusters.

Description

[0001] The invention relates to a method, a system and a computer program product for identifying critical scenarios for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS).

[0002] Advanced Driver Assistance Systems (ADAS) and Automated Driving Systems (ADS) are two different types of systems used in modern vehicles to increase safety and driving comfort.

[0003] Driver assistance functions support the driver in driving the vehicle and relieve them of certain tasks to make driving safer and more comfortable. They are usually partially automated, meaning the driver retains full responsibility for the vehicle and the driving task. Examples of driver assistance functions include parking assistance, adaptive cruise control, lane departure warning, warning functions, and automatic brake support. These functions use sensors and cameras to collect information about the vehicle's surroundings and can trigger certain actions such as braking, steering, or warnings to the driver.

[0004] Driving functions, especially for autonomous vehicles, aim to operate the vehicle autonomously in various traffic situations without human intervention. They are fully or highly automated and can perform certain driving tasks without the need for a human driver. Examples of driving functions include autonomous vehicles capable of driving autonomously and navigating various traffic situations. These functions utilize advanced sensors, cameras, maps, and artificial intelligence to perform driving tasks such as accelerating, braking, steering, and navigating in complex traffic situations without the intervention of a human driver.

[0005] Individual functions and the entire ADAS / ADS system undergo verification and validation to demonstrate their performance, functionality, and reliability. For example, individual system components such as adaptive cruise control, lane departure warning, and collision warning are tested individually and in combination. Functional tests verify whether the ADAS / ADS system operates correctly, provides precise feedback to the driver, and responds appropriately to various scenarios.

[0006] To validate various driver assistance functions or driving functions, such as overtaking maneuvers, test drives are conducted regularly. Various test cases are used to ensure that the driver assistance functions or driving functions function correctly under various conditions. These test cases can be simulated or conducted under real-life conditions.

[0007] When testing driver assistance and driving functions for semi-autonomous or autonomous vehicles, it is important to achieve balanced and statistically representative test coverage. This means that test drives must be planned to cover a broad range of real-world driving scenarios. Inadequate planning can lead to certain scenarios being overemphasized and others neglected. If certain scenarios are overrepresented, driver assistance or driving functions may perform well in those scenarios, but this does not necessarily mean that they will function safely and reliably in all situations, making the test drive results inconclusive. Furthermore, driving many test kilometers in already well-understood scenarios is a waste of resources, as these kilometers do not provide any new insights into the behavior of the driver assistance or driving function.

[0008] For the validation and verification of ADAS / ADS system functions, virtual testing is increasingly being used alongside real-world testing to simulate functionality. These virtual and simulated testing methods offer a number of advantages, including significant cost savings, as conducting tests in a real-world environment can be costly in terms of both the required equipment and personnel costs. Furthermore, safety can be increased, as virtual testing allows for the reproduction of hazardous scenarios in a safe, controlled environment.

[0009] Test cases are developed for simulation to create a simulation environment for testing ADAS / ADS systems. However, not all test cases represent critical traffic situations. However, since the evaluation of an ADAS / ADS system's performance depends particularly on its behavior in critical test cases, it is necessary to generate a large number of test cases.

[0010] The set of scenarios that can occur in the vehicle's driving environment and must be correctly detected and processed by an ADAS / ADS system is represented by an Operational Design Domain (ODD). This includes both everyday driving scenarios and very rare corner cases.

[0011] The term "corner cases" refers to special and often rare situations that place particular demands on the correct functioning of such systems. These situations go beyond normal, frequently occurring driving scenarios and pose particular challenges for the development and validation of autonomous vehicles. Corner cases particularly encompass rarely occurring traffic situations. These include unusual behavior of other road users, non-standard traffic regulations, construction sites or temporary road markings, but also interactions with pedestrians and animals, whose behavior is often unpredictable. This also includes unusual road infrastructure that does not meet the standard, whether due to unclear or missing traffic signs, unmarked roads, or unusual road geometries.In addition, unusual weather conditions such as heavy snowfall, heavy rain, fog or black ice, which impair the sensors and algorithms and make vehicle navigation more difficult, are considered corner cases.

[0012] Identifying, simulating, and managing corner cases is therefore a crucial aspect in the development of safe and reliable ADAS / ADS systems. This is the only way to ensure that semi-autonomous or autonomous vehicles with driver assistance and / or driving functions operate safely and reliably in a wide range of real-world environments.

[0013] DE10 2020 123 976 A1 discloses a method for determining safety-critical traffic scenarios for driver assistance systems and highly automated driving functions for at least one moving object, such as in particular a motor vehicle, wherein a safety-critical traffic scenario has at least one scenario type, a location determinable by means of geographical coordinates and a safety value.

[0014] The invention is based on the object of demonstrating possibilities for identifying critical scenarios for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS), so that in particular the verification and validation process takes less time and can be carried out with improved accuracy and efficiency.

[0015] This object is achieved according to the invention with respect to a method by the features of patent claim 1, with respect to a system by the features of patent claim 11, and with respect to a computer program product by the features of patent claim 15. The further claims relate to preferred embodiments of the invention.

[0016] According to a first aspect, the invention provides a method for identifying critical scenarios for the verification, validation, and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle. The method comprises the following method steps: - Recording sensor data by at least one sensor such as a camera, a lidar system, a radar system and / or an ultrasound system of a sensor device of at least one vehicle when driving a plurality of routes, each route comprising a plurality of route sections and each route section comprising one or more scenarios; - Creating a data set for each route section by fusing the sensor data from different sensors and additional data such as map data; - Extracting trajectories and scenario attributes from each dataset of a route section; - Arranging the trajectories of a route section in a digital map; - Determining intersection points of at least two intersecting trajectories; - Assigning features to the respective intersection point, wherein the features include the associated scenario and / or the respective scenario attributes of the scenario and / or trajectory properties; - Transforming the features of the intersection into a multidimensional intersection vector; - Training an embedding with a neural network by embedding the intersection vectors in a multidimensional vector space, whereby the embedding of the intersection vectors is carried out according to the degree of their similarity, so that intersections with similar features form clusters; - Determining clusters in the embedding using clustering algorithms, whereby clustering of intersection vectors indicates critical scenarios; - Identifying critical scenarios from the specific clusters.

[0017] By generating an embedding of intersection vectors containing the scenario attributes of the respective assigned scenario, clusters of intersections of trajectories of vehicles and other road users can be formed. Since the intersections each occur in a specific scenario, cluster analysis can identify scenarios that can be considered critical due to the accumulation of trajectory intersections, as the intersections represent an indicator of potential collisions. Furthermore, intersection vectors that cannot be assigned to a cluster are also of interest, as they can indicate rare but potentially relevant events.

[0018] The identified critical scenarios can now be used for targeted test drive planning to specifically cover these critical scenarios. The behavior of a driver assistance or driving function on road sections with critical scenarios, both in real-world test drives and in simulations, can then be used to validate and verify ADAS / ADS systems. By incorporating data from suitable times / weather conditions, the critical scenarios can be further specified.

[0019] In a further training, it is planned that output data with critical scenarios will be generated, particularly in the form of diagrams, graphics, text messages, video sequences and / or PowerPoint presentations.

[0020] In an advantageous embodiment, it is provided that a scenario attribute represents a physical quantity, a chemical quantity, a torque, a rotational speed, a voltage, an amperage, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a motorway, a traffic sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic density, a topographical structure such as a gradient, a time, a temperature, a precipitation value, a weather condition and / or a season.

[0021] In a further embodiment, it is provided that the scenario attributes are selected within the framework of a 5-layer model, wherein a first layer relates to the road geometry and topography, a second layer contains information about the roadside development, a third layer contains information about construction sites and temporary changes on the road, a fourth layer contains the trajectories of dynamically moving objects and a fifth layer relates to environmental influences such as weather conditions, time of day and lighting conditions as well as the road condition.

[0022] Advantageously, the trajectory properties refer to the angle of intersection with another trajectory and the respective speed at which a trajectory is traveled.

[0023] In particular, the network architecture of the neural network is designed as a simple feed-forward network (FFN) or as a more complex model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0024] In a further embodiment, cosine similarity is used to embed the intersection vectors into the vector space with regard to their similarity.

[0025] In particular, deep metric learning and / or constraint pretraining are used to train the neural network to create the embedding.

[0026] Advantageously, the weights of one or more scenario attributes are adjusted during the training process.

[0027] In a further embodiment, it is provided that hierarchical clustering algorithms such as in particular mean-shift clustering and / or partitioning clustering algorithms such as in particular K-Means are used as clustering algorithms.

[0028] According to a second aspect, the invention provides a system for identifying critical scenarios for the verification, validation, and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle. The system comprises a database, an input module, an extraction module, an evaluation module, and an output module. The system is configured to carry out the method according to the first aspect.

[0029] In a further development, it is provided that a scenario attribute represents a physical quantity, a chemical quantity, a torque, a rotational speed, a voltage, an amperage, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a motorway, a traffic sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic density, a topographical structure such as a gradient, a time, a temperature, a precipitation value, a weather condition and / or a season.

[0030] In a further embodiment, it is provided that the evaluation module comprises a neural network, wherein the network architecture of the neural network is designed as a simple feed-forward network (FFN) or as a more complex model, such as in particular a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0031] According to a third aspect, the invention provides a computer program product comprising executable program code configured to carry out the method according to the first aspect when executed.

[0032] The invention is explained in more detail below with reference to embodiments shown in the drawing.

[0033] It shows: Fig. 1 is a block diagram illustrating an embodiment of a system according to the invention; Fig. 2 a schematic representation of an embedding according to the invention; Fig. 3 a flow chart to explain the individual method steps of a method according to the invention; Fig. 4 a schematic representation of a computer program product.

[0034] Additional features, aspects and advantages of the invention or embodiments thereof will become apparent from the detailed description taken in conjunction with the claims.

[0035] Fig. 1 shows a system 100 according to the invention for identifying critical scenarios for the verification, validation, and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS). The system 100 comprises at least one vehicle 200, each with at least one sensor device 220, a database 300, an input module 400, an extraction module 500, an evaluation module 700, and an output module 800. The vehicle 200 can be configured as a motor vehicle, an automated vehicle, an agricultural vehicle such as a combine harvester, a production or service robot, a watercraft, or a flying object such as a drone.

[0036] The database 300, the input module 400, the extraction module 500, the evaluation module 700, and the output module 800 can be implemented as standalone computing units or as a cloud-based solution. In particular, they can each be equipped with a processor and / or a memory unit.

[0037] In the context of the invention, a “processor” can be understood to mean, for example, a machine or an electronic circuit. A processor can in particular be a main processor (Central Processing Unit, CPU), a microprocessor or a microcontroller, e.g. an application-specific integrated circuit or a digital signal processor, optionally in combination with a memory unit for storing program instructions, etc. A processor can also be a virtualized processor, a virtual machine or a soft CPU. It can also be, for example, a programmable processor that is equipped with configuration steps for carrying out the aforementioned method according to the invention or is configured with configuration steps such that the programmable processor implements the inventive features of the method, the modules or other aspects and / or sub-aspects of the invention.In particular, the processor can contain highly parallel computing units and powerful graphics modules.

[0038] In the context of the invention, a "storage unit" or "storage module" and the like can be understood to mean, for example, a volatile memory in the form of random access memory (RAM), a permanent memory such as a hard drive or a data storage device, or, for example, a removable storage module. The storage module can also be a cloud storage solution.

[0039] In the context of the invention, a "module" can be understood, for example, as a processor and / or a memory unit for storing program instructions. For example, the processor is specifically configured to execute the program instructions, so that the processor and / or the control unit performs functions to execute or implement the method according to the invention or a step of the method according to the invention.

[0040] The term “database” refers to both a storage algorithm and the hardware in the form of a storage unit.

[0041] In particular, the database 300 and the extraction module 500 and the evaluation module 700 can be integrated into a cloud computing infrastructure. A wireless communication connection is provided for communication between the modules. The wireless communication connection can be implemented as a WLAN (Wireless Local Area Network), NFC (Near Field Communication), or a mobile radio connection (e.g., 4G LTE, 5G, 6G).

[0042] A cloud computing infrastructure offers the ability to expand or reduce resources as needed, allowing computing power, storage space, or network resources to be easily adapted to changing requirements. This scalability enables cost optimization and efficient resource allocation without large investments in hardware. Furthermore, users can access applications and data from anywhere with internet access. This accessibility enables collaboration and seamless integration across different devices and locations. Cloud computing also offers flexibility in software deployment, allowing companies to deploy and update applications quickly and without disruption.

[0043] In particular, a cloud computing infrastructure is very computationally efficient, as cloud providers have large data centers with a vast pool of computing resources. Since specifying test cases and testing software functions with the specified test cases is very computationally intensive, high computational efficiency with minimization of resource and energy consumption as well as computation time while maximizing output is an important aspect of the present invention.

[0044] A vehicle 200 or a group of vehicles 200 travels a plurality of routes, each of which is divided into a plurality of route sections S1, S2, ..., Sn. A route section Sj of a route can be defined differently within the scope of the invention depending on requirements and context. For example, a route section can be defined by geographical coordinates that determine the start and end points of the route section. This can be done, for example, using longitude and latitude or other coordinate systems. However, a route section can also be defined by a specific number of road kilometers. Furthermore, a route section can be defined by known reference points, e.g., a section between two exits on a motorway or between two intersections in a city.

[0045] The sensor device 220 of the vehicle 200 typically comprises a plurality of sensors with different sensor technologies such as radar sensors, lidar sensors, cameras, ultrasonic sensors and GPS (Global Positioning System), which collect sensor data 250 and information about the vehicle 200 and the vehicle environment while traveling the route sections S1, S2, ..., Sn of a route.

[0046] Sensors typically include one or more cameras. Different types of cameras can be used to enhance comprehensive detection and analysis of the surroundings. Mono cameras, for example, have only a single lens and are used in an ADAS / ADS system for basic functions such as lane departure warning, traffic sign recognition, and driver monitoring. Mono cameras are inexpensive and lightweight, but are not suitable for depth perception. Stereo or depth cameras can capture depth information in the image. This enables more accurate distance estimation to objects and supports functions such as collision warning and automatic emergency braking. Wide-angle cameras offer a larger field of view and are often used to detect vehicles to the side and for parking. Fisheye cameras have an extremely wide field of view and are often used to monitor the area close to the vehicle, especially for parking assistance.Infrared cameras capture images in the infrared spectrum, improving the ability to capture information in low-light or nighttime conditions.

[0047] Additionally, ultrasonic sensors can be used to measure distances to obstacles. Radar sensors are used for adaptive cruise control and collision avoidance systems because they can measure the distance and speed of surrounding vehicles. Lidar (Light Detection and Ranging) sensors are increasingly used in advanced driver assistance systems and autonomous vehicles. They use laser pulses to map the environment in 3D and detect obstacles. Speed ​​sensors measure the vehicle's speed and enable the control of cruise control systems such as cruise control and ABS (anti-lock braking system). GPS sensors determine the vehicle's precise position.Other sensors include speed sensors, steering angle sensors, acceleration sensors that measure the vehicle's acceleration in different directions, gyro sensors that measure the vehicle's rotational movement, humidity sensors, and temperature sensors.

[0048] Within the scope of the present invention, the sensor data 250 (vehicle swarm data) recorded by the sensor device 220 of the vehicle 200 or a plurality of vehicles is transmitted to the input module 400 or stored in the database 300. Vehicle swarm data is data collected and exchanged by a group of networked vehicles. This data comprises a variety of information collected by sensors and communication systems in the vehicles and sent via wireless connections to central servers or other vehicles.

[0049] The data 250 therefore comprises data from different sensor types, which are integrated and fused, particularly within the framework of sensor fusion, to obtain a complete picture of the vehicle's surroundings on a section of road Sj. Artificial intelligence algorithms can be used to leverage the strengths of the individual sensors and compensate for their weaknesses. In particular, the integration of camera data with data from other sensor sources, such as lidar, radar, and ultrasonic sensors, enables redundant and robust environmental detection, which is crucial for autonomous vehicles. The data acquired by the various sensors is combined using data fusion or sensor fusion software. This enables a holistic perception of the environment and increases the detection accuracy of objects and obstacles.The software creates a model of the vehicle's environment, including the positions and trajectories of road users such as other vehicles, pedestrians, cyclists, motorcyclists, and even animals, the road surface, and other relevant information. The sensor and image processing software components analyze the sensor data, detect and classify objects, determine distances and speeds, and extract relevant information.

[0050] The result is an overall perception of the respective route section Sj. The input module 400 and / or the database 300 contain a correspondingly designed software application for carrying out such a sensor data fusion.

[0051] In addition, the input module 400 can retrieve additional data 350, in particular map data, from the database 300 and merge it with the sensor data 250. Map data or cartographic data includes a variety of geographical and topographical information in digital form that is used to represent geographical areas, roads, buildings, and other geographical objects. Map data is used in various applications and services, including navigation software and online mapping applications. Map data contains information about geographical features such as rivers, lakes, mountains, forests, roads, and buildings linked to coordinates (longitude and latitude). In particular, map data contains information about road networks. Furthermore, map data can contain detailed information about buildings such as location, elevation, type of use, and number of floors.Additional information may relate to the type of land use, such as residential, commercial, agricultural, or natural land. Map data can be obtained from various sources, including companies such as Google Maps, Apple Maps, and OpenStreetMap.

[0052] The additional data 350 may also contain traffic information, e.g., typical traffic situations at a specific time and location. Furthermore, further information may be assigned to a route section Sj, for example, statistical weather data containing information such as temperature, precipitation, wind speed, and humidity.

[0053] Overall, the input module 400 generates a data set 450 for a route section Sj from the sensor data 250 and the additional data 350, such as map data, which represents the most complete image possible of the respective route section Sj or of a plurality of route sections S1, S2, ..., Sn. Since a plurality of vehicles 200 travel the respective route section Sj, a plurality of data sets 450 are thus generated for each route section Sj. These data sets 450 are transferred to the extraction module 500.

[0054] A route section Sj can comprise various scenarios that occur sequentially or simultaneously on the route section Sj. A scenario is a specific traffic situation used to test and evaluate the performance and behavior of vehicles with driver assistance functions and / or driving functions in various real or potential traffic environments. Scenarios cover a wide range of traffic situations, including normal driving situations such as driving on motorways, country roads, and inner-city streets, as well as special traffic situations such as crossing intersections, parking and reversing, overtaking other vehicles, and driving in construction zones or in difficult weather conditions. Furthermore, the scenarios include critical situations such as sudden braking, evasive maneuvers, reacting to obstacles on the roadway, or unexpected behavior of other road users.

[0055] A scenario can be characterized by various scenario attributes A1, A2, ..., An. A scenario attribute Aj is, for example, a physical quantity, a chemical quantity, a torque, a rotational speed, a voltage, an amperage, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a motorway, a traffic sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic density, a topographical structure such as a gradient, a time, a temperature, a precipitation value, a weather condition and / or a season. Scenario attributes A1, A2, ..., An thus characterize properties and characteristics of a scenario within the scope of the present invention.

[0056] The selection of the scenario attributes A1, A2, ..., An relevant for the description of a route section Sj can be done within a 5-layer model with the following layers: - The first layer relates to road geometry and topography. This includes the description of the various road types such as highways, rural roads, and inner-city roads and their characteristics. It also includes information on road geometry such as road type, road shape, and layout, including curves, intersections, number of lanes, junctions, roundabouts, and topographic elevation information, as well as the representation of the connections and relationships between different roads and traffic routes. - The second layer contains information about roadside development, i.e. information about traffic signs, traffic lights and road markings, as well as information about buildings, shops, parking lots and other roadside features that may affect traffic. - The third layer includes information about construction sites and temporary changes on the road, such as detours and alternative routes. - The fourth layer includes the trajectories of dynamically moving objects, i.e. the description of the behavior and movement of the ego vehicle as well as other vehicles on the road, including cars, trucks, motorcycles and bicycles, as well as pedestrians, athletes, children, animals and other objects that can move on the road, such as trains and trams. - The fifth layer refers to environmental factors such as weather conditions (factors such as rain, snow, fog and their impact on road conditions and visibility), time of day and light conditions (taking into account changes in light conditions during the day, twilight and night), road surface and condition (information on the condition of the road surface, including bumps, potholes and road repairs).

[0057] This five-layer model enables a detailed analysis and description of scenarios and considers the most important factors and influencing variables that are important in the development and implementation of driver assistance systems and driving functions. It serves as a basis for the development and testing of semi-autonomous and autonomous vehicles, as well as for the simulation of traffic situations in different environments. The layers and the information they contain can be further refined according to specific requirements and objectives.

[0058] Within the scope of the present invention, the fourth layer containing the trajectories of the dynamically moving objects is examined in more detail. For this purpose, the extraction module 500 comprises a software application 520 that extracts a plurality of trajectories T1, T2, ..., Tn of the vehicle 200 and the plurality of road users, as well as associated scenarios SZ1, SZ2, ..., SZn, from the data sets 450 for at least one route section Sj.

[0059] The software application 520 of the extraction module 500 for extracting the trajectories T1, T2, ..., Tn and the scenarios SZ1, SZ2, ..., SZn with the respective scenario attributes A1, A2, ..., An from the data sets 450 includes artificial intelligence algorithms such as classification and extraction algorithms. In particular, deep neural networks such as convolutional neural networks (CNNs) for images and recurrent neural networks (RNNs) for sequences are used. These neural networks are trained with suitable test data sets.

[0060] Neural networks are often cited as prototypes for AI algorithms. Neural networks are also interesting because they can be combined to increase learning capacity. Such coupled neural networks are also known as deep learning. A neural network consists of neurons arranged in multiple layers and connected to each other in various ways. A neuron is capable of receiving information from outside or from another neuron at its input, evaluating it in a specific way, and passing it on in a modified form at the neuron's output to another neuron or outputting it as the final result. Hidden neurons are located between the input neurons and the output neurons. Depending on the type of network, there can be several layers of hidden neurons. They ensure the forwarding and processing of information. Output neurons ultimately provide a result and pass it on to the outside world.The arrangement and connection of neurons creates different types of neural networks, such as feedforward neural networks (FNN), recurrent neural networks (RNN) or convolutional neural networks (CNN).

[0061] CNNs were specifically designed for image processing and use layers of convolutional operations, pooling, and activation functions to extract features from images. They are widely used for tasks such as image classification, object detection, and segmentation.

[0062] The trajectories T1, T2, ..., Tn extracted from the data sets 450 are arranged on the basis of the geographical position data in a digital map with a plurality of route sections S1, S2, ..., Sn. This creates a network of trajectories T1, T2, ..., Tn that overlap and intersect at intersection points P1, P2, ..., Pn. The intersection points P1, P2, ..., Pn are not evenly distributed across a specific geographical area, but rather clusters of intersection points arise at certain locations on the map. These intersection points P1, P2, ..., Pn indicate possible collisions between a vehicle 200 and another road user and are thus indicators of a critical scenario.

[0063] An intersection point Pj between at least two trajectories Tj, Tk can be assigned characteristics such as the intersection angle and the respective speed at which a trajectory Tj is traveled. Furthermore, each intersection point Pj is assigned the associated scenario SZj, which is characterized by various scenario attributes A1, A2, ..., An. The intersection points P1, P2, ..., Pn with the associated scenario attributes A1, A2, ..., An of the respective scenario SZj, as well as other characteristics, are then transferred to an evaluation module 700.

[0064] The evaluation module 700 comprises a software application 720 configured to train an embedding 750 of the intersection points P1, P2, ..., Pn with the extracted scenario attributes A1, A2, ..., An using a neural network 770. For this purpose, the extracted scenario attributes A1, A2, ..., An of an intersection point Pj are transformed by the software application 720 into a multidimensional intersection point vector 785 located in a multidimensional vector space 780. The representation of the scenario attributes A1, A2, ..., An of an intersection point Pj as an intersection point vector 785 enables data processing by the neural network 770.

[0065] The neural network 770 is trained with the plurality of intersection vectors 785 for the plurality of intersection points P1, P2, ..., Pn between two or more trajectories Tj, Tk to generate an embedding 750 of the intersection vectors 785. Training an embedding 750 with the neural network 770 refers to the process of embedding the intersection vectors 785 into the multidimensional vector space 780. The embedding 750 of the intersection points P1, P2, ..., Pn is thus the representation of the scenario attributes A1, A2, ..., An for each of the intersection points P1, P2, ..., Pn in the multidimensional vector space 780. The embedding of the intersection vectors 785 is performed according to their degree of similarity, so that similar intersection points Pi, Pj are close to each other in this vector space 780. Similar intersection points Pi, Pj contain similar scenario attributes A1, A2, ..., An and therefore also have a similar vector representation. Fig. 2 shows a schematic representation of an embedding 750.

[0066] For embedding the intersection vectors 785 in terms of their similarity, cosine similarity or cosine distance is used. Cosine similarity is a metric for quantifying the similarity between two vectors in a multidimensional space and measures the cosine of the angle between two vectors. The closer the cosine similarity value is to 1, the more similar the vectors are; the closer it is to -1, the more different they are. A value of 0 means that the vectors are orthogonal to each other and therefore have no similarity.

[0067] Depending on the type of data and the specific task, the network architecture of the neural network 770 can be a simple feed-forward network (FFN) or a more complex model such as a convolutional neural network (CNN) or a recurrent neural network (RNN). Additionally, an embedding layer is added to the network architecture, which generates the embeddings for the intersection vectors 785 to be learned. Furthermore, the size of the embedding vectors 785 and other hyperparameters are defined. The neural network 780 is now trained to perform clustering of the intersection vectors 785.

[0068] Various methods such as deep metric learning or constraint pretraining can be used to train the neural network 770 to create the embedding 750. During the training process, the weightings of one or more scenario attributes A1, A2, ..., An can be adjusted in the embedding layer. In this way, an embedding 750 can be created that delivers optimized results for specific tasks related to a particular driver assistance or driving function.

[0069] Deep metric learning is a method that focuses on learning distance or similarity measures between data points. This approach aims to develop models that represent data in a space where similar data points are close together and dissimilar data points are far apart. By creating a model of the similarity relationships between data points, similar intersections P1, P2, ..., Pn can be identified in the data sets 450. An important part of deep metric learning is the transformation of the data sets 450 into the embedding space, where the similarity relationships between the individual data points of the different intersections P1, P2, ..., Pn become more clearly visible.

[0070] Contrastive pretraining refers to a model training method in which a model is trained to learn differences (contrasts) between data points. The model learns to bring similar data points closer together (similar code) and separate dissimilar data points (dissimilar code). During training, the model learns to represent the data in an embedding space. In this space, similar data points are close together, while dissimilar data points are farther apart. After an initial training phase, the model can also be fine-tuned. For example, the model can be further refined for a specific driver assistance or driving function by adding additional layers or neurons. The particular advantage of contrastive pretraining is that it can also be trained with unlabeled datasets.

[0071] After the embedding 750 has been created, appropriate metrics can be used to check whether the quality and accuracy are satisfactory.

[0072] The determination of clusters in an embedding 750 is performed by applying clustering algorithms to the embedding vectors 785. Since similar vectors 785 are close to each other in the vector space 780, by clustering these vectors 785, similar intersection points Pi, Pj can be identified and organized into groups or clusters.

[0073] Clustering algorithms are machine learning and data analysis techniques used to organize similar data points into groups or clusters. Clusters are groups of data points that are more similar to each other than data points in other groups. There are different types of clustering algorithms, which are further divided into hierarchical and partitioning clustering algorithms.

[0074] Hierarchical clustering algorithms organize data into a hierarchical tree structure of clusters. They often begin with each data point as a separate cluster and then gradually group clusters until the desired grouping is achieved. An example of hierarchical clustering is mean-shift clustering, a technique that uses density estimates to find cluster centers and assigns data points to the nearest centers.

[0075] Partitioning clustering algorithms divide the data into a specified number of clusters and attempt to find an optimal distribution of the data within the clusters. K-Means, for example, is a partitioning clustering algorithm in which K clusters are specified and the data points are assigned to the nearest cluster centers. The selection of the appropriate clustering algorithm also depends on the specific requirements of the respective driver assistance or driving function.

[0076] The selected clustering algorithm is now applied to the embedding data and assigns the vectors 785 to the corresponding clusters based on their similarities in the embedding space 780.

[0077] The clustering results can be visualized and evaluated using an additional software application. The visualization shows how the intersection points P1, P2, ..., Pn are arranged within the clusters. Various evaluation metrics can be used to assess the quality of the clusters.

[0078] The clusters found can now be interpreted. The clusters resulting from the embedding 750 provide information about the frequency of certain intersections P1, P2, ..., Pn with certain scenario attributes A1, A2, ..., An. In this way, representative route sections Si, Sj can be found for test drives for testing a driver assistance function or a driving function that can cover the widest possible spectrum of critical scenarios. This enables optimized test route planning based on critical route sections S1, S2, ..., Sn for both real test drives and simulations, thus reducing the size of the test set.

[0079] The results of the clustering in the form of visualizations and evaluations can be displayed by an output module 800 as output data 850. The output module 800 can be located, in particular, on a computer or a mobile device. The output data 850 can be output, in particular, in the form of a report and can contain diagrams, graphics, text messages, video sequences, PowerPoint presentations, etc.

[0080] In Fig. Figure 3 shows the process steps of a procedure for identifying critical scenarios SZ1, SZ2, ..., SZn for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle.

[0081] In a step S20, sensor data are recorded by at least one sensor such as a camera, a lidar system, a radar system and / or an ultrasound system of a sensor device of at least one vehicle while driving along a plurality of routes, wherein each route comprises a plurality of route sections and each route section comprises one or more scenarios.

[0082] In a step S20, a data set is generated for each route section by data fusion of the sensor data from different sensors and additional data such as map data.

[0083] In a step S30, trajectories T1, T2, ..., Tn and scenario attributes A1, A2, ..., An are extracted from each data set 450 of a route section Sj.

[0084] In a step S40, the trajectories T1, T2, ..., Tn of a route section Sj are arranged in a digital map.

[0085] In a step S50, intersection points P1, P2, ..., Pn of at least two intersecting trajectories Tj, Tk are determined.

[0086] In a step S60, features are assigned to the respective intersection point Pj, wherein the features comprise the associated scenario SZj and / or the respective scenario attributes A1, A2, ..., An of the scenario SZj and / or trajectory properties.

[0087] In a step S70, the features of the intersection point Pj are transformed into a multidimensional intersection point vector 785.

[0088] In a step S80, an embedding 750 is trained with a neural network 770 by embedding the intersection vectors 785 into a multidimensional vector space 780, wherein the embedding of the intersection vectors 785 is carried out according to the degree of their similarity, so that intersections P1, P2, ..., Pn with similar features form clusters.

[0089] In a step S90, clusters in the embedding 750 are determined using clustering algorithms, whereby cluster formations of intersection vectors 785 indicate critical scenarios.

[0090] In a step S100, critical scenarios SZ1, SZ2, ..., SZn are identified from the specific clusters.

[0091] Fig. 4 schematically illustrates a computer program product 900 comprising executable program code 950 configured to perform the method according to the first aspect of the present invention when executed.

[0092] By generating an embedding of intersection vectors containing the scenario attributes A1, A2, ..., An of the respectively assigned scenario SZj, clusters of intersection points P1, P2, ..., Pn of trajectories T1, T2, ..., Tn of vehicles and other road users can be formed. Since the intersection points P1, P2, ..., Pn each occur in a specific scenario SZj, the analysis of the clusters can be used to identify scenarios SZ1, SZ2, ..., SZn that can be considered critical due to the accumulation of trajectory intersection points, as the intersection points represent an indicator of possible collisions. In addition, intersection vectors that cannot be assigned to a cluster are also of interest, as they can indicate rare but potentially relevant events.

[0093] The identified critical scenarios can now be used for targeted test drive planning to specifically cover these critical scenarios. The behavior of a driver assistance or driving function on road sections with critical scenarios, both in real-world test drives and in simulations, can then be used to validate and verify ADAS / ADS systems. By incorporating data from suitable times / weather conditions, the critical scenarios can be further specified. Reference symbol 100 systems 200 vehicles 220 Sensor device 250 sensor data 300 database 350 additional data 400 input module 450 data sets 500 extraction module 520 software application 700 evaluation module 720 software application 750 Embedding 770 neural network 780 Vector space 785 route section vector 800 output model 850 Report 900 computer program product 950 program code

Claims

[1] Procedure for the identification of critical scenarios (SZ1, SZ2, ..., SZn) for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle, with the following procedural steps: - recording (S10) of sensor data (250) by at least one sensor such as a camera, a lidar system, a radar system and / or an ultrasound system of a sensor device (220) of at least one vehicle (200) when driving a plurality of routes, wherein each route comprises a plurality of route sections (S1, S2, ..., Sn) and each route section (Sj) comprises one or more scenarios (SZj) scenarios; - generating (S20) a data set (450) for each route section (Sj) by data fusion of the sensor data (250) of various sensors and additional data (350) such as map data; - extracting (S30) trajectories (T1, T2, ..., Tn) and scenario attributes (A1, A2, ..., An) from each data set (450) of a route section (Sj); - arranging (S40) the trajectories (T1, T2, ..., Tn) of a route section (Sj) in a digital map; - determining (S50) intersection points (P1, P2, ..., Pn) of at least two intersecting trajectories (Tj, Tk); - Assigning (S60) features to the respective intersection point (Pj), wherein the features comprise the associated scenario (SZj) and / or the respective scenario attributes (A1, A2, ..., An) of the scenario (SZj) and / or trajectory properties; - transforming (S70) the features of the intersection point (Pj) into a multidimensional intersection point vector (785); - training (S80) an embedding (750) with a neural network (770) by embedding the intersection vectors (785) in a multidimensional vector space (780), wherein the embedding of the intersection vectors (785) is carried out according to the degree of their similarity, so that intersections (P1, P2, ..., Pn) with similar features form clusters; - determining (S90) clusters in the embedding (750) by means of clustering algorithms, wherein cluster formations of intersection vectors (785) indicate critical scenarios; - Identifying (S100) critical scenarios (SZ1, SZ2, ..., SZn) from the specific clusters. [2] Method according to claim 1, wherein output data (850) with critical scenarios (SZ1, SZ2, ..., SZn) are generated, in particular in the form of diagrams, graphics, text messages, video sequences and / or PowerPoint presentations. [3] The method according to claim 1 or 2, wherein a scenario attribute (Aj) represents a physical quantity, a chemical quantity, a torque, a rotational speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a motorway, a traffic sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic density, a topographical structure such as a gradient, a time, a temperature, a precipitation value, a weather condition and / or a season. [4] Method according to one of claims 1 to 3, wherein the scenario attributes (A1, A2, ..., An) are selected within the framework of a 5-layer model, wherein a first layer relates to the road geometry and topography, a second layer contains information about the roadside development, a third layer contains information about construction sites and temporary changes on the road, a fourth layer contains the trajectories of dynamically moving objects and a fifth layer relates to environmental influences such as weather conditions, time of day and lighting conditions as well as the road condition. [5] Method according to one of claims 1 to 4, wherein the trajectory properties relate to the angle of intersection with another trajectory (Tj) and the respective speed at which a trajectory (Tj) is travelled. [6] Method according to one of claims 1 to 5, wherein the network architecture of the neural network (770) is designed as a simple feed-forward network (FFN) or as a more complex model, such as in particular a convolutional neural network (CNN) or a recurrent neural network (RNN). [7] Method according to one of claims 1 to 6, wherein the cosine similarity is used to embed the intersection vectors (785) into the vector space (780) with regard to their similarity. [8] Method according to one of claims 1 to 7, wherein deep metric learning and / or constraint pretraining are used for training the neural network (770) to create the embedding (750). [9] Method according to one of claims 1 to 8, wherein during the training process the weights of one or more scenario attributes (A1, A2, ..., An) are adjusted. [10] Method according to one of claims 1 to 9, wherein hierarchical clustering algorithms such as in particular mean-shift clustering and / or partitioning clustering algorithms such as in particular K-Means are used as clustering algorithms. [11] System (100) for identifying critical scenarios (SZ1, SZ2, ..., SZn) for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle, comprising a database (300), an input module (400), an extraction module (500), an evaluation module (700) and an output module (800), wherein the system (100) is designed to carry out the method according to one of claims 1 to 10. [12] The system (100) of claim 11, wherein a scenario attribute (Aj) represents a physical quantity, a chemical quantity, a torque, a rotational speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a motorway, a traffic sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic density, a topographical structure such as a gradient, a time, a temperature, a precipitation value, a weather condition and / or a season. [13] System (100) according to claim 11 or 12, wherein the evaluation module (700) comprises a neural network (770), wherein the network architecture of the neural network (770) is designed as a simple feed-forward network (FFN) or as a more complex model, such as in particular a convolutional neural network (CNN) or a recurrent neural network (RNN). [14] A computer program product (900) comprising an executable program code (950) configured to carry out the method according to any one of claims 1 to 10 when executed.

Images