Method and system for labeling vehicle data

Abstract

The invention relates to a method for labeling vehicle data, in particular for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle, comprising the following method steps: - recording vehicle data (250) by a data acquisition device (220) of at least one vehicle (200) when driving a plurality of routes; - input of feedback data (250) by a user relating to the behavior of the vehicle (200) in the form of an audio recording, a text message or a symbol; - converting the feedback data (250) into an annotation (550), wherein the annotation (550) reflects the essential meaning of the feedback data (450); - Assigning the annotation (550) to the vehicle data (250) recorded during a defined time interval (Δt) or while driving along a specific route section (Sj), and generating a data set (Dj); - transforming (S50) the data set (Dj) into a multidimensional data vector (785); - training an embedding model (750) with a neural network (770) by embedding the data vectors (785) in a multidimensional vector space (780), - Determining clusters in the embedding model (750) by means of clustering algorithms, whereby critical vehicle data (250) can be identified.

Description

[0001] The invention relates to a method, a system and a computer program product for labeling vehicle data, in particular for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS).

[0002] Advanced Driver Assistance Systems (ADAS) and Automated Driving Systems (ADS) are two different types of systems used in modern vehicles to increase safety and driving comfort.

[0003] Driver assistance functions support the driver in driving the vehicle and relieve them of certain tasks to make driving safer and more comfortable. They are usually partially automated, meaning the driver retains full responsibility for the vehicle and the driving task. Examples of driver assistance functions include parking assistance, adaptive cruise control, lane departure warning, warning functions, and automatic brake support. These functions use sensors and cameras to collect information about the vehicle's surroundings and can trigger certain actions such as braking, steering, or warnings to the driver.

[0004] Driving functions, especially for autonomous vehicles, aim to operate the vehicle autonomously in various traffic situations without human intervention. They are fully or highly automated and can perform certain driving tasks without the need for a human driver. Examples of driving functions include autonomous vehicles capable of driving autonomously and navigating various traffic situations. These functions utilize advanced sensors, cameras, maps, and artificial intelligence to perform driving tasks such as accelerating, braking, steering, and navigating in complex traffic situations without the intervention of a human driver.

[0005] Individual functions and the entire ADAS / ADS system undergo verification and validation to demonstrate their performance, functionality, and reliability. For example, individual system components such as adaptive cruise control, lane departure warning, and collision warning are tested individually and in combination. Functional tests verify whether the ADAS / ADS system operates correctly, provides precise feedback to the driver, and responds appropriately to various scenarios.

[0006] To validate the various driver assistance functions or driving functions, such as an overtaking maneuver, test drives are carried out regularly with test vehicles, during which various types of data are collected in order to evaluate the performance and safety of the individual functions and the overall system.

[0007] But data can be collected not only from test vehicles; a vehicle manufacturer's customer vehicles also collect data, which is then transmitted to the vehicle manufacturer with the vehicle user's consent. This vehicle data, provided by customers or users such as lessees, thus represents another data source.

[0008] The real-life journeys of the customer vehicle fleet thus generate a large volume of data, which, due to the volume of data generated, offers a high degree of realism and high statistical representativeness. This data can therefore provide valuable insights into usage behavior, the malfunction of functions of an ADAS / ADS system in certain scenarios, and customer acceptance. However, this data is highly redundant, unlabeled, and unstructured. Therefore, this customer data has hardly been used to date, even though it is precisely this customer data that contains, in addition to everyday driving scenarios, very rare corner cases that represent great value for the verification and validation of an ADAS / ADS system.

[0009] EP 3 882 812 A1 discloses a method for generating label data representing an image content of images with a scene, wherein image data is received from an image recording device in a processing device and a segmentation unit detects at least one object in the image data and a graphic processing unit generates a respective graphic object marking, and wherein an input receiving unit receives for each object marking a respective user input providing the image content of the image area marked by the object marking.

[0010] The invention is based on the object of demonstrating possibilities for efficiently labelling, structuring and sorting data from real journeys of the customer vehicle fleet so that they can be used in particular for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS).

[0011] This object is achieved according to the invention with respect to a method by the features of patent claim 1, with respect to a system by the features of patent claim 8, and with respect to a computer program product by the features of patent claim 12. The further claims relate to preferred embodiments of the invention.

[0012] By labeling vehicle data with feedback data from a vehicle user according to the invention, data sets are generated that can be analyzed using an embedding model. The embedding model is trained with data vectors generated from the generated data sets. Vehicle data with similar feedback data form clusters and can then be analyzed specifically. Depending on the significance of the respective feedback data, which is summarized in an annotation, critical vehicle data can be identified. The data volume of a customer vehicle fleet can thus be used in a targeted and efficient manner.

[0013] According to a first aspect, the invention provides a method for labeling vehicle data, particularly for the verification, validation, and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle. The method comprises the following method steps: - Capturing vehicle data by at least one sensor such as an acceleration sensor, a steering angle sensor, a camera, a lidar system, a radar system and / or an ultrasound system of a data acquisition device of at least one vehicle while driving a plurality of routes, each route comprising a plurality of route sections and each route section comprising one or more scenarios; - Input of feedback data by a user in the form of an audio recording, a text message or a symbol, where the feedback data relates to the behavior of the vehicle; - Converting the feedback data into an annotation, where the annotation reflects the essential meaning of the feedback data; - Assigning the annotation to the vehicle data recorded during a defined time interval or while driving a specific route section and generating a data set, the data set containing the annotation and the vehicle data; - Transforming the data set into a multidimensional data vector; - Training an embedding model with a neural network by embedding the data vectors in a multidimensional vector space, whereby the embedding of the data vectors is carried out according to the degree of their similarity, so that vehicle data with similar annotations form clusters; - Determining clusters in the embedding model using clustering algorithms, which allows identifying critical vehicle data.

[0014] In a further development, the feedback data is converted into an annotation by a software application, whereby the software application uses speech recognition algorithms for acoustic sequences, word processing algorithms for text messages and image processing algorithms for graphic symbols.

[0015] In an advantageous embodiment, it is provided that the vehicle data are bus data of the vehicle, which are recorded as time series data and stored in a cloud-based database.

[0016] In a further embodiment, it is provided that the network architecture of the neural network is designed as a simple feed-forward network (FFN) or as a more complex model, such as in particular a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0017] Advantageously, cosine similarity is used to embed the data vectors into the vector space with regard to their similarity.

[0018] In particular, deep metric learning and / or constraint pretraining are used to train the neural network to create the embedding model.

[0019] In a further embodiment, it is provided that hierarchical clustering algorithms such as in particular mean-shift clustering and / or partitioning clustering algorithms such as in particular K-Means are used as clustering algorithms.

[0020] According to a second aspect, the invention provides a system for labeling vehicle data, in particular for the verification, validation, and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle. The system comprises a database, an input module, an extraction module, an evaluation module with an embedding model, and an output module, wherein the system is configured to execute the method according to the first aspect.

[0021] In a further development, the feedback data is converted into an annotation by a software application, whereby the software application uses speech recognition algorithms for acoustic sequences, word processing algorithms for text messages and image processing algorithms for graphic symbols.

[0022] In an advantageous embodiment, it is provided that the vehicle data are bus data of the vehicle, which are recorded as time series data and stored in a cloud-based database.

[0023] In a further embodiment, it is provided that the network architecture of the neural network is designed as a simple feed-forward network (FFN) or as a more complex model, such as in particular a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0024] According to a third aspect, the invention provides a computer program product comprising executable program code configured to carry out the method according to the first aspect when executed.

[0025] The invention is explained in more detail below with reference to embodiments shown in the drawing.

[0026] It shows: Fig. 1 is a block diagram illustrating an embodiment of a system according to the invention; Fig. 2 a schematic representation of an embedding model according to the invention; Fig. 3 a flow chart to explain the individual method steps of a method according to the invention; Fig. 4 a schematic representation of a computer program product.

[0027] Additional features, aspects and advantages of the invention or embodiments thereof will become apparent from the detailed description taken in conjunction with the claims.

[0028] In a scenario during a test drive with a test vehicle to validate advanced driver assistance or driver functions of an ADAS / ADS system, e.g. an overtaking maneuver, various types of data are collected to evaluate the performance and safety of the ADAS / ADS system. Typical data collected during such an overtaking maneuver relates to the speed of the vehicle throughout the overtaking maneuver, the acceleration and deceleration during the overtaking maneuver, the steering angle of the vehicle during the overtaking maneuver, the exact position of the vehicle on the road, the vehicle health with data on vehicle parameters such as brake pressure, accelerator pedal position, gear shift, etc. as well as traffic conditions, i.e. information about other road users such as vehicles and people on the road including their position, speed and behavior.Further data includes weather conditions such as rain, snow, or sunshine, and information about road conditions, including bumps, curves, and gradients. Added to this are sensor data such as radar and lidar data, which measure distances and speeds to other vehicles or obstacles; camera data, which captures visual information about the surroundings, traffic signs, lane markings, and other vehicles; ultrasonic or sonar data, which measures the distance to objects; and GPS data, which indicates the vehicle's precise position in relation to the road and the surroundings. Then there is the driver's steering and braking behavior, i.e., how the driver uses the steering wheel and brakes during overtaking. Another aspect concerns the driver's attention based on physiological data or driver activity monitoring.In the context of the verification and validation of an ADAS / ADS system, the timing and duration of activation of driver assistance functions, as well as the ADAS / ADS system's response to the environment, e.g., through steering intervention, acceleration, or braking, are important. Data on special events during overtaking, such as obstacle detection, emergency braking, or other safety-relevant events, can be of great interest.

[0029] This data is then analyzed to ensure that the overtaking maneuver was performed safely and efficiently. The data can thus be used to improve the performance and safety of the ADAS / ADS system.

[0030] This data can also be collected during real journeys by customer vehicles equipped with appropriate sensors. Since a customer vehicle fleet consists of a large number of customer vehicles, a large amount of vehicle data is generated in a variety of scenarios. This large amount of data ensures comprehensive coverage of different situations and scenarios and thus a high level of statistical representativeness, which forms the basis for reliable and meaningful statistical analyses. Since the vehicle data originates from real journeys by customer vehicles, it reflects real road conditions, traffic patterns, and driver behavior. Vehicle data therefore exhibits a high degree of realism. Furthermore, very diverse data sets are generated, since vehicle data from a customer vehicle fleet represents a broad spectrum of driving environments, vehicle types, and driver profiles.This leads to a diversification of data sets, which in turn increases the variety of collected information and its usability for different application areas. Due to their realism and statistical representativeness, data from customer vehicle fleets can be crucial for the development and improvement of driver assistance systems and autonomous driving, as practical data is essential for validating and optimizing the performance and safety of ADAS / ADS systems in real-world traffic situations.

[0031] However, this vehicle data, which is usually unstructured and raw, must be prepared for effective use through the process of labeling and structuring. Labeling the data is especially important if it is to be used for machine learning. Adding labels or tags to the data identifies specific patterns or characteristics that an AI model can learn. Structuring vehicle data allows information to be organized into a specific form or structure that allows for easier handling and analysis. Labeled and structured data is of higher quality and is therefore better suited for analytics and AI models.

[0032] Fig. 1 shows a system 100 according to the invention for labeling vehicle data 250 of a plurality of vehicles, in particular for the verification, validation, and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS). The system 100 comprises a plurality of vehicles 200 of a customer vehicle fleet, each having at least one data acquisition device 220 with a plurality of sensors, as well as a database 300, an input module 400, an extraction module 500, an evaluation module 700, and an output module 800.

[0033] The data acquisition device 220, the database 300, the input module 400, the extraction module 500, the evaluation module 700 and the output module 800 can each be equipped with a processor and / or a memory unit.

[0034] In the context of the invention, a "processor" can be understood, for example, as a machine or an electronic circuit. A processor can in particular be a central processing unit (CPU), a microprocessor, or a microcontroller, e.g., an application-specific integrated circuit or a digital signal processor, optionally in combination with a memory unit for storing program instructions, etc. A processor can also be a virtualized processor, a virtual machine, or a soft CPU. It can, for example, also be a programmable processor equipped with configuration steps for executing the aforementioned method according to the invention or configured with configuration steps such that the programmable processor implements the inventive features of the method, the modules, or other aspects and / or sub-aspects of the invention.In particular, the processor can contain highly parallel computing units and powerful graphics modules.

[0035] In the context of the invention, a "storage unit" or "storage module" and the like can be understood to mean, for example, a volatile memory in the form of random access memory (RAM), a permanent memory such as a hard drive or a data storage device, or, for example, a removable storage module. The storage module can also be a cloud storage solution.

[0036] In the context of the invention, a "module" can be understood, for example, as a processor and / or a memory unit for storing program instructions. For example, the processor is specifically configured to execute the program instructions, so that the processor and / or the control unit performs functions to execute or implement the method according to the invention or a step of the method according to the invention.

[0037] The term “database” refers to both a storage algorithm and the hardware in the form of a storage unit.

[0038] In particular, the database 300, the extraction module 500, and the evaluation module 700 are integrated into a cloud computing infrastructure. A wireless communication connection is provided for communication between the sensor device 220 of the vehicle 200 and the cloud-based database 300. The wireless communication connection can be implemented as a WLAN (Wireless Local Area Network), NFC (Near Field Communication), or a mobile radio connection (e.g., 4G LTE, 5G, 6G).

[0039] A cloud computing infrastructure offers the ability to expand or reduce resources as needed, allowing computing power, storage space, or network resources to be easily adapted to changing requirements. This scalability enables cost optimization and efficient resource allocation without large hardware investments. Furthermore, users can access applications and data from anywhere with internet access. This accessibility enables collaboration and seamless integration across different devices and locations. Cloud computing also offers flexibility in software deployment, allowing companies to deploy and update applications quickly and without disruption.

[0040] In particular, a cloud computing infrastructure is highly computationally efficient, as cloud providers have large data centers with an extensive pool of computing resources. Since the storage and processing of customer fleet data is very memory- and computationally intensive, high computational efficiency with minimization of resource and energy consumption as well as computing time while maximizing output is an important aspect of the present invention.

[0041] The various vehicles 200 of a customer vehicle fleet travel a plurality of routes, each of which is divided into a plurality of route sections S1, S2, ..., Sn. A route section Sj of a route can be defined differently within the scope of the invention depending on requirements and context. For example, a route section can be defined by geographical coordinates that determine the start and end points of the route section. This can be done, for example, by longitude and latitude or other coordinate systems. However, a route section can also be defined by a specific number of road kilometers. Furthermore, a route section can be defined by known reference points, e.g., a section between two exits on a motorway or between two intersections in a city.

[0042] A route section Sj can comprise various scenarios that occur sequentially or simultaneously on the route section Sj. A scenario is a specific traffic situation used to test and evaluate the performance and behavior of vehicles with driver assistance functions and / or driving functions in various real or potential traffic environments. Scenarios cover a wide range of traffic situations, including normal driving situations such as driving on motorways, country roads, and inner-city streets, as well as special traffic situations such as crossing intersections, parking and reversing, overtaking other vehicles, and driving in construction zones or in difficult weather conditions. Furthermore, the scenarios include critical situations such as sudden braking, evasive maneuvers, reacting to obstacles on the roadway, or unexpected behavior of other road users.

[0043] The data acquisition device 220 of a vehicle 200 typically comprises a plurality of sensors with different sensor technologies such as acceleration sensors, steering angle sensors, radar sensors, lidar sensors, cameras, ultrasonic sensors and GPS (Global Positioning System), which acquire vehicle data 250 with information about the vehicle 200 and the vehicle environment while driving the route sections S1, S2, ..., Sn of a route.

[0044] While a user is driving a vehicle 200, vehicle data 250 is thus acquired as a time series by various sensors of the data acquisition system 220 and transmitted serially to the database 300. The vehicle data 250 is, in particular, bus data that is acquired via a vehicle bus and transmitted to the database 300 by means of a communication device. A vehicle bus is a communication system that interconnects various electronic control units in the vehicle. Data exchange via the vehicle bus enables coordinated communication between the various vehicle components. In particular, the vehicle bus is designed as a Controller Area Network (CAN), which is used for communication between electronic control units (ECUs) in the vehicle. In addition, other bus systems, such as FlexRay or Ethernet, can also be used.FlexRay is a high-speed bus primarily used in modern vehicles for demanding applications such as driver assistance systems. Ethernet is gaining increasing importance as a vehicle bus due to increasing networking and the need for higher bandwidths.

[0045] Bus data includes information sent by various vehicle components such as sensors, control units, actuators, and other systems. This data can contain a wide variety of information: Engine data relates to engine speed, temperature, and fuel consumption. Driving dynamics data contains information about speed, acceleration, steering angle, etc. Sensor data includes information from sensors such as radar, lidar, and cameras. Operating status data contains information about the status of brakes, headlights, airbags, etc. Communication data includes messages between different control units. Position data contains information about the vehicle's position.

[0046] The vehicle data 250 is recorded as a time series. A time series can include various types of data, such as periodic measurements at specific time intervals; event-related measurements collected during specific events or activities over time; continuous measurements of data collected continuously and without interruption to accurately capture changes over time; and irregular measurements of data collected at different times without a predetermined schedule.

[0047] According to the invention, the user of the vehicle 200 can use the input module 400 to input feedback in the form of feedback data 450 about the behavior of the vehicle 200 at a specific point in time. For this purpose, the input module 400 comprises a user interface designed as a microphone or a touchscreen. In particular, the input module 400 can be designed as a headset. The feedback data 450 can be a voice message saved as an audio recording. For example, the voice message can include the sentence: "The speed limit of 30 was not correctly recognized." However, it can also be provided that certain text messages or graphic symbols are shown on the display, which the user can click on to provide feedback about the current behavior of the vehicle 200.Examples of text messages include: “Worked very well”, “I am not convinced”, “The traffic sign was not recognized”, “The overtaking maneuver was supported safely and comfortably”.

[0048] The user can enter the feedback data 450 on their own initiative if they notice an unusual feature. However, it can also be provided that the data acquisition device 220 at least partially analyzes the acquired vehicle data 250 and, upon detecting vehicle data 250 that indicate special or unusual events, sends a notification to the user requesting feedback, for example, as a visual display or as an acoustic message. With a visual display, the display can also show suggested text for a text message or symbols that can be selected by the user based on their assessment of the driving situation.

[0049] The feedback data 450 in the form of an audio recording for acoustic sequences, a text message, or a graphic symbol is transmitted to the extraction module 500. The extraction module 500 includes a software application 520 that converts the feedback data 450 into an annotation 550. The annotation 550 reflects the essential meaning of the feedback data 450. The annotation 550 can be a single term. However, it can also consist of a longer sentence or text.

[0050] The software application 520 includes corresponding processing algorithms, such as speech recognition algorithms for acoustic sequences, text processing algorithms for text messages, and image processing algorithms for graphic symbols, to transform the feedback data 450 into an annotation 550. This step involves interpreting and analyzing the feedback data 450 to capture its essential meaning. In the case of an acoustic sequence, the software application 520 uses speech recognition algorithms to extract the spoken text and understand its meaning. The resulting interpretation of the feedback data 450 is referred to as annotation 550 within the scope of the present invention. The annotation 550 represents the essential meaning or content of the original feedback data 450. According to the invention, it is available in a structured form and contains the extracted information.The annotation 550 allows the feedback data 450 captured by speech, text or symbol to be presented in a form suitable for further processing steps.

[0051] The software application 520 assigns the respective annotation 550 to the vehicle data 250 that were recorded during a defined time interval Δt or during a specific route section Sj and generates a data set Dj from the vehicle data 250 and the annotation 550. The time interval Δt typically comprises a few seconds, in particular 2 to 5 seconds.

[0052] In addition to the transmitted vehicle data 250, this data set Dj can contain further additional data 350, in particular map data, retrieved from the database 300. Map data or cartographic data comprises a variety of geographical and topographical information in digital form used to represent geographical areas, roads, buildings, and other geographical objects. Map data is used in various applications and services, including navigation software and online mapping applications. Map data contains information about geographical features such as rivers, lakes, mountains, forests, roads, and buildings linked to coordinates (longitude and latitude). In particular, map data contains information about road networks. Furthermore, map data can contain detailed information about buildings such as location, elevation, type of use, and number of floors.Additional information may relate to the type of land use, such as residential, commercial, agricultural, or natural land. Map data may be obtained from various sources, including companies such as Google Maps, Apple Maps, and OpenStreetMap.

[0053] The additional data 350 can also contain traffic information, e.g., typical traffic situations at a specific time and location. Furthermore, further information can be assigned to a route section Sj, e.g., statistical weather data containing information such as temperature, precipitation, wind speed, and humidity.

[0054] Overall, the software application 520 of the extraction module 500 generates a data set Dj from the vehicle data 250, which were recorded during a time interval Δt or while driving along a specific route section Sj, the annotation 550 and any additional data 350, such as map data.

[0055] Since each data set Dj contains an annotation 550, the data sets D1, D2, ..., Dn thus generated are each provided with a label. The terms "label" and "tag" are used synonymously within the scope of the invention and mean that additional information or categories have been added to the vehicle data 250 to identify certain patterns or behaviors.

[0056] The data sets D1, D2, ..., Dn with the vehicle data 250 and the annotation 550 are now transferred to the evaluation module 700 for data analysis. For pattern recognition in the labeled data sets D1, D2, ..., Dn, an embedding model 750 is trained using the data sets D1, D2, ..., D3 with different annotations 550 and different vehicle data 250. This creates a semantic representation that can be used for data analysis. In order to be able to represent the data, the vehicle data 250 has already been converted into a standardized format either by the data acquisition device 220 of the vehicle 200 or by a software application of the database 300 or by the software application 720 of the evaluation module 700, so that the vehicle data 250 has the same structure regardless of the type of annotation 550.

[0057] To train the embedding model 750, the data sets D1, D2, ..., Dn, each consisting of the vehicle data 250 and the annotation 550, are transformed by the software application 720 into a data vector 785 located in a multidimensional vector space 780. Techniques such as one-hot encoding, word embedding, or other vectorization methods can be used for this purpose. It can also be provided to convert the vehicle data 250 into a first data vector and the annotation 550 into a second data vector, which are then used to train various layers of the embedding model 750. The representation of the data sets D1, D2, ..., Dn as a data vector 785 enables the data to be processed by a neural network 770.

[0058] Neural networks are often cited as prototypes for AI algorithms. Neural networks are also interesting because they can be combined to increase learning capacity. Such coupled neural networks are also known as deep learning. A neural network consists of neurons arranged in multiple layers and connected to each other in various ways. A neuron is capable of receiving information from outside or from another neuron at its input, evaluating it in a specific way, and passing it on in a modified form at the neuron's output to another neuron or outputting it as the final result. Hidden neurons are located between the input neurons and the output neurons. Depending on the type of network, there can be several layers of hidden neurons. They ensure the forwarding and processing of information. Output neurons ultimately provide a result and pass it on to the outside world.The arrangement and connection of neurons creates different types of neural networks, such as feedforward neural networks (FNN), recurrent neural networks (RNN) or convolutional neural networks (CNN).

[0059] The neural network 770 is trained with the plurality of data vectors 785 for the plurality of data sets D1, D2, ..., Dn to generate an embedding model 750 of the data vectors 785 in the multidimensional vector space 780. Training the embedding model 750 with the neural network 770 thus refers to the process of embedding the data vectors 785 in the multidimensional vector space 780. The embedding model 750 of the data sets D1, D2, ..., Dn represents the annotations 550 for the respective vehicle data 250. In particular, the embedding model 750 can contain different layers for different data types such as speed, position, sensor data, etc.

[0060] Fig. 2 shows a schematic representation of an embedding model 750. Since the embedding of the data vectors 785 is carried out according to the degree of their similarity, similar vehicle data 250 with similar annotations are located close to each other in this vector space 780.

[0061] For embedding data vectors 785 based on their similarity, cosine similarity or cosine distance is used. Cosine similarity is a metric for quantifying the similarity between two vectors in a multidimensional space and measures the cosine of the angle between two vectors. The closer the cosine similarity value is to 1, the more similar the vectors are; the closer it is to -1, the more different they are. A value of 0 means that the vectors are orthogonal to each other and therefore have no similarity.

[0062] Depending on the type of data and the specific task, the network architecture of the neural network 770 can be a simple feed-forward network (FFN) or a more complex model such as a convolutional neural network (CNN) or a recurrent neural network (RNN). Additionally, an embedding layer is added to the network architecture, which generates the embedding model 750 for the data vectors 785 to be learned. Furthermore, the size of the data vectors 785 and other hyperparameters are defined. The neural network 770 is then trained to cluster the data vectors 785 to generate the embedding model 750.

[0063] Various methods such as deep metric learning or constraint pretraining can be used to train the neural network 770 to create the embedding model 750. During the training process, the weights of one or more annotations in the embedding layer can be adjusted. This allows the creation of an embedding model 750 that delivers optimized results for specific tasks related to a particular driver assistance or driving function.

[0064] Deep metric learning is a technique that focuses on learning distance or similarity measures between data points. This approach aims to develop models that represent data in a space where similar data points are close together and dissimilar data points are far apart. By building a model of the similarity relationships between data points, similar time series can be identified in the vehicle data 250.

[0065] Contrastive pretraining refers to a model training method in which a model is trained to learn differences (contrasts) between data points. The model learns to bring similar data points closer together (similar code) and to separate dissimilar data points (dissimilar code).

[0066] After the embedding model 750 has been created, appropriate metrics can also be used to check whether the quality and accuracy are satisfactory.

[0067] The generated embedding model 750 is then used to perform data analysis. This may include clustering similar data sets Dj, Dk, comparing semantic similarities between different annotations 550, or performing classification tasks.

[0068] Clustering algorithms are used to determine clusters in an embedding model 750. Since similar data vectors 785 are located close to each other in the vector space 780, clustering these data vectors 785 allows similar data structures, such as certain structured time series in the vehicle data 250, to be identified for similar annotations 550 and organized into clusters. For example, similar vehicle data 250 can be assigned to the annotations "critical engine noise" or "traffic sign not recognized."

[0069] Clustering algorithms are machine learning and data analysis techniques used to organize similar data points into groups or clusters. Clusters are groups of data points that are more similar to each other than data points in other groups. There are different types of clustering algorithms, which are further divided into hierarchical and partitioning clustering algorithms.

[0070] Hierarchical clustering algorithms organize data into a hierarchical tree structure of clusters. They often begin with each data point as a separate cluster and then gradually group clusters until the desired grouping is achieved. An example of hierarchical clustering is mean-shift clustering, a technique that uses density estimates to find cluster centers and assigns data points to the nearest centers.

[0071] Partitioning clustering algorithms divide the data into a specified number of clusters and attempt to find an optimal distribution of the data within the clusters. K-Means, for example, is a partitioning clustering algorithm in which K clusters are specified and the data points are assigned to the nearest cluster centers. The selection of the appropriate clustering algorithm also depends on the specific requirements of the respective driver assistance or driving function.

[0072] The selected clustering algorithm is now applied to the data vectors 785 in the embedding model 750 and assigns the data vectors 785 to the corresponding clusters based on their similarities in the vector space 780.

[0073] The results of the cluster analysis can be visualized and evaluated using another software application. The visualization shows how the annotations 550 are arranged in the clusters. Various evaluation metrics can be used to assess the quality of the clusters.

[0074] The clusters found can now be interpreted. For example, the clusters from the embedding model 750 provide information about the relationship between a specific environment, such as a parking garage, and the performance of a parking function. While the parking function always works well in a first parking garage, this is not the case for a large portion of the fleet's vehicles in a second parking garage.

[0075] The results of the cluster analysis in the form of visualizations and evaluations can be displayed by an output module 800 as output data 850. The output module 800 can be located, in particular, on a computer or a mobile device. The output data 850 can be output, in particular, in the form of a report and can contain diagrams, graphics, text messages, video sequences, PowerPoint presentations, etc.

[0076] In a further development of the invention, it can be provided that the embedding model 750 is also used for data records that only have the vehicle data 450 but no annotation 550, for example because no feedback has been provided by a user. Although this vehicle data 450 is now unlabeled, the data record can be converted into a data vector 785. This data vector 785 is now inserted into the embedding model 770 by the neural network 770 and is now located near data vectors 785 with a similar data structure. Since this data structure is provided with a specific annotation 550, this known annotation 550 can now also be used for the unlabeled data record, so that in this way a label, i.e. an annotation 550, can now be assigned to the originally unlabeled data record.

[0077] In Fig. 3 shows the process steps of a method for labeling vehicle data, in particular for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle.

[0078] In a step S10, vehicle data 250 are acquired by at least one sensor such as an acceleration sensor, a steering angle sensor, a camera, a lidar system, a radar system and / or an ultrasound system of a data acquisition device 220 of at least one vehicle 200 while driving a plurality of routes, wherein each route comprises a plurality of route sections S1, S2, ..., Sn and each route section Sj comprises one or more scenarios.

[0079] In a step S20, feedback data 250 is entered by a user in the form of an audio recording, a text message, or a symbol, wherein the feedback data 250 relates to the behavior of the vehicle 200.

[0080] In a step S30, the feedback data 250 are converted into an annotation 550, wherein the annotation 550 reflects the essential meaning of the feedback data 450.

[0081] In a step S40, the annotation 550 is assigned to the vehicle data 250 that were recorded during a defined time interval Δt or while driving along a specific route section Sj, and a data set Dj is generated, wherein the data set Dj contains the annotation 550 and the vehicle data 250.

[0082] In a step S50, the data set Dj is transformed into a multidimensional data vector 785.

[0083] In a step S60, an embedding model 750 is trained with a neural network 770 by embedding the data vectors 785 into a multidimensional vector space 780, wherein the embedding of the data vectors 785 is carried out according to the degree of their similarity, so that vehicle data 250 with similar annotations 550 form clusters.

[0084] In a step S70, clusters in the embedding model 750 are determined using clustering algorithms, whereby critical vehicle data 250 can be identified.

[0085] Fig. 4 schematically illustrates a computer program product 900 comprising executable program code 950 configured to perform the method according to the first aspect of the present invention when executed.

[0086] By labeling vehicle data with feedback data from a vehicle user according to the invention, data sets are generated that can be analyzed using an embedding model. The embedding model is trained with data vectors generated from the generated data sets. Vehicle data with similar feedback data form clusters and can then be analyzed specifically. Depending on the significance of the respective feedback data, which is summarized in an annotation, critical vehicle data can be identified. The data volume of a customer vehicle fleet can thus be used in a targeted and efficient manner. Reference symbol 100 systems 200 vehicles 220 Data acquisition device 250 vehicle data 300 database 350 additional data 400 input module 450 Data entry 500 extraction module 520 software application 550 Annotations 700 evaluation module 720 software application 750 Embedding 770 neural network 780 Vector space 785 data vector 800 output model 850 Report 900 computer program product 950 program code

Claims

[1] Method for labelling vehicle data, in particular for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle, comprising the following process steps: - capturing (S10) vehicle data (250) by at least one sensor such as an acceleration sensor, a steering angle sensor, a camera, a lidar system, a radar system and / or an ultrasound system of a data acquisition device (220) of at least one vehicle (200) when driving a plurality of routes, each route comprising a plurality of route sections (S1, S2, ..., Sn) and each route section (Sj) comprising one or more scenarios; - inputting (S20) feedback data (250) by a user in the form of an audio recording, a text message or a symbol, wherein the feedback data (250) relates to the behavior of the vehicle (200); - converting (S30) the feedback data (250) into an annotation (550), wherein the annotation (550) reflects the essential meaning of the feedback data (450); - Assigning (S40) the annotation (550) to the vehicle data (250) that were recorded during a defined time interval (Δt) or while traveling a specific route section (Sj), and generating a data set (Dj), wherein the data set (Dj) contains the annotation (550) and the vehicle data (250); - transforming (S50) the data set (Dj) into a multidimensional data vector (785); - training (S60) an embedding model (750) with a neural network (770) by embedding the data vectors (785) in a multidimensional vector space (780), wherein the embedding of the data vectors (785) is carried out according to the degree of their similarity, so that vehicle data (250) with similar annotations (550) form clusters; - Determining (S70) clusters in the embedding model (750) by means of clustering algorithms, whereby critical vehicle data (250) can be identified. [2] The method of claim 1, wherein the feedback data (450) is converted into an annotation (550) by a software application (520), the software application (520) using speech recognition algorithms for acoustic sequences, word processing algorithms for text messages, and image processing algorithms for graphic symbols. [3] The method according to claim 1 or 2, wherein the vehicle data (250) is bus data of the vehicle (200) which is recorded as time series data and stored in a cloud-based database (300). [4] Method according to one of claims 1 to 3, wherein the network architecture of the neural network (770) is designed as a simple feed-forward network (FFN) or as a more complex model, such as in particular a convolutional neural network (CNN) or a recurrent neural network (RNN). [5] Method according to one of claims 1 to 4, wherein the cosine similarity is used to embed the data vectors (785) in the vector space (780) with regard to their similarity. [6] Method according to one of claims 1 to 5, wherein deep metric learning and / or constraint pretraining are used for training the neural network (770) to create the embedding model (750). [7] Method according to one of claims 1 to 6, wherein hierarchical clustering algorithms such as in particular mean-shift clustering and / or partitioning clustering algorithms such as in particular K-Means are used as clustering algorithms. [8] System (100) for labeling vehicle data, in particular for the verification, validation and testing of a driver assistance system (ADAS) and / or an automated driving system (ADS) for a vehicle, comprising a database (300), an input module (400), an extraction module (500), an evaluation module (700) with an embedding model (750) and an output module (800), wherein the system (100) is designed to carry out the method according to one of claims 1 to 7. [9] The system (100) of claim 8, wherein the feedback data (450) is converted into an annotation (550) by a software application (520), the software application (520) using speech recognition algorithms for acoustic sequences, word processing algorithms for text messages, and image processing algorithms for graphic symbols. [10] System (100) according to claim 8 or 9, wherein the vehicle data (250) is bus data of the vehicle (200) which is recorded as time series data and stored in a cloud-based database (300). [11] System (100) according to one of claims 8 to 10, wherein the network architecture of the neural network (770) is designed as a simple feed-forward network (FFN) or as a more complex model, such as in particular a convolutional neural network (CNN) or a recurrent neural network (RNN). [12] A computer program product (900) comprising an executable program code (950) configured to carry out the method according to any one of claims 1 to 7 when executed.

Images