Method for marking raw sensor data

Abstract

The invention relates to a method for marking raw sensor data of a vehicle, in which raw sensor data and audio data are acquired and recorded, wherein the raw sensor data of the vehicle are marked with time-synchronized audio data that specifies meta-information of the raw sensor data marked therein.

Description

The invention relates to a method for marking raw sensor data, an audio detection and recording device for carrying out this method and a preferred use of this device.In many automobile developments, particularly data driven developments, data is an important factor. Data markers (tags) help describe data facts that allow efficient data use. The large amount of data arising in data-controlled developments can thus be structured in a meaningful manner, subsequently be analyzed better and used, for example, for training neural networks. However, the tags must also be able to be introduced easily and reliably under experimental conditions in order to enable marking.An object of the present invention is to specify a method which enables simple marking of raw sensor data of a vehicle.The object is achieved by the subject matter of claim 1. Accordingly, it is proposed to mark raw sensor data with audio data as input, which is synchronized with the raw sensor data via time synchronization. Such an audio input can also be carried out easily and reliably under test conditions in order to mark (tag) raw sensor data. The dependent claims comprise preferred embodiments.According to an advantageous embodiment of the method according to the invention, the audio data are converted into text before the marking of the raw sensor data, and a respective data marker for marking the raw sensor data is derived therefrom. As a result, the search for a predefined day in the raw sensor data, for example, is very particularly simple and a data area to be analyzed therein can easily be found.It is advantageous in this case if the audio data is at least partially classified and class-dependent predefined data markers are determined, from which it is known which raw sensor data describe them and which are to be specified thereafter in the case of a search.In the case of at least partially unclassified audio data, on the other hand, it is preferred to use the text itself as a data marker. For a non-predefined tag, the tags converted to text are recorded and then manually inspected. Their use is then defined.According to a further advantageous embodiment of the method according to the invention, the audio data are recorded continuously or discontinuously during a recording process, whereby different triggering mechanisms can be realized. Triggering by specific events would be typical, for example.In yet another advantageous embodiment of the method according to the invention, the raw sensor data comprise image data, the images (frames) of which are marked individually and / or across groups. During a vehicle attempt, frames may be recorded continuously, for example, in order to detect the environment of a vehicle. Each camera can record 30 different images per second. The tag may now be frame- or event-based, meaning that a tag may provide metadata of a single frame or metadata of multiple frames. Event-based tags may be pre-defined or non-pre-defined and used to identify driving scenarios.In the method according to the invention, the data markers are preferably recorded together with the raw sensor data in a metadata database. For a predefined tag, the tag is captured from the converted text and stored in this metadata database. Along with the day, the acquired raw sensor data is recorded. The tag can then be used as a keyword in a database query and the corresponding frames located for further use.In the method according to the invention, it is advantageous if the data markers are embedded in the raw sensor data in order to ensure rapid and easy access to sought raw sensor data, for example, of an event such as passing through a circular traffic.Preferably, the data markers are used to find raw sensor data. The raw sensor data retrieved may be used to develop Advanced Driver Assistance System (ADAS) perception functions. For example, the raw sensor data may be labeled to train a deep neural network for detection of particular objects. The raw sensor data may also be labeled to validate and test software components or systems for sensing.In principle, the method according to the invention is not restricted to application to image data. The environment of a vehicle can also be described by further data, such as environmental influences (wind, temperature, precipitation, road condition, lighting, etc.). Preferably, however, the raw sensor data represent a pictorial environment of the vehicle, the knowledge of which is fundamental to autonomous driving.It is therefore also particularly preferred if the data markers mark driving scenarios in the raw sensor data in order to obtain information about this raw sensor data and to supply this raw sensor data to a further use.The present object is also achieved by an audio data recording and processing apparatus for generating data markers in the method according to the invention as claimed in claim 12. In this device, audio data can be converted into text and data markers for raw sensor data can be derived and output therefrom, which contain meta-information of the marked raw sensor data.A data recording system for the raw sensor data is a device or a function that records time-series data during a certain period of time. An example is the AVL Dynamic Ground Truth System (DGT), a standard measurement solution for acquiring ADAS / AD (Autonomous Driving) sensor data. The DGT system is mounted on the roof of a self-driving (ego) vehicle and is used to record environmental data during the travel of the ego vehicle. Tags are assigned to these environmental data with the audio data recording and processing device according to the invention, which tags make it possible to locate and analyze and to further use the raw sensor data.In the audio data recording and processing device according to the invention, it is advantageous if the audio data recording and processing device is designed as a real-time capable system in order to meet the time standard of the data recording device. This is because the audio tags can then be synchronized directly-i.e. without further temporal references-with the recording device.A further advantage of the audio data recording and processing device according to the invention arises if it is a tablet. This is because a tablet ensures flexible handling of the audio data recording and processing device in a vehicle test.The audio data recording and processing device according to the invention is preferably used for marking raw sensor data in order to train a deep neural network for detecting predetermined objects on the basis of the marked raw sensor data.Further advantages, goals and properties of the present invention will be explained with reference to the following description of the attached figures. Like components may have like reference numerals in the various embodiments. In the figures, the following are shown: FIG. 1 is a functional diagram of an audio recording device with integrated processing software according to the invention, and FIG. 2 is a functional diagram of an alternative audio recording and processing apparatus according to the invention.FIG. 1 shows a functional diagram of an audio recording device 300 according to the invention with integrated processing software 302. The audio recording apparatus 300 has an audio data input interface 303 and audio storing and processing hardware 304, 305 having a real time recording function. Audio data recording software 301 is developed and installed on this device 300 to record audio inputs and store the audio inputs to the storage location, here audio data storage hardware 305. A recording task is defined as the entire process from the deployment of an ego vehicle and a beginning of the data recording to the end of the data recording and the return of the ego vehicle. During a data recording task, the audio data recording and processing apparatus 300 is operated to record audio data.Meanwhile, audio data is recorded together with a data recording system for the raw sensor data.The audio data processing is executed after the audio data recording, whereby the audio data is converted into text data from which tags are derived. This embodiment can be performed with sufficient processing capacity on the audio data recording and processing apparatus 300. Alternatively, the audio data is transferred to other processing hardware, where the audio data processing is then carried out.FIG. 2 shows a functional diagram of an alternative audio recording and processing device 300' according to the invention. The essential difference from the device 300 of FIG. 1 is that here an audio data recording software 301' and an audio data processing software 302' run on respectively different hardware, of which the first hardware acquires an audio data input 100 via an audio data interface 303', and stores it in an audio data storage hardware 305' with the aid of an audio data storage software 301' executable on an audio data processing hardware 304'. The audio data are subsequently passed on to further hardware, which stores the audio data 201 in audio data storage hardware 305' and converts it into text with the aid of audio data processing software 302' executable on the audio data processing hardware 304', derives corresponding tags from said text and provides it via a data marker output 200.In other words, the functions of storing and further processing the audio data are divided among different hardware, whereby larger audio data quantities can be processed in real time. Real time here means that the provision of the tags can proceed synchronously in time with a data recording system for the raw sensor data, so that the relevant raw sensor data can be marked with the corresponding tags without time delay.Namely, ADAS and AD systems generate enormous amounts of data used for software development, testing, validation, etc. Data-controlled software development and tests are becoming increasingly important. A typical perception function is, for example, to recognize objects in the environment of an ego vehicle. One detection approach is to train a deep neural network with marked raw sensor data. The target object must be marked in the raw ADAS / AD sensor data in sufficient quantity and variety. To cover the diversity, the acquisition of raw sensor data from many scenarios that may be considered a kind of day is required. An example scenario is the travel of an ego vehicle through a circular traffic. During data recording, when the ego vehicle is travelling near a circular traffic, an audio input 100 may be recorded to indicate the beginning and end of this scenario, where both beginning and end are tags.For practical reasons, a tablet can be used as a host for the audio recording and processing software 301, 302 shown in FIG. 1. During data recording, the recording of audio data may be triggered by the person performing the recording due to an event. The person performing the recording is also referred to as a recorder below. The recorder monitors the scenario in advance, for example before the ego vehicle is driving, and triggers the audio recording. The recorder then records the audio input representing the scenario.The audio data input 100 may be predefined prior to the recording task, such as 'circular traffic'. In this case, the audio data input 100 is converted to text and the tag is 'circular traffic'. During the data recording task, the recorder can record each undefined audio data input 100. The text converted from the audio data input is used to derive tags based on the information provided by the recorder. There are several approaches to translating audio data input 100 into text. For example, with defined tags, i.e. a classifier for classifying the recorded audio sequences as a predefined tag. For all predefined tags, a certain number of audio examples must be recorded for each tag and the machine learning approach used to train, for example, a classifier to perform the conversion or classification of each new shot. To convert any undefined audio data input, a more complex approach is required. For example, a dataset consisting of audio data and the corresponding texts is created and a deep neural network trained to perform the conversion of each new shot.The audio data recording and processing device 300, 300' for generating data markers in the method according to the invention thus allows simple marking of raw sensor data of a vehicle, in particular of image data of a detected environment of the vehicle.List of reference characters100 Audio data input 200 Data marker output 201 Audio data 300, 300' Audio data recording and processing apparatus 301, 301' Audio data recording software 302, 302' Audio data processing software 303, 303' Audio data input interface 304, 304' Audio data processing hardware 305, 305' Audio data storage hardware

Claims

Method for marking raw sensor data of a vehicle, in which raw sensor data and audio data are recorded and recorded, wherein the raw sensor data of the vehicle are marked with audio data which are synchronized in time and indicate meta-information of the raw sensor data marked therewith.Method according to claim 1, characterised in that the audio data is converted into text before the marking of the raw sensor data, and a respective data marker for marking the raw sensor data is derived therefrom.Method according to Claim 2, characterized in that the audio data are at least partially classified and class-dependent predefined data markers are determined.Method according to claim 3, characterized in that the audio data is at least partially not classified and marked by the text as data markers.Method according to one of the preceding claims, characterized bycontinuously or discontinuously recording the audio data during a recording operation.Method according to one of the preceding claims, characterized in that the raw sensor data comprise image data, the image frames (frames) of which are marked individually and / or across groups.Method according to one of the preceding claims, characterized in that the data markers are recorded together with the raw sensor data in a metadata database.Method according to one of the preceding claims, characterized in that the data markers are embedded in the raw sensor data.Method according to one of the preceding claims, characterized in that the data markers are used to find raw sensor data.Method according to one of the preceding claims, characterized in that the raw sensor data represent a pictorial environment of the vehicle.Method according to one of the preceding claims, characterized in that the data markers mark driving scenarios in the raw sensor data.Audio data recording and processing apparatus for generating data markers in a method according to one of the preceding claims, comprising an audio data input interface, at least one audio data processing hardware, at least one audio data memory, and data recording and processing software which is designed for recording the audio data via the audio data interface, for recording the audio data in the audio data memory and for processing the audio data in the audio data processing hardware, wherein audio data can be converted into text and from this data markers for raw sensor data can be derived and output, which data markers comprise meta information of the marked raw sensor data.Audio data recording and processing device according to claim 12, characterised in that the audio data recording and processing device is designed as a real-time capable system.Audio data recording and processing device according to claim 13, characterized in that the audio data recording and processing device is a tablet.Use of an audio data recording and processing device according to one of Claims 12 to 14 for marking raw sensor data in order to train a deep neural network for detecting predetermined objects on the basis of the marked raw sensor data.

Images