Method for generating data for machine learning of systems in vehicles

Abstract

Method for generating data for machine learning of systems (1) for evaluating sensor data from environmental sensors (2) in vehicles (3), comprising the following steps: a) creating video data with a camera (5) installed on a drone (4, 4') that follows a road (6) or accompanies a vehicle (3) traveling on the road above; b) Determining environmental data using environmental sensors (2) of a vehicle (3) traveling on the road; c) detecting identifiable markers (7) in the video data and creating an abstract representation of the identifiable markers (7) to generate the data for machine training of systems (1) for evaluating sensor data from environmental sensors (2) in vehicles (3).

Description

State of the art

[0001] The invention relates to a novel approach that can be used to create a reference model of a road and the road's surroundings. Such a reference model can be used for machine training of systems in vehicles, and in particular for training neural networks for evaluating sensor data from vehicle environmental sensors. Such a reference model, created using the method described here, can also be used, in particular, to train systems in vehicles to directly construct a lane model for describing lanes in the vehicle's surroundings based on sensor data from environmental sensors.

[0002] A reference model describes a specific traffic situation in such a way that the model can be used to train vehicle systems. A reference model, in particular, contains training data for training neural networks.

[0003] Such data is also referred to as "labeled data" or "ground truth" data. This means that certain information is known about the data and that a neural network is supposed to recognize independently from comparable data. Such data is required for training neural networks because information can be incorporated into neural networks using such data. For training neural networks to evaluate sensor data from a vehicle, regular "ground truth" data, for example, is required. Such data shows where certain objects in the vehicle's surroundings, which can be detected by the vehicle's sensors, actually are located in a ground-referenced coordinate system used to describe the vehicle's surroundings. Vehicle sensors can only ever detect objects from the respective perspective / viewing perspective of the sensor from the vehicle.With the help of “ground truth” data concerning such objects, neural networks can be trained to evaluate sensor data from such sensors in order to reliably detect the position of objects in the ground-related coordinate system for describing the environment of the motor vehicle.

[0004] Obtaining such "ground truth" data is complex. A well-known approach is to obtain such data with the help of a drone that accompanies a vehicle conducting test drives to obtain data for training neural networks. Such a drone preferably hovers above the test vehicle at all times during the test drive. Such a drone is preferably equipped with a camera that detects the test vehicle and its surroundings, as well as the objects in this surroundings. Due to the top-down perspective with which such a camera views the vehicle and the objects, "ground truth" data can be detected much more easily and accurately with such a drone and its camera, using less complex analysis systems than those required for evaluating data from sensors on the vehicle.By positioning a drone at an adjustable distance above the vehicle, obtaining training data in this way is very advantageous – especially if the training data to be obtained includes the described "ground truth" data. Such approaches, with a drone flying above a vehicle, have so far been used primarily to generate training data for detecting moving objects in the vehicle's vicinity, such as other vehicles or other road users.

[0005] Reference models that can be obtained using the method described here include, for example, but are not limited to, such "ground truth" data. Reference models that can be obtained using the method described here are particularly suitable for training systems in vehicles to build a lane model for describing lanes in the vehicle's environment based on sensor data from environmental sensors in the vehicle. Such reference models or the data obtained using the method described can also be used to test the functions of algorithm-based systems that process data from environmental sensors in vehicles.

[0006] The following approaches are currently known to obtain data for such reference models: (1) During test drives with a test vehicle, data is acquired using regular sensor systems installed on the test vehicle. This data is recorded and then electronically processed to generate reference models. Regular sensor systems, in this context, are the sensor systems that correspond to the sensors whose data analysis will later be trained using the reference model. (2) It is known to install additional sensors on test vehicles that go beyond the regular sensor systems or that can acquire additional data with which the data detected by regular sensor systems can be validated. Such additional sensors can, for example, be high-precision sensors. The data from such additional sensors is used to create reference models. With such additional sensors, for example, distances to objects that were also detected with the regular sensor systems can be measured much more precisely. The data from such additional sensors can therefore be used to train systems for evaluating the data acquired with regular sensor systems. Such additional sensors and the processing of data acquired with such additional sensors requires a great deal of effort.This is partly because the additional sensors provide data of significantly higher quality than regular sensor systems, which are often much more expensive. This is also often because the amount of data such additional sensors acquire is often very large. (3) Furthermore, it is known that additional sensors are used to create reference models, which are not located in the respective test vehicle but, for example, on other vehicles driving in front of or behind the respective test vehicle or with sensors in other infrastructure that is present along the test track covered by the test vehicle. Such approaches are very complex. In particular, if the sensors are located in infrastructure along the test track, this requires complex preparation of the test track for the tests to be carried out to generate reference models. Due to the need for complex preparation of test tracks, the scenarios presented on such test tracks often lack a certain degree of realism and there is also a risk that systems are trained with unrealistic data.In principle, data obtained in real traffic situations has a higher probability of corresponding to real traffic situations. (4) In addition, approaches are already known for using high-resolution aerial photographs or videos taken from the air (from above), for example using flying drones. In particular, if such aerial photographs or videos were taken independently of the test drive actually being carried out, the amount of data from such aerial photographs and videos is very high if their use in connection with the test drive to generate reference models is to be meaningful. In addition, the effort required to link such aerial photographs and / or videos with data acquired by the sensors on the test vehicle is extremely complex. This is particularly true when it comes to creating so-called HD maps. In general, the effort required to create HD maps is already very high. The use of videos from drones that fly over a terrain without further coordination increases this effort even further.The amount of data to be processed is increasing dramatically. In particular, differences between the recording times of individual videos can cause additional problems in the data processing required to create HD maps. Aligning sensor data from the vehicle's sensors and the resulting HD map requires additional effort. (5) Furthermore, it is known to use the trajectories along which vehicles are moving to simplify the linking of data from different data sources (e.g., the linking of data from the test vehicle's sensors and data from aerial photographs) and, in particular, to detect the location of lanes. However, this approach also presents difficulties. The following difficulties of such approaches can be mentioned here: • The trajectories of test vehicles can, under certain circumstances, be determined very precisely using high-precision satellite navigation. However, inferences about lanes from these trajectories are often limited, because there is no guarantee that the test vehicles will follow lanes precisely during test drives. For this reason, the trajectories of test vehicles cannot usually be used directly to determine lane trajectories. • In particular, it is possible that test vehicles may not travel in every lane with the same degree of precision. Furthermore, vehicles may also violate traffic regulations (especially when data is collected in the field). This occasionally occurs unplanned or unintentionally, or may be triggered by the specific traffic situation. This results in uncertainties in the data. • If lanes are to be identified based on the observation of vehicles from a video recorded with a drone camera, a very long observation of the lane with the drone camera is necessary. Such observation requires collecting a large amount of video footage so that statistical analyses of the movement of vehicles along the lane can be used to make conclusions about the lanes. (6) Furthermore, it is known to use data obtained in the field from vehicles during their regular operation to generate reference models. Vehicles or field trips are referred to as trips with regular vehicles (not specially prepared test vehicles) performing typical trips made by users of such vehicles. These can be, for example, private trips, commutes, or any other type of trip undertaken by a driver in a motor vehicle. Such approaches are also referred to as "crowd-based." • Collecting data in the field also requires a great deal of effort. Especially if special sensors are necessary / helpful for data collection, the sensors must be implemented in a large number of vehicles. Essentially, every vehicle involved in data collection must be equipped with the appropriate sensors. • Conversely, if the vehicle's regular sensors are used, the quality of the acquired data is limited or does not exceed the quality of the sensors whose evaluation is to be trained. This presents a fundamental difficulty with such approaches. • If data from different journeys by different vehicles from the “crowd” are combined, further errors can arise - for example from differences in the situation at the respective point on a road at different times or from inaccurate or different calibration of sensors from different vehicles. • A high dependence on the accuracy of positioning vehicles in the field regularly poses a problem. The precision of (especially GNSS-based) navigation modules used in the field is regularly limited. Disclosure of the invention

[0007] Based on this, the object of the present invention is to at least partially solve the problems described with reference to the prior art. This object is achieved by the invention according to the features of the independent patent claims. Further advantageous embodiments are specified in the dependent claims as well as in the description and, in particular, in the description of the figures. It should be noted that the person skilled in the art can combine the individual features in a technologically expedient manner and thus arrive at further embodiments of the invention.

[0008] In particular, an improved approach is proposed here with which reference models of roadways or routes can be created on which systems for highly automated / autonomous driving are to be trained.

[0009] A method for generating data for machine learning of systems for evaluating sensor data from environmental sensors in vehicles is described, comprising the following steps: (a) Creating video data using a camera installed on a drone that follows a road or accompanies a vehicle traveling on the road above; b) Determining environmental data using environmental sensors of a vehicle driving on the road; c) Detecting identifiable markers in the video data and creating an abstract representation of the identifiable markers to generate the data for machine training of systems for evaluating sensor data from environmental sensors in vehicles.

[0010] It is particularly advantageous if, in step c), identifiable markers and / or abstract representations of the identifiable markers are linked to environmental data determined according to step b).

[0011] According to step a) of the described method, a drone is used which acquires video data containing information about the road either by accompanying a vehicle driving on the road or by directly following the course of the road, whereby the amount of other data (which does not concern the road and its immediate surroundings) is relatively small.

[0012] According to the described method, it is proposed to detect identifiable markers in the video data thus obtained and to generate abstract representations of the identifiable markers (see step c). These abstract representations of the identifiable markers can then be used as data for machine training of systems for evaluating sensor data from environmental sensors in vehicles.

[0013] Identifiable markers can be all types of structures on and / or in the surroundings of the road that can be detected by environmental sensors in a vehicle and whose classification / assignment / recognition is helpful through the evaluation of sensor data from such environmental sensors.

[0014] Fundamentally, vehicle systems for autonomous and / or highly automated driving should be capable of generating the most complete image of the vehicle's surroundings possible, using both sensor data and map data for this purpose. Abstract representations of identifiable markers detected in video data acquired according to step a) provide highly suitable training data for training such systems. In particular, the amount of such training data is relatively small compared to the achievable training results.

[0015] The method described here can be used, in particular, to generate training data that enables highly efficient training of systems for detecting lanes in the vicinity of vehicles. Such systems make it possible to detect lanes in the vicinity of a vehicle using sensor data from the vehicle's environmental sensors. For this purpose, the detection of lane markings is usually necessary. Lane markings are preferably recognized as identifiable in the video data, and abstract representations of lane markings are created. These abstract representations, together with the environmental data from the vehicle's environmental sensors determined in step b), can be used as training data for training such systems.

[0016] Particularly preferably, the identifiable markers or abstract representations are detected and determined in a vehicle coordinate system. A vehicle coordinate system preferably moves together with the vehicle whose environmental sensors are used to determine environmental data in step b). Such a coordinate system typically also contains the environmental data from the environmental sensors determined in step b).

[0017] Preferably, sensor data from environmental sensors of a vehicle or test vehicle are also recorded in parallel, simultaneously, or in another way linked to the creation of the video data with the camera installed on the drone according to step a). Preferably, in a further processing step following step b), this sensor data is linked to the identifiable markers detected in step b) or the abstract representations of these identifiable markers. The term "linked" here means that data is generated that contains sensor data from environmental sensors and also contains the identifiable markers or their abstract representations that a system for evaluating the sensor data is intended to later recognize in comparable sensor data without using video data recorded with a camera on a drone.

[0018] It is particularly preferred if step a) comprises the following sub-steps: a1) Creating initial video data using a camera installed on a drone following a road; a2) Creating second video data with a camera installed on a drone, which accompanies a measuring vehicle driving on the road above; and wherein step b) is carried out for first video data and second video data, and wherein the following step is carried out further after step b): d) linking the first video data and the second video data based on identifiable markers in the first video data and in the second video data to generate linked data;

[0019] In particular, when identifiable markers and / or abstract representations of identifiable markers are not determined in the vehicle coordinate system, but in another (e.g., a global (superordinate) coordinate system), the linking of first video data and second video data generated in step d) enables the representation of information extracted from the video data (the identifiable markers and / or their abstract representations) in a vehicle coordinate system.

[0020] In particular, it is proposed to record the journey of a test vehicle (step a1) as well as the course of a road over which the test vehicle is moved (step a2), each with flying drones.

[0021] By detecting identifiable markers in the video data obtained according to step a1) and step a2) and creating an abstract representation of these identifiable markers with the first video data and the second video data, the first video data and the second video data can be very well assigned to each other.

[0022] The two videos are specifically linked. The video data recorded in step a1) enables a link to a global coordinate system. The video data recorded in step a2) allows the positions of objects along the route to be precisely recorded in a vehicle coordinate system.

[0023] In principle, a vertically downward-facing camera on a (flying) drone is suitable for obtaining particularly advantageous data concerning a road and / or traffic scene due to its viewing angle.

[0024] The advantages of video data obtained with such a flying drone are used in a particularly efficient way with the approach described here.

[0025] It is particularly advantageous if steps c) and / or d) are sub-steps in the creation of a reference model for the machine training of systems for evaluating sensor data from environmental sensors in vehicles.

[0026] A reference model, created based on, among other things, first video data and second video data acquired and linked using the method described here, can be used to represent and simulate a test vehicle and other road users in the reference model. This allows scenes to be recreated, which can be used to train sensors and neural networks for evaluating sensor data for a wide variety of traffic situations.

[0027] It is also advantageous if the reference model includes a virtual test track for training systems for evaluating sensor data from environmental sensors of highly automated / autonomously operated vehicles.

[0028] Furthermore, it is advantageous if the identifiable markers comprise at least one of the following objects: - Lane markings on the road and / or - Road surface structures.

[0029] Road markings or markings on the road surface can be used to precisely link the first video data and second video data generated according to steps a), a1), and / or a2). Specifically trained neural networks can be used to create this link. The linking of first video data and second video data can be performed offline. This means that first video data and second video data are first generated and stored according to steps a) and b). Step c) can then be performed at any later time, possibly with the addition of a large amount of additional data.

[0030] It is also advantageous if drones in step a) fly along an observation path and generate first video data and second video data from observation positions arranged above the road and / or above the vehicle.

[0031] It is also advantageous if an observation path along which the drone is moved according to step a1) / a2) is specified by GNSS data that describe the course of the road.

[0032] Furthermore, it is advantageous if an observation path along which the drone is moved according to step a2) is predetermined by a travel route of the measuring vehicle, wherein a marking is arranged on the measuring vehicle, which is observed by the drone during step a2), wherein the drone follows the marking.

[0033] It is also advantageous if the detection of identifiable markers in the video data and the creation of an abstract representation of the identifiable markers according to step c) are carried out using neural networks trained for this purpose.

[0034] In particular, so-called feature detection algorithms can be used for linking, which also include neural networks and which may also have been trained with image data or video data obtained in other ways (in particular not with cameras on drones).

[0035] Furthermore, it is advantageous if in step c) and / or d) the detection of the position of objects in the environment of the measuring vehicle is / is included in a global coordinate system and in a vehicle coordinate system.

[0036] For all data relating to objects detected in the first video data and / or the second video data, the described approach makes it possible to describe this data either in a vehicle coordinate system or in a global coordinate system. The data obtained with the described method thus provides a particularly advantageous starting point for further steps in the creation of reference models for machine training of neural networks for evaluating sensor data from environmental sensors in vehicles.

[0037] Furthermore, it is advantageous if, according to step c), data relating to the position of objects in the environment of the measuring vehicle are obtained, which data can be used to train the position determination of objects in the environment of the measuring vehicle using environmental sensors in the vehicle.

[0038] In connection with the linking of first video data and second video data, a neural network trained for the linking can also be used simultaneously to create abstract representations of the features used for the linking. For example, polygons can be created that describe road markings.

[0039] It has been found that precise knowledge of lanes in such a reference model is particularly relevant for a high quality of the reference model. This is already clear from the example of a relatively simple driver assistance function such as a lane keeping assistant, for whose functionality the lanes of a road are of great importance. As described above, the described method generates data that is very well suited to training systems for such driver assistance functions, while at the same time the amount of data is relatively small for the quality of the training results to be achieved. Particularly preferably, map data is created according to steps c) and / or d) that describe a road model based on data in the vehicle coordinate system and on data in the global coordinate system, whereby the map data takes both perspectives into account.In this context, the linking of the data obtained in steps a), a1), a2) and the data obtained in step b) is preferably carried out via road markings.

[0040] The described method is also advantageous if the following step is carried out following step b) and / or c): e) Training of a system for evaluating sensor data from environmental sensors in vehicles with data obtained according to the method.

[0041] Also to be described is a system for evaluating sensor data from environmental sensors in vehicles, which was trained with data generated according to a described method.

[0042] Such a system has, in particular, at least one processor and one or more memory devices, and is preferably configured to provide a neural network that assists in evaluating the sensor data from environmental sensors. This neural network of the system is preferably trained using the described method.

[0043] Also to be described here is a device for data processing comprising a processor which is adapted and / or configured to carry out the described method and in particular the evaluation of video data generated by cameras on drones according to method steps b) and c).

[0044] A computer program product is also to be described, comprising instructions which, when the computer program product is executed by a computer, cause the computer to carry out the described method.

[0045] Furthermore, a computer-readable storage medium is to be described, comprising instructions which, when executed by a computer, cause the computer to carry out the described method.

[0046] The invention and the technical context of the invention are explained in more detail below with reference to the figures. The figures show preferred embodiments to which the invention is not limited. It should be noted in particular that the figures, and in particular the proportions depicted in the figures, are only schematic. They show: Fig. 1: a schematic representation of a drone observing a vehicle; Fig. 2: a road along which a vehicle is moved and on which the described procedure is applied; and Fig. 3: a flowchart of the described procedure.

[0047] The Fig. 1 shows a schematic representation of a drone 4 that accompanies a vehicle 3 and observes from an observation position 9 above the vehicle 3. The observation position 9 can (but does not have to) be located exactly vertically above the vehicle 3. Depending on the driving speed of the vehicle 3, there may be a certain shift between the position exactly above the vehicle 3 and the observation position 9, because the drone 4 must follow the vehicle 3. The drone 4 has a camera 5 with a downward-facing field of view 17 and with which camera data or video data can be generated in which the vehicle 3 and other objects 13 in the vicinity of the vehicle 3 are recognizable / visible. Camera data acquired with the camera 5 on the drone 4 are present in a drone coordinate system 16, which is indicated schematically here.The camera data is to be used to build a reference model with which sensors on the vehicle 3 can be calibrated or algorithms for evaluating the data from such sensors can be improved. Data obtained with environmental sensors 2 on the vehicle 3 is available in a vehicle coordinate system 14. In order to use the camera data from camera 5, a conversion from the vehicle coordinate system 14 to the drone coordinate system 16 and / or vice versa is required. To enable this conversion, the orientation of the camera 5 and the observation position 9 of the vehicle 3 must be known in the camera data or relative to the camera 5 of the drone 4. Such calibration is possible using a marking 10.

[0048] Fig. 2: shows a road 6 over which a vehicle 3 is moving and on which the described method is applied. Two drones 4, 4' are shown. One drone 4 follows (as in Fig. 1) is attached to the vehicle 3 and is controlled in an observation position 9 above the vehicle 3. This drone 4 thus flies an observation path 8 that corresponds to the route traveled by the vehicle 3. The other drone 4' follows the course of the road 6 along a corresponding observation path 8. This observation path 8 can, for example, be specified by GNSS coordinates that depict the course of the road 6 in map data. Both drones 4, 4' generate or collect video data with their cameras, which can be linked together using the method described here. The linking of the video data recorded by the two drones 4, 4' along the various observation paths 8 can, for example, be done using identifiable markers 7 that are recognizable in both video data from both drones 4, 4'. These markers 7 can (for example) be road markings, as shown here.In both video data from both drones 4, 4', in addition to the vehicle 3, other objects 13 in the vicinity of the vehicle 3 can be seen, which can be detected by environmental sensors 2 on the vehicle 3. These other objects 13 can also be other vehicles 12 on the observed road. The video data recorded by the drone 4 following the course of the road 6 enables a link between a drone coordinate system 16 of the drone 4' following the course of the road 6 and a global coordinate system 15, in which, for example, further map data is also available. The video data recorded by the drone 4 following the vehicle 3 enables a link between the drone coordinate system 16 of this drone 4 and a vehicle coordinate system 14, in which, for example, environmental sensors 2 on the vehicle 3 detect objects.The identifiable markers 7 allow the drone coordinate systems 16 of the two drones 4, 4' with their different observation trajectories 8 to be linked. The described approach allows particularly high-quality data to be generated for a reference model, which can be used to train algorithms or systems for evaluating sensor data from environmental sensors 2 on the vehicle 3. In particular, it becomes possible to train algorithms for evaluating sensor data from environmental sensors 2 particularly well for detecting distances to other objects 13 in the environment of the vehicle 3.

[0049] The Fig.3 schematically shows a flow diagram of the described method as well as the embedding of the described method in the creation of a reference model 11 for training systems 1 for evaluating data acquired with environmental sensors 2 of a motor vehicle. With steps a) or a1) and a2) and b), video data is acquired, which is then combined with each other according to step c) and, if necessary, d). The resulting (combined) data can be used for step e), in particular for improving or creating a reference model 11. This reference model 11 contains so-called "labeled" data, which contains information concerning objects 13 in the environment of the vehicle 3, for example with precise distance information. At the same time, data from the environmental sensors 2 is preferably available.Systems 1 for evaluating the data from the environmental sensors 2 can be trained with this precise distance information, so that such systems 1 can also use the data from environmental sensors 2 alone to obtain correspondingly precise distance information.

Claims

[1] Method for generating data for machine learning of systems (1) for evaluating sensor data from environmental sensors (2) in vehicles (3) comprising the following steps: a) Creating video data with a camera (5) installed on a drone (4,4') which follows a road course (6) or which accompanies a vehicle (3) traveling on the road above; b) Determining environmental data using environmental sensors (2) of a vehicle (3) traveling on the road; c) Identifying identifiable markers (7) in the video data and creating an abstract representation of the identifiable markers (7) to generate the data for machine learning of systems (1) for evaluating sensor data from environmental sensors (2) in vehicles (3). [2] Method according to claim 1, wherein in step c) identifiable markers (7) and / or abstract representation of the identifiable markers are linked with environmental data determined according to step b). [3] The method of claim 1, wherein step a) comprises the following sub-steps: a1) Creating initial video data with a camera (5) installed on a drone (4') following a road course (6); a2) Creating second video data with a camera (5) installed on a drone (4) which accompanies a vehicle (3) traveling on the road above; and wherein step c) is performed for first video data and second video data, and wherein, further after step c), the following step is performed: d) Linking the first video data and the second video data using identified identifiable markers (7) in the first video data and in the second video data to generate linked data. [4] Method according to one of the preceding claims, wherein steps c) and / or d) are partial steps of the creation of a reference model (11) for machine learning of systems (1) for evaluating sensor data from environmental sensors (2) in vehicles (3). [5] Method according to claim 3, wherein the reference model comprises a virtual test track for training systems (1) to evaluate sensor data from environmental sensors (2) of highly automated / autonomously operable vehicles (3). [6] Method according to any of the preceding claims, wherein the identifiable markers (7) comprise at least one of the following objects: - Road markings on the road and / or - Road surface structures. [7] Method according to one of the preceding claims, wherein drones (4,4') in step a) flying along an observation path (8) from observation positions (9) arranged above the road (6) and / or above the vehicle (3) generate first video data and / or second video data. [8] Method according to one of the preceding claims, wherein an observation path (8) along which the drone (4,4') is moved according to step a) is defined by GNSS data which describe the road course (6). [9] Method according to one of the preceding claims, wherein an observation path (8) along which the drone (4) is moved according to step a) is defined by a travel path of the measuring vehicle (3), wherein a marking (10) is arranged on the measuring vehicle which is observed by the drone (4) during step b), wherein the drone (4) follows the marking (10). [10] Method according to one of the preceding claims, wherein the detection of identifiable markers (7) in the video data and the creation of an abstract representation of the identifiable markers according to step b) is carried out using neural networks trained for this purpose. [11] Method according to any of the preceding claims, wherein step c) comprises the detection of the position of objects (13) in the vicinity of the vehicle (3) in a global coordinate system (15) and / or in a vehicle coordinate system (14). [12] Method according to one of the preceding claims, wherein according to step c) data relating to the position of objects (13) in the vicinity of the vehicle (3) are obtained which can be used to train the position determination of objects (13) in the vicinity of the vehicle (3) with environment sensors (2) in the vehicle (3). [13] Method according to any of the preceding claims, wherein the following step is carried out in addition to step c) and / or d): e) Training of a system (1) for evaluating sensor data from environmental sensors (2) in vehicles (3) using data obtained according to the method. [14] System (1) for evaluating sensor data from environmental sensors (2) in vehicles (3), which was trained with data generated according to a method according to one of the preceding claims.

Images