Method for detecting the spatial orientation of an object in camera data

Abstract

Method for detecting a spatial orientation of an object (1) in camera data, comprising the following steps a) providing at least one light source (2) which emits light (10) to the object (1); b) providing a polarization filter (4) with a predetermined orientation in front of a camera (3) for capturing camera data and detecting the light source (2) in video data recorded with the camera (3); c) determining a light intensity of the light source (2) in the camera data; and d) Determining a spatial orientation (5) of the object (1) by evaluating the light intensity determined according to step c).

Description

State of the art

[0001] Correctly recognizing the real world in a vehicle's environment using sensors is a fundamental challenge in the development of algorithms for highly automated and autonomous driving of motor vehicles.

[0002] The development of algorithms requires, in particular, a very large amount of training data that can be used to train neural networks. Such data is also referred to as "labeled data." This means that certain information is known about the data, which a neural network is supposed to recognize independently from comparable data. Such data is needed to train neural networks because such data can be used to incorporate information into neural networks. Regular "ground truth" data is required to train neural networks to evaluate a vehicle's sensor data. Such data shows where certain objects in the vehicle's surroundings, which can be detected by the vehicle's sensors, actually are located in a ground-referenced coordinate system used to describe the vehicle's surroundings.Vehicle sensors can only detect objects from the respective perspective of the vehicle's view. Using ground truth data regarding such objects, neural networks can be trained to evaluate sensor data from such sensors to reliably detect the position of objects in a ground-referenced coordinate system.

[0003] Obtaining such "ground truth" data is complex. A well-known approach is to obtain such data with the help of a drone that accompanies a vehicle conducting test drives to obtain data for training neural networks. Such a drone preferably hovers above the test vehicle at all times during the test drive. Such a drone is preferably equipped with a camera that detects the test vehicle and its surroundings, as well as the objects in this surroundings. Due to the top-down perspective with which such a camera views the vehicle and the objects, "ground truth" data can be generated with such a drone and its camera. By positioning a drone at an adjustable distance above the vehicle, obtaining training data in this way is very advantageous - especially if the training data to be obtained includes the described "ground truth" data.

[0004] Using drones to obtain training data often requires the ability to determine the relative position between the drone and the test vehicle very precisely. This is particularly necessary to correctly align the drone's camera image within the test vehicle's coordinate system. An (undetected) rotational shift of the drone relative to the vehicle is often problematic when using camera images captured with the drone as training data. For example, an angular error of just 0.1 degrees can result in relative (false) shifts of up to 10 cm for objects located 50 meters away from the test vehicle.However, the detection of positions of objects in the surroundings of a vehicle using on-board sensors in the centimeter range is a goal that should be achieved by appropriate training of neural networks for the evaluation of sensor data from such sensors. Disclosure of the invention

[0005] Based on this, the object of the present invention is to at least partially solve the problems described with reference to the prior art. This object is achieved by the invention according to the features of the independent patent claims. Further advantageous embodiments are specified in the dependent claims as well as in the description and, in particular, in the description of the figures. It should be noted that the person skilled in the art can combine the individual features in a technologically expedient manner and thus arrive at further embodiments of the invention.

[0006] In particular, a particularly advantageous method for detecting the spatial orientation of objects in camera data is to be described.

[0007] This describes a method for detecting a spatial orientation of an object in camera data, comprising the following steps a) providing at least one light source which emits light at the object; b) providing a polarizing filter with a predetermined orientation in front of a camera for capturing camera data and detecting the light source in video data recorded by the camera; c) determining a light intensity of the light source in the camera data; and d) Determining a spatial orientation of the object by evaluating the light intensity determined in step c).

[0008] The camera data is in particular video data recorded with the camera. What is novel about the approach described here is that the object is configured to recognize a spatial orientation based on the intensity of light emitted by the object or by light sources arranged on the object. The intensity of the light is perceived by a polarization filter on the camera. The polarization filter allows light to pass through at different intensities, depending on how the camera is aligned to the polarization filter. The alignment of the polarization filter on the camera or the polarization direction of the polarization filter on the camera is preferably predetermined. When determining the spatial orientation of the object by evaluating the light intensity, the alignment or polarization direction of the polarization filter is preferably taken into account.

[0009] It is particularly advantageous if at least one light source generates polarized light.

[0010] A polarizing filter only allows the portion of light that oscillates parallel to the optical axis of the filter to pass through. All other portions are absorbed by the polarizing filter. Therefore, the intensity of the light decreases as it passes through a polarizing filter.

[0011] Light can be described as a wave that oscillates perpendicular to its direction of propagation, i.e., a transverse wave. The polarization, or polarization direction, of the wave describes the direction in which the wave oscillates.

[0012] Polarized light has a specific polarization direction. Polarized light is only allowed to pass through a polarizing filter if the polarization of the light and the polarizing filter match.

[0013] It is also advantageous if the at least one light source is covered by a polarization filter with a predetermined orientation. By means of a polarization filter on

[0014] A polarization filter arranged on the light source can be used to impart polarization to the light from the light source. The alignment of polarization filters on the light source is preferably precisely known and set. Based on the light intensity determined in step c), a relative alignment of the polarization filters on the light source and the polarization filter in front of the camera to one another can be detected. The polarization filter on the light source is attached to the object and the alignment of this polarization filter to the object is known. The polarization filter provided in step b) is attached to the camera and the alignment of this polarization filter to the camera is known. By determining the light intensity in step c), the alignment of the polarization filters to one another can be determined.This allows the determination of the spatial orientation of the object relative to the camera according to step d).

[0015] It is also advantageous if the object is a vehicle whose spatial orientation is recognized in the camera data.

[0016] The vehicle is, in particular, a test vehicle with which tests are carried out to obtain training data for training systems or neural networks for evaluating sensor data for autonomous driving.

[0017] It is further advantageous if in step a) several light sources are provided, each emitting light.

[0018] It is also advantageous if different light sources emit light with different polarization.

[0019] Preferably, each light source has its own polarization filter, with which a predetermined polarization is imposed on the light from each light source.

[0020] It is further advantageous if in step a) three light sources are provided, each emitting light with a different polarization.

[0021] It is also advantageous if at least two light sources emit light at different wavelengths.

[0022] In particular, the light from the at least two light sources has different colors. Particularly preferably, there are three light sources with different colors, for example, red, green, and blue. Preferably, each of the light sources has its own polarization filter. The polarization filters of the three light sources are each arranged at an angle of 120 degrees to each other, for example.

[0023] It is also advantageous if a spectrum of light from light sources on the object is evaluated in the camera data in order to determine the spatial orientation.

[0024] It is also advantageous if the camera data was recorded with a camera on a flight-capable drone in an observation position above the vehicle.

[0025] In other applications, a checkerboard pattern of white and black squares arranged on the roof of the test vehicle has already been used to determine the spatial orientation of the test vehicle relative to the camera or the flight-capable drone. This checkerboard pattern can be recognized in the camera image of a drone hovering above the test vehicle. The orientation of the camera image relative to the vehicle can be determined based on this checkerboard pattern. When detecting spatial orientations based on such a checkerboard pattern, edges of the individual tiles of the checkerboard pattern and / or corners where individual tiles of the checkerboard pattern meet were regularly detected in the past.

[0026] The method described here proposes using a polarization filter in front of a camera on a drone and, in particular, observing several precisely polarized light sources on the roof of the test vehicle with the camera and through a polarization filter. The precise orientation of the drone and the vehicle relative to each other can be determined by measuring the relative intensity of the light from the individual light sources on the roof of the test vehicle, as determined through the polarization filter.

[0027] The described method with the polarizing filter in front of a camera can also be used in addition to a pattern of markings on the roof of the vehicle, which can be seen in the video data recorded with the drone's camera.

[0028] The described method is used in particular to subsequently link video data acquired and stored with a camera on a drone with additional data acquired and stored, for example, with sensors in a test vehicle. The described method can be used to determine the orientation / alignment of the test vehicle in the video data.

[0029] Also to be described here is a device for data processing comprising a processor which is adapted and / or configured to carry out the described method

[0030] A computer program product is also to be described, comprising instructions which, when the computer program product is executed by a computer, cause the computer to carry out what has been described.

[0031] Furthermore, a computer-readable storage medium is to be described, comprising instructions which, when executed by a computer, cause the computer to carry out the described method.

[0032] The invention and the technical context of the invention are explained in more detail below with reference to the figures. The figures show preferred embodiments to which the invention is not limited. It should be noted in particular that the figures, and in particular the proportions depicted in the figures, are only schematic. They show: Fig. 1: a schematic representation of a drone observing a vehicle; Fig. 2: Light sources on an object; Fig. 3: Light intensity curves of light sources on the object in camera data; and Fig. 4: a flowchart of the described procedure.

[0033] The Fig. 1 shows a schematic representation of a drone 7 that accompanies a vehicle and observes it from an observation position 9 above the vehicle. The vehicle here is the object 1 with which the described method is carried out. The drone 7 has a camera 3 with a downward-facing field of view 9 and with which camera data or video data can be generated in which the vehicle (object 1) and other objects 6 in the surroundings of the vehicle are recognizable / visible. Camera data acquired with the camera 3 on the drone 7 are present in a drone coordinate system 15, which is indicated schematically here. Data acquired with environmental sensors (not shown here) on the vehicle (object 1) are present in a vehicle coordinate system 16.In order to use the camera data from camera 3 in conjunction with sensor data from environmental sensors on the vehicle (object 1), a conversion from the vehicle coordinate system 15 to the drone coordinate system 16 and / or vice versa is required. To enable this conversion, the orientation 17 and position 18 of the vehicle (object 1) in the camera data or relative to the camera 3 of the drone 7 must be known. Here, a method is proposed for achieving this using light 10 emitted by the vehicle (object 1).

[0034] The Fig. Figure 2 shows light sources 2 on an object 1 (e.g., on a vehicle) that can be used to detect the position of the object 1. The light sources 2 each emit light 10 in different colors and are each provided with a polarization filter 4 with which a specific polarization is imposed on the light 10. The light sources 2 are each arranged on the object 1. The Fig. 2 also shows a light observation unit 19, which is provided on the drone, for example on a camera of the drone or on a separate light receiving unit (not shown separately here) for detecting the light emitted by the light sources 2 to determine a spatial orientation. The light observation unit 19 also has a polarization filter 4. Depending on how the polarization filters 4 on the light sources 2 and the polarization filter 4 on the light observation unit 19 are aligned to each other, different amounts of light from the individual light sources 2 reach the light observation unit 19.

[0035] The Fig. Figure 3 now shows the light intensities 14 of the light from the individual light sources as seen by the light observation unit 19. The intensity 11 of two light sources is plotted as an intensity curve 13 over the time axis 12. These light intensities 14 are recognizable in the camera data. Based on these light intensities 14, a spatial orientation can be determined and an angular profile 20 can be calculated, which can be Fig. 3 is also plotted on the time axis 12. It can be seen here that the angle described by the angle curve 20 describes the angle between the light sources or the object and the camera on the drone, or in particular the angle between polarization filters in front of the light sources and the polarization filter in front of the camera. Fig. The course of the light intensity curves 13 and the angle course 20 over time shown in Figure 3 arise because the object and the camera rotate relative to each other.

[0036] The Fig. Figure 4 shows a flow chart of the described method, showing the method steps a), b), c) and d), which are carried out to implement the described method.

Claims

[1] Method for detecting a spatial orientation of an object (1) in camera data comprising the following steps a) Providing at least one light source (2) which emits light (10) at the object (1); b) Providing a polarization filter (4) with a predetermined orientation in front of a camera (3) for capturing camera data and capturing the light source (2) in video data recorded with the camera (3); c) Determining the light intensity of the light source (2) in the camera data; and d) Determining a spatial orientation (5) of the object (1) by evaluating the light intensity determined according to step c). [2] Method according to claim 1, wherein the at least one light source (2) produces polarized light (10). [3] Method according to one of the preceding claims, wherein the at least one light source (2) is covered by a polarizing filter (4) with a predetermined orientation. [4] Method according to any of the preceding claims, wherein the object (1) is a vehicle whose spatial orientation (5) is detected in the camera data. [5] Method according to one of the preceding claims, wherein in step a) several light sources (3) are provided, each emitting light (10). [6] Method according to claim 5, wherein different light sources (3) emit light (10) with a different polarization. [7] Method according to claim 6, wherein in step a) three light sources (2) are provided, each emitting light (10) with a different polarization. [8] Method according to claim 6 or 7, wherein at least two light sources (2) emit light (10) in different wavelengths (12). [9] Method according to one of the preceding claims, wherein a spectrum (14) of light (10) from light sources (2) at the object (1) is evaluated in the camera data to determine the spatial orientation (5). [10] Method according to one of the preceding claims, wherein the camera data were recorded with a camera (3) on a flyable drone (7) in an observation position (8) above the vehicle (1). [11] Device for data processing comprising a processor adapted / configured to perform the method according to any one of claims 1 to 10. [12] Computer program product comprising instructions which, when executed by a computer, cause the computer to execute the method according to any one of claims 1 to 10. [13] Computer-readable storage medium comprising instructions which, when executed by a computer, cause it to execute the method according to any one of claims 1 to 10.

Images