Method for detecting the spatial orientation of an object in camera data

Abstract

Method for detecting a spatial orientation of an object (1) in camera data, comprising the following steps: a) providing a regular pattern (2) comprising a plurality of rows of circular markings (3) on the object (1); b) detecting the circular markings (3) in the camera data; c) determining the center (4) of at least two of the circular markings (3) in the camera data; and d) Determining a spatial orientation (5) and / or position (6) of the object (1) by finding at least one connecting line (7) of at least two centers (4) detected in step c).

Description

State of the art

[0001] Correctly recognizing the real world in a vehicle's environment using sensors is a fundamental challenge in the development of systems for highly automated and autonomous driving of motor vehicles.

[0002] Such systems typically feature neural networks that analyze and evaluate sensor data obtained from the vehicle's environmental sensors.

[0003] The development of such systems therefore requires, in particular, a very large amount of training data that can be used to train neural networks. Such data is also referred to as "labeled data." This means that certain information is known about the data, which a neural network is supposed to recognize independently from comparable data. Such data is required to train neural networks because such data can be used to incorporate information into neural networks. Regular "ground truth" data is required to train neural networks to evaluate sensor data from a vehicle. Such data shows the actual location of certain objects in the vehicle's surroundings, which can be detected by the vehicle's sensors, in a ground-referenced coordinate system used to describe the vehicle's surroundings.Vehicle sensors can only detect objects from the sensor's current viewing perspective. Using ground truth data regarding such objects, neural networks can be trained to evaluate sensor data from such sensors to reliably detect the position of objects in the ground-referenced coordinate system used to describe the vehicle's surroundings.

[0004] Obtaining such "ground truth" data is complex. A well-known approach is to obtain such data with the help of a drone that accompanies a vehicle conducting test drives to obtain data for training neural networks. Such a drone preferably hovers above the test vehicle at all times during the test drive. Such a drone is preferably equipped with a camera that detects the test vehicle and its surroundings, as well as the objects in this surroundings. Due to the top-down perspective with which such a camera views the vehicle and the objects, "ground truth" data can be detected much more easily and accurately with such a drone and its camera, using less complex analysis systems than are required for evaluating data from sensors on the vehicle.By positioning a drone at an adjustable distance above the vehicle, obtaining training data in this way is very advantageous - especially if the training data to be obtained includes the described "ground truth" data.

[0005] When using drones to obtain training data, it is often necessary to be able to determine the relative position between the drone and the test vehicle very precisely. This is particularly necessary to correctly align the drone's camera image within the test vehicle's coordinate system. In particular, an (undetected) rotational shift of the drone relative to the vehicle is often problematic when using camera images captured with the drone to obtain training data. For example, an angular error of just 0.1 degrees can lead to relative (false) shifts of up to 10 cm for objects located 50 meters away from the test vehicle.However, detecting the positions of objects in a vehicle's surroundings with on-board sensors at centimeter levels is a goal that should be achieved through appropriate training of neural networks to evaluate sensor data from such sensors. Therefore, training data must achieve comparable or even higher accuracy. Disclosure of the invention

[0006] Based on this, it is the object of the present invention to at least partially solve the problems described with reference to the prior art. This object is achieved by the invention according to the features of the independent patent claim. In particular, a method for detecting the spatial orientation of objects in camera data is to be described. Further advantageous embodiments are specified in the dependent patent claims as well as in the description and, in particular, in the description of the figures. It should be noted that the person skilled in the art can combine the individual features in a technologically expedient manner and thus arrive at further embodiments of the invention.

[0007] The invention relates to a method for detecting a spatial orientation of an object in camera data, comprising the following steps: a) providing a regular pattern comprising multiple rows of circular markings on the object; b) Detecting the circular markers in the camera data; c) determining the center of at least two of the circular markers in the camera data; and d) Determining a spatial orientation and / or position of the object by finding at least one connecting line of at least two centers detected in step c).

[0008] It is particularly advantageous if the regular pattern is arranged on the roof of a vehicle whose spatial orientation and / or position is detected in the camera data. The described object, whose orientation and / or position is detected, is therefore in particular a vehicle, and most preferably a motor vehicle.

[0009] It is also advantageous if the camera data was recorded with a camera on a flight-capable drone in an observation position above the vehicle.

[0010] The method is used in particular in the context of processing data from a drone flying above the motor vehicle to create training data for training systems (or the neural networks contained therein) for evaluating sensor data from sensors (in particular environmental sensors) for monitoring the surroundings of a motor vehicle. The method is used in particular for generating "ground truth" data designed to train such systems for detecting the positions and distances of objects in the surroundings of a motor vehicle.

[0011] Video data acquired with a drone and the objects detected therein, as well as their position and orientation, are best described in a coordinate system defined by the drone itself, for example, with a center on an axis aligned with the camera on the drone and additional axes in a ground plane on which the observed object (the vehicle) is also located. Such a coordinate system is also referred to here as a drone coordinate system.

[0012] Sensor data from sensors on the motor vehicle is usually available in coordinate systems related to the vehicle, which are usually aligned according to the longitudinal and transverse directions of the vehicle and can also be referred to as vehicle coordinate systems. The spatial orientation of an object to be recognized in the camera data refers, in particular, to the spatial orientation of the object in the camera data. In particular, the orientation of a vehicle coordinate system relative to the drone coordinate system should be recognized.

[0013] The approach of the described method is that the orientation of the object can be identified in the camera data based on the markers on the object or the test vehicle. These markers can be evaluated in the camera data to automatically determine the object's orientation. Based on this, the position of other objects in the object's vicinity, which are initially identified in the drone coordinate system in the camera data, can then be converted and / or transferred to the vehicle coordinate system.

[0014] In other applications, a checkerboard pattern of white and black squares arranged on the roof of a test vehicle has already been used.

[0015] This checkerboard pattern can be detected in the camera image of a drone hovering above the test vehicle. The orientation of the camera image relative to the vehicle can be determined based on this checkerboard pattern. In the past, when detecting spatial orientations based on such a checkerboard pattern, edges of the individual tiles of the checkerboard pattern and / or corners where individual tiles of the checkerboard pattern meet were regularly detected.

[0016] It has been found that this can be problematic, especially given the resolution of the camera image. Especially when the entire checkerboard pattern is described by a small number of image pixels, edges and / or corners of tiles can no longer be reliably detected. It then becomes more difficult to identify alignments based on edges and / or corners of tiles.

[0017] Here, we propose to provide a regular pattern of circular markers that can be used to detect the spatial orientation. The circular markers are detected, and a center is determined for each circular marker. Determining the circle is much less dependent on the alignment of a pixel grid of the camera data to the alignment of the regular pattern.

[0018] At low resolutions, center detection is particularly reliable. This is especially successful regardless of orientation when the pixels in the camera image are arranged in a rectangular grid.

[0019] In particular, the centers of circular markers are further apart than the corners of tiles in a checkerboard pattern. Therefore, longer connecting lines can be found, further improving the detection of spatial orientation.

[0020] Furthermore, it is advantageous if the regular pattern comprises an arrangement of the circular markings in a hexagonal grid with rows of circular markings that are offset from one another in the individual rows.

[0021] The hexagonal arrangement offers the maximum distance between the centers of the individual tiles in a small space, so that such an arrangement is particularly advantageous for the method described here.

[0022] Furthermore, it is advantageous if the regular pattern has rows of three comprising three circular markings and rows of two comprising two circular markings.

[0023] It is also advantageous if the pattern is designed with five rows of circular markings.

[0024] It is also advantageous if the pattern has two rows of three and three rows of two circular markings.

[0025] It is also advantageous if the pattern has the following arrangement of rows arranged next to each other: - first row of three with circular markings; - first row of two with circular markings offset from the circular markings of the first row of three; - second row of three, the circular markings of which are aligned with the circular markings of the first row of three; - second row of two, whose circular markings are aligned with the circular markings of the first row of three; and third row of two, whose circular markings are aligned with the circular markings of the rows of three.

[0026] It is further advantageous if step c) is carried out with a “blob detector” algorithm.

[0027] So-called "blob detector" algorithms can be used to detect the center of a circle. Highly accurate algorithms for finding the center of a circle exist, which are described under the term "blob detector."

[0028] Furthermore, it is advantageous if step d) is carried out with a perspective N-point algorithm.

[0029] Once the centers of a plurality of circles have been determined, a so-called perspective-n-point algorithm can be used to determine the position and orientation / spatial alignment between the drone's camera image and the test vehicle.

[0030] It was found that even at very low resolutions, such as 50 x 30 pixels, showing the pattern of markers, the standard deviation of the detected orientations is very small. In particular, the deviation due to an unfavorable alignment of the pixel grid of the camera data is much smaller than for other types of markers, such as the described markers in a checkerboard grid.

[0031] Also to be described here is a device for data processing comprising a processor which is adapted and / or configured to carry out the described method

[0032] A computer program product is also to be described, comprising instructions which, when the computer program product is executed by a computer, cause the computer to carry out the described method.

[0033] Furthermore, a computer-readable storage medium is to be described, comprising instructions which, when executed by a computer, cause the computer to carry out the described method.

[0034] The invention and the technical context of the invention are explained in more detail below with reference to the figures. The figures show preferred embodiments to which the invention is not limited. It should be noted in particular that the figures, and in particular the proportions depicted in the figures, are only schematic. They show: Fig. 1: a schematic representation of a drone observing a vehicle; Fig. 2a: a board with a pattern of rectangular markings for detecting spatial orientation; Fig. 2b: a principle for detecting a spatial orientation based on the pattern from Fig. 2a; Fig. 3: a board with a pattern of circular markings for detecting spatial orientation; Fig. 4: a principle for detecting a spatial orientation based on the pattern from Fig. 3; and Fig. 5: schematically shows a sequence of the described method.

[0035] The Fig. 1 shows a schematic representation of a drone 10 that accompanies a vehicle 1 and observes from an observation position 11 above the vehicle 1. The drone has a camera 9 with a downward-facing field of view 18, which can be used to generate camera data in which the vehicle 1 and other objects 14 in the vicinity of the vehicle 1 are recognizable / visible. Camera data acquired with the camera 9 on the drone 10 are present in a drone coordinate system 16, which is schematically indicated here on the drone 10, but which can also be projected onto the ground on which the vehicle 1 is located. The camera data is intended to be used to calibrate sensors (not shown separately here) on the vehicle 1 or to improve algorithms for evaluating the data from such sensors. Data acquired with sensors on the vehicle 1 is present in a vehicle coordinate system 15.In order to use the camera data from camera 9, a conversion from the vehicle coordinate system 15 to the drone coordinate system 16 and / or vice versa is required. To enable this conversion, the orientation 5 and position 6 of vehicle 1 in the camera data 9 or relative to the camera 9 of the drone 10 must be known. Here, it is proposed to provide a panel 17 with a pattern (not shown here) on the roof 8 of vehicle 1. This pattern on panel 17 can be recognized and evaluated in the camera data, in particular to determine the orientation 5 and, if applicable, also the position 6 of vehicle 1 in the camera data.

[0036] The Fig. Figure 2a shows a first embodiment of a panel 17 with a regular pattern 2, which is not the subject of the invention here. The regular pattern 2 is formed here by rectangular markings. Fig. Figure 2b shows how this regular pattern 2 can be evaluated to detect an alignment. Edges or corners of the rectangular markings can be found, and connecting lines 7 between these edges or corners can be found to determine an alignment. Fig. Figure 2b shows how the regular pattern 2 is available in camera data. Camera data represents the camera image in the form of pixels in a pixel grid. This may distort edges of structures represented in the camera data. This distortion is particularly dependent on the orientation of the pixel grid of the camera data. In particular, when detecting a pattern of rectangular markings, an (unknown) orientation of the pixel grid relative to the orientation of the rectangular markings can create undesirable directional dependencies.

[0037] The Fig. The embodiment of the plate 17 shown in Figure 3 can be used for the invention described here. The markings here are circular markings 3 arranged in a regular pattern 2. The regular pattern 2 is a hexagonal pattern of offset rows 12, 13 of circular markings 3. Rows of two 12 and rows of three 13 of circular markings 3 are provided. Fig. Figure 4 shows how such a regular pattern 2 of circular markings 3 is available in camera data. The pixelation of the regular pattern 2 in the camera data can also be seen here. Due to the circularity of the circular markings 3, a direction-dependent effect of the pixel grid is possible compared to the Fig. The situation shown in Figure 2b is significantly reduced. Using special, tried-and-tested algorithms, it is possible to determine the centers 4 of circular markings 3. Connecting lines 7 can be found between centers 4 to determine spatial alignment.

[0038] Fig. Figure 5 shows a schematic flow diagram of the described method, as it can be implemented in a device for data processing and evaluating camera data. It shows the process steps a), b), c), and d), which are executed sequentially.

Claims

[1] Method for detecting a spatial orientation of an object (1) in camera data comprising the following steps: a) Providing a regular pattern (2) which has several rows of circular markings (3) on the object (1); b) Detection of the circular markings (3) in the camera data; c) Determining the center (4) of at least two of the circular markers (3) in the camera data; and d) Determining a spatial orientation (5) and / or position (6) of the object (1) by finding at least one connecting line (7) between at least two centers (4) identified in step c). [2] Method according to claim 1, wherein the regular pattern (2) is arranged on the roof (8) of a vehicle (1) whose spatial orientation (5) and / or position (6) is detected in the camera data. [3] Method according to claim 2, wherein the camera data were recorded with a camera (9) on a flyable drone (10) in an observation position (11) above the vehicle (1). [4] Method according to one of the preceding claims, wherein the regular pattern (2) comprises an arrangement of the circular markings (3) in a hexagonal grid with rows (12, 13) of circular markings (3) which are arranged offset from each other in the individual rows (12, 13). [5] Method according to one of the preceding claims, wherein the regular pattern (2) comprises rows of three (13) comprising three circular markings (3) and rows of two (12) comprising two circular markings (3). [6] Method according to one of the preceding claims, wherein the pattern is designed with five rows (12,13) ​​of circular markings (3). [7] Method according to claim 6, wherein the pattern has two rows of three (13) and three rows of two (12) circular markings. [8] Method according to claim 7, wherein the pattern has the following arrangement of adjacent rows (12,13): - first row of three (13) with circular markings (3); - first row of two (12) with circular markings (3) which are offset from the circular markings (3) of the first row of three (13); - second row of three (13), whose circular markings (3) are aligned according to the circular markings (3) of the first row of three (13); - second row of two (12) whose circular markings (3) are aligned according to the circular markings (3) of the first row of three (13); and third row of two (12) whose circular markings (3) are aligned according to the circular markings (3) of the rows of three (13). [9] Method according to any of the preceding claims, wherein step c) is performed using a “blob-detector” algorithm. [10] Method according to any of the preceding claims, wherein step d) is performed using a perspective N-point algorithm. [11] Device for data processing comprising a processor adapted / configured to perform the method according to any one of claims 1 to 10. [12] Computer program product comprising instructions which, when executed by a computer, cause the computer to execute the method according to any one of claims 1 to 10. [13] Computer-readable storage medium comprising instructions which, when executed by a computer, cause it to execute the method according to any one of claims 1 to 10.

Images