Method for detecting spatial orientation of object in camera data

Abstract

The invention relates to a method for identifying a spatial orientation of an object (1) in camera data, comprising the following steps: a) providing a regular pattern (2) on the object (1), the regular pattern having a plurality of rows of circular marks (3); b) identifying the circular marking (3) in the camera data; c) determining centers (4) of at least two of the circular marks (3) in the camera data; d) determining the spatial orientation (5) and / or position (6) of the object (1) by finding at least one connecting line (7) between the at least two centers (4) identified in step c).

Description

Technical Field

[0001] Correctly identifying the real world around a vehicle using sensors is a fundamental challenge in developing systems for highly automated and autonomous vehicles. Background Art

[0002] Such systems usually have a neural network, with the aid of which the sensor data acquired by vehicle surroundings sensors are analyzed and evaluated.

[0003] Therefore, developing such systems particularly requires a large amount of training data that can be used to train the neural networks. This data is also known as "labeled data" (English: labeled data, German: gekennzeichnete Daten). This means that certain information about the data is known, and the neural network should be able to independently identify it from comparable data. This data is necessary for training the neural network because it can be used to introduce information to the neural network. To train the neural network used to evaluate vehicle sensor data, so-called "ground truth" data is usually required on a regular basis. This data shows the actual positions of specific objects in the vehicle's surroundings that can be recognized by the vehicle sensors in a ground-based coordinate system that describes the vehicle environment. Vehicle sensors can only recognize objects from the sensor's respective perspective. With the help of this "ground truth" data about these objects, the neural network used to evaluate the sensor data from these sensors can be trained to reliably recognize the vehicle environment, which describes the positions of objects in the coordinate system.

[0004] Acquiring such "ground truth" data is complex. A well-known method is to acquire such data using drones, which perform vehicle tests in order to acquire training data for training the neural network. Such drones are preferably kept hovering above the test vehicle during the test. Preferably, such drones are equipped with a camera that can identify the test vehicle and its environment as well as the objects present in this environment. Since such cameras observe the vehicle and objects from an overhead perspective, the use of drones and their cameras allows for simpler and more accurate identification of "ground truth" data than would be required for evaluating vehicle sensor data, and without the need for complex evaluation systems. Acquiring training data in this way, by positioning the drone at an adjustable distance above the vehicle, is very advantageous—particularly when the training data to be acquired includes the aforementioned "ground truth" data.

[0005] When acquiring training data using drones, it is usually necessary to be able to determine the relative position of the drone and the test vehicle very accurately. This is particularly necessary in order to be able to correctly orient the drone's camera images into the vehicle's coordinate system. In particular, the (unidentified) rotational displacement of the drone is problematic for the camera images taken when acquiring training data using drones. For example, an angular error of 0.1 degrees can already lead to a relative (erroneous) displacement of up to 10 centimeters for objects 50 meters away from the test vehicle. However, the goal is to use the vehicle's own sensors to identify the position of objects around the vehicle within a centimeter range, which is achieved by appropriately training the neural networks used to evaluate such sensor data. Therefore, the training data must achieve similar or even higher accuracy. Summary of the Invention

[0006] Based on this, the present invention is based on the object of at least partially solving the problems described in the prior art. This object is achieved by the method, data processing device, computer program product, and storage medium according to the present invention for detecting the spatial orientation of an object in camera data. In particular, the present invention is intended to describe a method for detecting the spatial orientation of an object in camera data. Further advantageous embodiments are described in the following description, particularly in the illustrations. It should be noted that a person skilled in the art can combine individual features in technically meaningful ways to achieve further embodiments of the invention.

[0007] The invention relates to a method for identifying the spatial orientation of an object in camera data, comprising the following steps:

[0008] a) providing a regular pattern of multiple rows of circular markings on the object;

[0009] b) identifying circular markers in camera data;

[0010] c) finding the centers of at least two of the circular markers in the camera data; and

[0011] d) Determining the spatial orientation and / or position of the object by finding a connecting line between at least two centers identified in step c).

[0012] It is particularly advantageous if the regular pattern is arranged on the roof of a vehicle, whose spatial orientation and / or position is identified in the camera data. Thus, the described object, in particular a vehicle, in particular a motor vehicle, whose orientation and / or position is identified.

[0013] Furthermore, it is advantageous if the camera data is recorded by a camera on a flying drone from a viewing position above the vehicle.

[0014] The method is particularly useful for processing data collected by drones flying above a vehicle for creating training data for training a system (i.e., a neural network contained therein) for evaluating sensor data (in particular, environmental sensors) for monitoring the vehicle's environment. The method is particularly useful for generating "ground truth" data for training such a system to recognize the positions and distances of objects in the vehicle's surroundings.

[0015] The video data acquired by the drone and the objects detected therein, or their position and orientation, can preferably be described in a coordinate system established from the drone, for example centered around the axis corresponding to the drone's camera orientation and other axes in the ground plane where the observed object (motor vehicle) is located. Such a coordinate system is also referred to herein as the drone coordinate system.

[0016] Vehicle sensor data typically exists in a vehicle-related coordinate system, typically oriented in the vehicle's longitudinal and lateral directions, also referred to as the vehicle coordinate system. The spatial orientation of objects in camera data to be identified specifically refers to the spatial orientation of the objects in the camera data. In particular, the orientation of the vehicle coordinate system relative to the drone coordinate system must be determined.

[0017] The method described is based on the idea that the orientation of an object in the camera data can be identified using markers on the object or the test vehicle. These markers can be evaluated in the camera data to automatically determine the object's orientation. Based on this, the positions of other objects in the surroundings of the object, initially identified in the drone's coordinate system in the camera data, can be transformed and / or transferred to the vehicle's coordinate system.

[0018] Other applications have used a checkerboard pattern consisting of white and black squares placed on the roof of a test vehicle. This checkerboard pattern can be detected in camera images from a drone hovering above the test vehicle. This checkerboard pattern can be used to determine the orientation of the camera image relative to the vehicle. Previously, using this checkerboard pattern to identify spatial orientation typically involved identifying the edges of the individual squares in the pattern and / or the corners where the squares intersect.

[0019] This can prove problematic, especially given the camera image resolution. In particular, if the entire checkerboard pattern is described by only a few image pixels, the edges and / or corners of the squares may no longer be reliably discernible. This makes it even more difficult to identify the orientation based on these edges and / or corners.

[0020] Here, it is now proposed to provide a regular pattern of circular markings, with the help of which the spatial orientation can be identified. The circular markings are identified and a center is determined for each. The determination of the circle is much less dependent on the orientation of the camera data pixel grid relative to the regular pattern.

[0021] The center can be reliably detected at low resolutions, especially when the pixels in the camera image are arranged in a rectangular grid, regardless of their orientation.

[0022] The distance between the centers of the circular markers is also greater than the distance between the corners of the squares in the checkerboard pattern. As a result, longer connecting lines can be found, which further improves the recognition accuracy of spatial orientation.

[0023] Furthermore, it is advantageous if the regular pattern comprises an arrangement of circular markings in a hexagonal grid, the arrangement having rows of circular markings which are arranged offset with respect to one another in the individual rows.

[0024] The hexagonal arrangement provides the greatest distance between the centers of the individual blocks in a small space, and is therefore particularly advantageous for the method described here.

[0025] Furthermore, it is advantageous if the regular pattern has three rows comprising three circular markings and two rows comprising two circular markings.

[0026] Furthermore, it is advantageous if the pattern is implemented with five rows of circular markings.

[0027] Furthermore, it is advantageous if the pattern has two circular markings in three rows and three circular markings in two rows.

[0028] Furthermore, it is advantageous if the pattern has the following arrangement of rows arranged side by side:

[0029] the first three rows with circular markings (Dreierreihe);

[0030] The first two rows of circular markings are arranged offset with respect to the first three rows of circular markings;

[0031] The second three rows have circular markings oriented corresponding to the circular markings of the first three rows;

[0032] the second two rows, whose circular markings are oriented correspondingly to the circular markings of the first three rows;

[0033] The third two rows have circular markings oriented correspondingly to the circular markings of the three rows.

[0034] Furthermore, it is advantageous if step c) is performed using a "blob detection" algorithm.

[0035] To identify the center of a circle, a so-called "blob detection" algorithm can be used. There are highly accurate algorithms for finding the center of a circle, which are collectively referred to as "blob detection."

[0036] Furthermore, it is advantageous if step d) is performed using a perspective N-point algorithm.

[0037] Once the centers of the multiple circles are determined, the so-called perspective-n-Point algorithm can be used to determine the position and direction / spatial orientation between the drone camera image and the test vehicle.

[0038] It has been found that even at very low resolutions (e.g., 50×30 pixels for displaying a marker pattern), the standard deviation of the identified orientations is very small. In particular, the deviations due to unfavorable orientations of the camera data pixel grid are much smaller than for other types of markers (e.g., markers in the aforementioned checkerboard grid).

[0039] A data processing device is also described herein, comprising a processor adapted and / or configured to carry out the described method.

[0040] A computer program product shall also be described, which comprises instructions which, when executed by a computer, cause the computer to carry out the described method.

[0041] Furthermore, a computer-readable storage medium shall be described, which comprises instructions that, when executed by a computer, cause the computer to implement the described method. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention and its technical background will be further explained below by means of diagrams. The diagrams show preferred embodiments, but the present invention is not limited thereto. It should be noted that the diagrams, especially the proportional relationships shown in the diagrams, are only schematic.

[0043] Figure 1 : Schematic diagram of a drone observing a vehicle;

[0044] Figure 2a : A plate with a rectangular marking pattern for identifying spatial orientation;

[0045] Figure 2b :Based on Figure 2a The principle of spatial orientation of pattern recognition;

[0046] Figure 3 : A plate with a circular marking pattern for identifying spatial orientation;

[0047] Figure 4 :Based on Figure 3 The principle of spatial orientation of pattern recognition;

[0048] Figure 5 : Schematic diagram of the described method. DETAILED DESCRIPTION

[0049] Figure 1 A schematic diagram shows a drone 10 accompanying a vehicle 1 and observing it from an observation position 11 above the vehicle 1. The drone has a camera 9 with a downward-facing viewing angle 18, which can generate camera data in which the vehicle 1 and other objects 14 surrounding the vehicle 1 are identifiable / visible. The camera data acquired by the camera 9 on the drone 10 is in a drone coordinate system 16, which is schematically represented here on the drone 10, but could also be projected onto the ground on which the vehicle 1 is located. The camera data is used (not shown separately here) to calibrate the sensors on the vehicle 1 or to improve the algorithms used to evaluate such sensor data. The data acquired by the sensors on the vehicle 1 is in a vehicle coordinate system 15. To utilize the camera data from the camera 9, the vehicle coordinate system 15 must be converted to the drone coordinate system 16 and / or vice versa. To achieve this conversion, the orientation 5 and position 6 of the vehicle 1 in the camera data 9 or relative to the camera 9 of the drone 10 must be known. It is recommended that a plate 17 with a pattern (not shown here) be provided on the roof 8 of the vehicle 1. Such a pattern on the plate 17 can be recognized and evaluated in the camera data in order to determine in particular the orientation 5 of the vehicle 1 in the camera data and possibly also its position 6 .

[0050] Figure 2a A first embodiment variant of a plate 17 is shown, which carries a regular pattern 2, but which is not part of the present invention. The regular pattern 2 here consists of rectangular markings. Figure 2b It is shown how this regular pattern 2 can be used to evaluate and identify the orientation. The edges or corners of the rectangular markings can be found and the connecting lines 7 of these edges or corners can be found to determine the orientation. Figure 2b In Figure 2, we can see how a regular pattern 2 is provided in the camera data. The camera data represents the camera image in the form of pixels in a pixel grid. This can lead to distortions in the edges of structures shown in the camera data. This distortion depends, in particular, on the orientation of the camera data pixel grid. In particular, when recognizing rectangular marker patterns, the (unknown) orientation of the pixel grid relative to the orientation of the rectangular marker can lead to undesirable directional dependencies.

[0051] Figure 3 The embodiment variant of the plate 17 shown can be used for the invention described here. The markings here are circular markings 3 arranged in a regular pattern 2. The regular pattern 2 is a hexagonal pattern consisting of rows 12, 13 of circular markings 3 arranged in an offset manner. Two rows 12 of circular markings and three rows 13 of circular markings are provided. Figure 4Figure 2 shows how this regular pattern 2 consisting of circular marks 3 appears in the camera data. Here you can also see the pixelation of the regular pattern 2 in the camera data. Due to the circular shape of the circular marks 3, Figure 2b Compared to the case shown, the influence of the orientation dependence of the pixel grid is significantly reduced. Using a specially tested and validated algorithm, the centers 4 of the circular markings 3 can be determined. Connecting lines 7 can be found between the centers 4 to determine the spatial orientation.

[0052] Figure 5 The flow chart of the described method, which can be executed in a data processing device and a camera data evaluation device, is schematically shown. The method steps a), b), c) and d) can be seen as being executed in sequence.

Claims

1. A method for identifying the spatial orientation of an object (1) in camera data, comprising the following steps: a) providing a regular pattern (2) on an object (1), said regular pattern having a plurality of rows of circular markings (3); b) identifying the circular marker (3) in the camera data; c) determining the centers (4) of at least two circular marks (3) in the camera data; d) determining the spatial orientation (5) and / or position (6) of the object (1) by finding at least one connecting line (7) between at least two centers (4) identified in step c).

2. The method according to claim 1, wherein The regular pattern (2) is arranged on the roof (8) of a vehicle (1), the spatial orientation (5) and / or position (6) of the vehicle being identified in camera data.

3. The method according to claim 2, wherein: The camera data is recorded by means of a camera (9) on a flyable drone (10) in an observation position (11) above the vehicle (1).

4. The method according to any one of the preceding claims, wherein The regular pattern (2) comprises an arrangement of circular marks (3) in a hexagonal grid, the arrangement having rows (12, 13) of circular marks (3), wherein the circular marks in the individual rows (12, 13) are arranged offset with respect to one another.

5. The method according to any one of the preceding claims, wherein The regular pattern (2) has three rows (13) of three circular marks (3) and two rows (12) of two circular marks (3).

6. A method according to any one of the preceding claims, wherein: The pattern is implemented with five rows (12, 13) of circular markings (3).

7. The method according to claim 6, wherein: The pattern has two rows of three (13) circular marks and three rows of two (12) circular marks.

8. The method according to claim 7, wherein: The pattern has the following arrangement of rows (12, 13) arranged side by side: The first three rows (13) have circular marks (3); a first two rows (12) of circular markings (3) arranged offset from the circular markings (3) of the first three rows (13); a second three rows (13), the circular markings (3) of which are oriented correspondingly to the circular markings (3) of the first three rows (13); a second two rows (12), whose circular markings (3) are oriented correspondingly to the circular markings (3) of the first three rows (13); The circular markings (3) of the third two rows (12) are oriented correspondingly to the circular markings (3) of the three rows (13).

9. The method according to any one of the preceding claims, wherein: Step c) is performed with the aid of a "speckle detection" algorithm.

10. The method according to any one of the preceding claims, wherein Step d) is performed using the perspective N-point algorithm.

11. A device for data processing, comprising a processor adapted / configured in such a way that it carries out the method according to any one of claims 1 to 10.

12. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 10.

13. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 10.

Images