Generating input data for a convolutional neural network

Abstract

Method for generating input data for a convolutional neural network using at least one camera (3) and at least one area sensor (5, 6), wherein the camera (3) and the area sensor (5, 6) are arranged on the motor vehicle (1) such that the field of view of the camera (3) and the field of view of the area sensor (5, 6) overlap at least partially, wherein the method comprises the following steps: - Capturing an image frame by a camera (3), wherein the image frame consists of image data for directions relative to the position of the camera (3) and within the solid angle covered by the camera (3), wherein the directions are represented by coordinates in a camera coordinate system; - simultaneous acquisition of depth information by the range sensor (5, 6), wherein the depth information consists of depth data for directions relative to the position of the range sensor (5, 6) and within the solid angle covered by the range sensor (5, 6), wherein the directions are represented by coordinates in a range sensor coordinate system; - Providing a vehicle coordinate system that is related to the camera coordinate system and the area sensor coordinate system by appropriate sets of translations and rotations given by the position of the camera (3) and the position of the area sensor (5, 6) relative to the origin of the vehicle coordinate system; and - Transforming the coordinates in the camera coordinate system and the coordinates in the area sensor coordinate system into coordinates in the vehicle coordinate system based on the sets of translations and rotations, thereby obtaining the input data for the convolutional neural network.

Description

[0001] The present invention relates to a method for generating input data for a convolutional neural network using at least one camera and at least one area sensor.

[0002] One of the most fundamental problems in automotive computer vision is the semantic segmentation of objects in an image. The segmentation approach refers to the problem of assigning each pixel to its corresponding object class. Recently, there has been a surge in the research and design of convolutional neural networks (CNNs), supported by increased computing power in computer architectures and the availability of large annotated datasets.

[0003] CNNs are very successful at classification and categorization tasks, but much of the research focuses on standard photometric RGB images and not on embedded automotive devices. Automotive hardware must have low power consumption and therefore low computing power.

[0004] In machine learning, a convolutional neural network (CNN) is a class of deep, forward-coupled artificial neural networks that have been successfully used to analyze visual images. CNNs employ a variation of multilayered perceptrons designed to require minimal preprocessing. Convolutional networks were inspired by biological processes, where the connectivity pattern between neurons is inspired by the organization of the animal visual cortex. Individual cortical neurons respond to stimuli only in a limited area of ​​the visual field known as the receptive field. The receptive fields of different neurons partially overlap, thus covering the entire visual field.

[0005] CNNs use relatively little preprocessing compared to other image classification algorithms. This means the network learns the filters that were manually developed in traditional algorithms. This independence from prior knowledge and human effort in feature design is a major advantage. CNNs are used in image and video recognition, recommendation systems, and natural language processing, e.g., DE69 422 446 T2, US2017 / 0032222 A1, WO 2017 / 156243 A1, or WO 2017 / 100903 A1.

[0006] The article “Multimodal Deep Learning for Robust RGB-D Object Recognition,” by Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin Riedmiller, and Wolfram Burgard, published at the IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS) in Hamburg, Germany, in 2015, proposes an RGB-D architecture for object recognition. This architecture consists of two separate CNN processing streams—one for each modality—sequentially combined using a late fusion network. The focus is on learning with imperfect sensor data, a typical problem in real-world robotics tasks. For accurate learning, a multi-stage training methodology and two crucial factors for handling depth data with CNNs are introduced. The first is an effective encoding of depth information for CNNs, enabling learning without the need for large depth datasets.The second is a data augmentation scheme for robust learning with depth images by degrading them with realistic noise patterns.

[0007] From US patent 2017 / 0099200 A1, it is known that data is received that characterizes a request for agent-based computation of sensor data. The request includes the required trust and latency for the completion of the agent-based computation. Agents to be requested are determined based on the required trust. Data is then transmitted to request the selected agents to perform an analysis of the sensor data.

[0008] The invention is based on the objective of providing a way to improve the semantic segmentation of objects in an image in automotive computer vision.

[0009] The problem is solved by the subject matter of the independent claims. Preferred embodiments are specified in the dependent claims.

[0010] The invention therefore provides a method for generating input data for a convolutional neural network using at least one camera and at least one area sensor, wherein the camera and the area sensor are arranged on the motor vehicle such that the field of view of the camera and the field of view of the area sensor overlap at least partially, wherein the method comprises the following steps: - Capturing an image frame by a camera, wherein the image frame consists of image data for directions relative to the position of the camera and within the solid angle covered by the camera, wherein the directions are represented by coordinates in a camera coordinate system, - simultaneous acquisition of depth information by the range sensor, wherein the depth information consists of depth data for directions relative to the position of the range sensor and within the solid angle covered by the range sensor, wherein the directions are represented by coordinates in a range sensor coordinate system, - Providing a vehicle coordinate system that is related to the camera coordinate system and the area sensor coordinate system by appropriate sets of translations and rotations given by the position of the camera and the position of the area sensor relative to the origin of the vehicle coordinate system, and - Transforming the coordinates in the camera coordinate system and the coordinates in the area sensor coordinate system into coordinates in the vehicle coordinate system based on the sets of translations and rotations, thereby obtaining the input data for the convolutional neural network.

[0011] A key aspect of the invention is that the input data for the convolutional neural network includes both image data and depth data for common viewing directions relative to the origin of the vehicle coordinate system, with the directions represented by coordinates of the common vehicle coordinate system, which serves as a common reference frame. In other words, the input data for the convolutional neural network consists of image data and depth data for directions represented in the vehicle coordinate system, even though such data was originally represented as data in the coordinate system of the camera or area sensor. Transforming this data into the common vehicle coordinate system allows data from different sensors / cameras to be used in a single dataset that is fed into the convolutional neural network.Preferably, the camera continuously captures image frames, and the area sensor continuously captures depth information. Preferably, as a final step of the method described above, the generated data set, consisting of the depth data and the image data, is fed into the CNN.

[0012] According to a preferred embodiment of the invention, the method further comprises the following steps: - Representing the coordinates in the camera coordinate system using a direction cosine matrix and - Representing the coordinates in the area sensor coordinate system using a direction cosine matrix.

[0013] As is known to those skilled in the art, the direction cosines of a vector are the cosines of the angles between the vector and the three coordinate axes. Equivalently, they are the contributions of each component of the basis to a unit vector in that direction. Direction cosine is an analogous extension of the usual concept of slope to higher dimensions. Therefore, the direction cosine refers to the cosine of the angle between any two vectors. They are used, among other things, to construct direction cosine matrices, which represent a set of orthonormal basis vectors with respect to another set, or to represent a known vector in a different basis.

[0014] Preferably the procedure further comprises the following steps: - Representing the image data by a color value, preferably an RGB value, for each coordinate triplet of the cosine matrix, and - Representing the depth data by a distance value for each coordinate triplet of the cosine matrix.

[0015] In this way, a data set that has a color value (as part of the image frame) and a respective distance value (as part of a depth map) for several directions relative to the origin of the vehicle coordinate system can be entered into the CNN and processed together within it.

[0016] In general, various types of cameras can be used. However, according to a preferred embodiment of the invention, the camera is a fisheye camera with a field of view of at least 180°. Furthermore, a single camera can generally suffice for the method according to the invention. However, according to a preferred embodiment of the invention, several cameras are used to generate the input data for the convolutional neural network. Preferably, these cameras have different fields of view. Even more preferably, these cameras cover the entire surroundings of the motor vehicle.

[0017] Furthermore, several area sensors are preferably used to generate the input data for the convolutional neural network. Generally, these area sensors can be of the same type. However, according to a preferred embodiment of the invention, the area sensors comprise at least two different types, preferably at least one LiDAR sensor and at least one ultrasonic sensor. Preferably, these area sensors have different fields of view. Even more preferably, these area sensors cover the entire surroundings of the vehicle.

[0018] The invention also relates to the use of a method described above in a motor vehicle, a sensor arrangement for a motor vehicle configured to perform such a method, and a non-volatile, computer-readable medium containing instructions stored thereon which, when executed by a processor, cause a sensor arrangement of a motor vehicle to perform such a method.

[0019] They show: Fig. 1 schematically a motor vehicle with a sensor arrangement for detecting an object according to a preferred embodiment of the invention; Fig. 2 schematically the camera coordinate system and the area sensor coordinate system according to the preferred embodiment of the invention; and Fig. 3 schematically the motor vehicle coordinate system according to the preferred embodiment of the invention.

[0020] As in Fig. As shown schematically in Figure 1, a sensor arrangement 2 comprising a camera 3, an evaluation unit 4, an ultrasonic sensor 5, and a LiDAR sensor 6 is provided in a motor vehicle 1 according to a preferred embodiment of the invention. As indicated by the dashed lines, the camera 3, the ultrasonic sensor 5, and the LiDAR sensor 6 each have overlapping fields of view. This enables the capture of scenes with image data or depth data, which can be input into a convolutional neural network provided in the evaluation unit 4 for classifying objects, such as a person 7, in front of the motor vehicle 1.

[0021] By using different types of area sensors 5, 6, i.e., an ultrasonic sensor 5 and a LiDAR sensor 6, it is possible to generate multiple input depth maps with RGB image data for use in a CNN network that can detect and classify objects. The application here is to use vehicle sensors, such as the camera 3, the ultrasonic sensor 5, and the LiDAR sensor 6, to generate depth information around a vehicle and to combine this data with environmental image data. Therefore, such vehicle sensors are preferably arranged on all sides of the vehicle 1 such that the entire surroundings of the vehicle can be monitored. For the sake of clarity, the present preferred embodiment of the invention focuses only on the three vehicle sensors mentioned above as an example.

[0022] An important aspect of the present preferred embodiment of the invention is to encode the area sensors 5, 6, i.e., the ultrasonic sensor 5 and the LiDAR sensor 6, in the same coordinate system as the camera data in order to generate CNN input data that uses RGB and multi-depth maps together. This input data can then be fed into a convolutional neural network for classification.

[0023] As in Fig. As shown schematically in Figure 2, each sensor has its own mechanical coordinate system. Due to the two-dimensionality of the figure, only the x-axis and the z-axis are shown here, i.e., x C and z C for camera 3, x U and z U for the ultrasonic sensor 5 and x L and z L for the LIDAR sensor. Furthermore, as in Fig.As shown schematically in Figure 3, a vehicle coordinate system is defined as a common reference coordinate system for all vehicle sensors 3, 5, 6. The vehicle coordinate system has its origin (0, 0, 0) at the center of the front section of the vehicle 1 at road level. With respect to the respective positions of the vehicle sensors 3, 5, 6 (and any other vehicle sensors that may be arranged on the vehicle), a set of rotations and translations exists to define the relationship between each sensor and the vehicle coordinate system. All sensor data can then be transformed into the vehicle coordinate system as a common reference system and transferred to the CNN.

[0024] In detail, this method is implemented according to the present preferred embodiment of the invention as follows: The camera 3 continuously captures image frames, the image frames consisting of image data for directions relative to the position of the camera 3 and within the solid angle covered by the camera 3, the directions being represented by coordinates in the camera coordinate system described above. Simultaneously, depth information is acquired by the area sensors 5, 6, i.e., the ultrasonic sensor 5 and the LiDAR sensor 6, the depth information consisting of depth data for directions relative to the positions of the area sensors 5, 6 and within the solid angles covered by the area sensors 5, 6, the directions being represented by coordinates in the coordinate systems of the area sensors.

[0025] As described above, a vehicle coordinate system is provided, which is related to the camera coordinate system and the coordinate systems of the area sensors by respective sets of translations and rotations. These sets of translations and rotations are defined by the position of camera 3 and the positions of area sensors 5 and 6 relative to the origin of the vehicle coordinate system. The coordinates in the camera coordinate system and the coordinates in the area sensor coordinate systems are then transformed into coordinates in the vehicle coordinate system based on these sets of translations and rotations. This provides the input data for the convolutional neural network (CNN), which is then fed into the CNN for object classification.

[0026] According to the preferred embodiment of the invention described herein, the coordinates in the camera coordinate system and the coordinates in the area sensor coordinate system are represented by a respective direction cosine matrix. Furthermore, the image data are represented by a color value, i.e., an RGB value, for each coordinate triplet of the cosine matrix, and the depth data are represented by a distance value for each coordinate triplet of the cosine matrix.

[0027] In this way, using image information from camera 3 together with depth information from various area sensors 5, 6, semantic segmentation of objects in an image in automotive computer vision can be significantly improved. Reference symbol list 1 motor vehicle 2 Sensor arrangement 3 cameras 4 evaluation units 5 ultrasonic sensor 6 LiDAR sensors 7 Person

Claims

[1] Method for generating input data for a convolutional neural network using at least one camera (3) and at least one area sensor (5, 6), wherein the camera (3) and the area sensor (5, 6) are arranged on the motor vehicle (1) such that the field of view of the camera (3) and the field of view of the area sensor (5, 6) overlap at least partially, the method comprising the following steps: - Capturing an image frame by a camera (3), wherein the image frame consists of image data for directions relative to the position of the camera (3) and within the solid angle covered by the camera (3), wherein the directions are represented by coordinates in a camera coordinate system; - simultaneous acquisition of depth information by the range sensor (5, 6), wherein the depth information consists of depth data for directions relative to the position of the range sensor (5, 6) and within the solid angle covered by the range sensor (5, 6), wherein the directions are represented by coordinates in a range sensor coordinate system; - Providing a vehicle coordinate system that is related to the camera coordinate system and the area sensor coordinate system by appropriate sets of translations and rotations given by the position of the camera (3) and the position of the area sensor (5, 6) relative to the origin of the vehicle coordinate system; and - Transforming the coordinates in the camera coordinate system and the coordinates in the area sensor coordinate system into coordinates in the vehicle coordinate system based on the sets of translations and rotations, thereby obtaining the input data for the convolutional neural network. [2] The method of claim 1, wherein the method further comprises the following steps: - Representing the coordinates in the camera coordinate system using a direction cosine matrix; and - Representing the coordinates in the area sensor coordinate system using a direction cosine matrix. [3] Method according to claim 1 or 2, wherein the method further comprises the following steps: - Representing the image data by a color value, preferably an RGB value, for each coordinate triplet of the cosine matrix; and - Representing the depth data by a distance value for each coordinate triplet of the cosine matrix. [4] Method according to any of the preceding claims, wherein the camera (3) is a fisheye camera with a field of view of at least 180°. [5] Method according to one of the preceding claims, wherein several cameras (3) are used to generate the input data for the convolutional neural network. [6] Method according to one of the preceding claims, wherein multiple area sensors (5, 6) are used to generate the input data for the convolutional neural network. [7] Method according to claim 6, wherein the area sensors (5, 6) comprise at least two different types of area sensors, preferably at least one LIDAR sensor (6) and at least one ultrasonic sensor (5). [8] Use of the method according to any of the preceding claims in a motor vehicle (1). [9] Sensor arrangement (2) for a motor vehicle (1) configured to perform the method according to any one of claims 1 to 7. [10] Non-volatile computer-readable medium containing instructions stored thereon which, when executed by a processor, cause a sensor arrangement (2) of a motor vehicle (1) to execute the method according to any one of claims 1 to 7.

Images