CREATING A TOP VIEW OF A MOTOR VEHICLE

Description

[0001] The invention relates to a device for generating a top view of a motor vehicle and a method for training this device.

[0002] A top-down view of a vehicle can be generated, for example, using one or more cameras mounted on the underside of the exterior mirrors. Such systems are already available on the market, for example under the name BMW Top View.

[0003] A disadvantage of the known method for generating a top-down view of a motor vehicle is, for example, that the image area of ​​the top view is predetermined and limited by the position of the camera(s). Furthermore, the appropriate cameras are absolutely necessary for this method.

[0004] In the conference paper "Generative Adversarial Frontal View to Bird View Synthesis" by Xinge Zhu et al. (2018 IEEE International Conference on 3D Vision, pages 454-463, DOI: 10.1109 / 3DV.2018.00059), a so-called BridgeGAN for generating a synthetic bird's-eye view is presented. Using a multi-GAN (GAN: Generative Adversarial Network) model, a frontal view is transformed into a bird's-eye view. Intermediate perspectives are introduced to simplify the transformation challenges. Furthermore, consistency constraints are presented, which, according to the authors, ensure one-to-one cross-correspondences and consistent representations of features across different views.

[0005] The object of the invention is to provide a device for generating a top view of a motor vehicle and a method for training this device, which eliminate the aforementioned disadvantages.

[0006] The problem is solved by the features of the independent claims. Advantageous embodiments are described in the dependent claims.

[0007] A first aspect of the invention relates to a device for generating a first top view of a motor vehicle from at least one image from at least one camera, the optical axis of which is essentially parallel to a plane spanned from the longitudinal direction and the transverse direction of the vehicle, depending on a first visual information.

[0008] The first visual information is, in particular, information that describes the orientation of the optical axis of a camera, or view, possibly virtual, relative to a reference system. For example, the first visual information is information that describes the orientation of the optical axis of the top view to be generated relative to a plane spanned by the vehicle's longitudinal and transverse directions.

[0009] The top view is a view from above, whose optical axis is essentially orthogonal to a plane spanned by the vehicle's longitudinal direction and the vehicle's transverse direction.

[0010] Thus, the first visual information is, for example, information that characterizes a view whose optical axis is essentially orthogonal to a plane spanned by the vehicle's longitudinal direction and the vehicle's transverse direction, where the angle between the optical axis and the plane is between 80° and 100°, for example, if it is 90°.

[0011] The optical axis of the camera is, in particular, essentially parallel to the plane spanned by the longitudinal direction of the vehicle and the transverse direction of the vehicle if the angle between the optical axis and the plane is <10°, for example if it is 0°.

[0012] The device is set up to receive at least one image from the camera.

[0013] Furthermore, the device is designed to extract a second piece of visual information from the image using a coding unit. This second piece of visual information specifically describes the orientation of the camera's optical axis relative to the plane defined by the vehicle's longitudinal and transverse directions.

[0014] Furthermore, the device is set up to extract view-independent content information from the image using the coding unit, and to generate the first top view based on the first view information and the view-independent content information using a first generator unit.

[0015] The core of the invention is to present the view-independent content information from the perspective of the first view information.

[0016] The device additionally comprises a first discriminator unit configured to receive the top view generated by the first generator unit, a second generator unit configured to generate a second top view from a third view information and noise information, and a second discriminator unit configured to receive the top view generated by the second generator unit.

[0017] The first discriminator unit and the second discriminator unit are each specifically configured to decide whether the first top view or the second top view was generated by the first generator unit or the second generator unit, respectively, or whether the first top view or the second top view was generated by a camera whose optical axis is essentially orthogonal to a plane spanned by the longitudinal direction and the transverse direction of the vehicle.

[0018] The optical axis of this camera is, in particular, essentially orthogonal to the plane spanned by the longitudinal direction of the vehicle and the transverse direction of the vehicle if the angle between the optical axis and the plane is between 80° and 100°, for example if it is 90°.

[0019] The core of the invention lies in the fact that the two discriminator units are configured to distinguish a "real" camera image from an artificially generated one. For the training methods described later, this results in the effect that the top-view images generated by the two generator units are of high quality and, ideally, indistinguishable from real camera images.

[0020] The first generator unit, the second generator unit, the first discriminator unit, and the second discriminator unit each comprise a neural network.

[0021] A neural network (less commonly also called a neural network) is a collection of individual information processing units (neurons) arranged in layers within a network architecture. In the context of artificial intelligence, these are referred to as artificial neural networks.

[0022] The neurons (also called nodes) of an artificial neural network are arranged layer by layer and are generally interconnected in a fixed hierarchy. The neurons are most often connected between two layers (inter-neuron-layer connection), but less frequently also within a single layer (intra-neuron-layer connection).

[0023] Starting with the input layer, information flows through one or more intermediate layers (hidden layers) to the output layer. The output of one neuron becomes the input of the next.

[0024] The input layer is the starting point of the information flow in an artificial neural network.

[0025] Input signals are typically received by the neurons at the beginning of this layer and, after weighting, passed on to the neurons of the first intermediate layer. In this process, a neuron in the input layer passes the respective information to the neurons of the first intermediate layer.

[0026] Between the input and output layers, every artificial neural network contains at least one intermediate layer (also called an activity layer or hidden layer). The more intermediate layers there are, the "deeper" the neural network; in English, this is also referred to as deep learning.

[0027] Theoretically, the number of possible hidden layers in an artificial neural network is unlimited. In practice, however, each additional hidden layer also increases the computing power required to operate the network.

[0028] The output layer lies behind the intermediate layers and forms the final layer in an artificial neural network. Neurons in the output layer are each connected to the neurons of the last intermediate layer. The output layer represents the endpoint of the information flow in an artificial neural network and contains the result of the network's information processing.

[0029] Weights describe the intensity of information flow along a connection in a neural network. Each neuron assigns a weight to the incoming information and then passes it on to the neurons of the next layer, weighted and, if necessary, after adding a value for neuron-specific bias. Typically, the weights and biases are initialized at the start of training. The result of the weighting and bias is often processed by an activation function (e.g., sigmoid or tanh) before being passed on to the neurons of the next layer.

[0030] The weights and distortions are adjusted during the training process so that the final result meets the requirements as closely as possible.

[0031] The neural network of the first generator unit and the neural network of the second generator unit each have at least partially the same edge weights, and the neural network of the first discriminator unit and the neural network of the second discriminator unit each have at least partially the same edge weights.

[0032] In particular, the neural network of the first generator unit and the neural network of the second generator unit, or the neural network of the first discriminator unit and the neural network of the second discriminator unit, have at least partially equal edge weights at corresponding locations in the neural network, i.e., locations that, for example, have the same structure and / or whose nodes are semantically equivalent.

[0033] The edge weights of neural networks are at least partially equal, in particular if at least 90% of the edge weights are equal, or, more preferably, if 100% of the edge weights are equal.

[0034] A second aspect of the invention relates to a method for training the device according to one of claims 1, 2 or 3.

[0035] One step in the process is providing a training data pair consisting of a training image and an initial training top view. The training data pair is selected such that the initial training top view is derived from the training image, taking into account the relevant viewing information.

[0036] A further step in the process is to extract the second view information from the training image using the coding unit, and to extract the view-independent content information from the training image using the coding unit.

[0037] A further step in the process is the generation of the first top view depending on the first view information and the view-independent content information using the first generator unit.

[0038] A further step in the process is to use the first discriminator unit to decide whether the first top view was generated by the first generator unit, or whether the first top view was generated by a camera whose optical axis is essentially orthogonal to a plane spanned by the vehicle's longitudinal direction and the vehicle's transverse direction.

[0039] A further step in the process is to compare the first top view with the first training top view, and to adjust the coding unit and / or the first generator unit and / or the first discriminator unit depending on the comparison of the first top view with the first training top view.

[0040] In particular, the coding unit and the first generator unit are adapted such that the probability increases that the first discriminator unit classifies the first top view as an image from a camera whose optical axis is essentially orthogonal to a plane spanned by the vehicle's longitudinal direction and the vehicle's transverse direction.

[0041] Alternatively or additionally, the discriminator unit is adjusted in such a way that the probability increases that the first discriminator unit recognizes the first top view as an image generated by the first generator unit.

[0042] The core of the invention lies in the fact that the first discriminator unit, on the one hand, and the first generator unit and the coding unit, on the other, are trained simultaneously with regard to achieving competing goals. This leads to an increase in the quality of the first top-down view generated by the first generator unit, making it increasingly difficult to distinguish from an actual camera image.

[0043] In an advantageous embodiment, the method comprises the step of adjusting at least two of the following units depending on the comparison of the first top view with the first training top view: coding unit, first generator unit, first discriminator unit.

[0044] Adjusting the coding unit, the first generator unit and / or the first discriminator unit means, in the case that these units each comprise a neural network, an adjustment of the edge weights of the neural networks.

[0045] The neural network "learns" by adjusting the edge weights.

[0046] Learning can work as follows: After the network structure is built, each neuron receives a random initial weight. Input data is then fed into the network, and each neuron weights the input signals with its assigned weight, passing the result on to the neurons of the next layer. The output layer then calculates the overall result – and this will usually have little to do with the known actual result, since all neurons have a random initial weight. However, one can calculate the magnitude of the error and the contribution of each neuron to it, and then adjust each neuron's weight slightly in the direction that minimizes the error. The next iteration then takes place, with another measurement of the error and adjustment of the weights, and so on. In this way, a neural network increasingly "learns" to infer the known output data from the input data.

[0047] In detail, input data is fed into one side of the neural network. Each input signal is distributed to each individual neuron of the first layer. Each neuron then weights the incoming signal with an input-specific weight (which was randomly assigned at the beginning), adds a so-called neuron-specific bias term, and sums all the weighted input data to produce the output of that single neuron.

[0048] Often, the output is further processed by a non-linear activation function to, for example, enforce a specific output value range. The output of each neuron is then passed as input to all neurons in the next layer. This process continues until the output layer is reached, which provides the result of all calculations.

[0049] Up to this point, the artificial neural network has not yet learned anything. Since all weights are randomly selected within a predefined range of values ​​during the initialization of a neural network, the result will be a purely random value. The most widely used method for training a network is currently supervised learning, which refers to learning by example.

[0050] In this case, an example means a combination of real input-output data pairs. These examples are used in the training of artificial neural networks to optimally adjust all weights and bias terms so that, at the end of the training, the network can calculate the correct result for all input data, including previously unseen new input data.

[0051] Here, the untrained neural network calculates a result for each set of input data (also called features). This result is then compared with the known results of the example dataset (also called targets or labels), and the magnitude of the deviation or error is calculated. To represent both positive and negative deviations simultaneously, the mean squared deviation or another error function is used, for example.

[0052] Then the actual "learning" begins: The measured error is fed back into the artificial neural network, and each weight and bias term is adjusted a small amount in the direction that reduces the error. The size of this adjustment is calculated firstly via the contribution that a particular neuron weight had to the result, and secondly via the so-called learning rate, which is one of the most important parameters of neural networks.

[0053] Common learning rates are, for example, 0.001 or 0.01, meaning that only one hundredth to one thousandth of the calculated error is corrected per iteration. If the adjustment per iteration is too large, the minimum of the error curve may be missed, and the deviations will become increasingly larger instead of smaller. Therefore, the learning rate is sometimes progressively reduced during training to better determine the minimum of the error function.

[0054] Another potential problem is error functions with local minima, where the neural network gets "stuck" and therefore cannot find the actual minimum. The direction of the correction is calculated by differentiating the respective functions; their negative value indicates the direction in which the error function is minimized. Minimizing the chosen error function is the goal of training or learning.

[0055] Once all weights are adjusted, another iteration of all input data is performed, the error is measured again, and this error is backward propagated to readjust the weights once more. Each complete iteration of all input data is called an epoch. The number of training epochs is also an important hyperparameter for training neural networks. Depending on the size of the dataset, the input data can be divided into equally sized groups, and training can be performed in batches.

[0056] This can be useful, for example, to allow an artificial neural network to learn faster, or to comply with the processing capacity limitations of the computer running the calculation. When dividing the data into batches, it is important that the values ​​within each batch are normally distributed compared to the entire dataset. An epoch is completed when all batches have been processed by the neural network once.

[0057] The more examples an artificial neural network receives for training, and the more often it sees them, the smaller the error in its results becomes. The process of the error curve approaching and converging with the 100% line is called convergence, and during training, the error curve's trajectory is monitored to allow for adjustments to the hyperparameters, if necessary, by stopping the training. However, a small error does not always guarantee good overall performance of the neural network.

[0058] Because if the artificial neural network has seen all known data very often during training, it may simply memorize this data rather than learning an abstract concept. This problem is also known as overfitting. Since neural networks can also model highly complex functions, there is a risk that they will eventually find the perfect function for every known data point, but this function will not work well for new data.

[0059] To ensure that a neural network can abstract from known example data and also deliver correct results for previously unlearned input data, the example data is divided before training into training data, test data and blind test data, e.g. in the ratio 70 / 20 / 10.

[0060] During training, only the training data is used, and the error rate is measured for both the training and test data. However, the measured error of the test data is not fed back into the artificial neural network. The neural network is then improved by adjusting all variables until it achieves maximum performance with respect to both the training and test data. This "tuning" of neural networks is one of the core tasks of artificial intelligence engineers.

[0061] Only when the network is supposedly fully trained are the blind test data used. If the artificial neural network performs well in the blind test as well, there is a high probability that it has learned an abstract concept well.

[0062] The supervised learning described requires a large amount of example data. A large amount here means, for example, one million examples. While artificial neural networks can sometimes achieve remarkable results even with smaller datasets, the more data available, the better. For image classification, for example, usable results are achieved with approximately 1,000 example images per class. An entire field of artificial intelligence research also focuses on methods for so-called one-shot learning, i.e., learning from very few or even just one example.

[0063] Supervised learning itself can be further subdivided into different methods of data use and transmission within artificial neural networks: In so-called recurring neural networks, for example, the result of the previous input data flows into the calculation of the current output, so that, for example, time series and texts can be analyzed and processed. This is also the case, for example, with long-short-term memory networks (LSTM) and sequence-to-sequence networks, which are used, for example, for speech recognition and text translation. For image processing, so-called convolutional neural networks (CNNs) are used, which scan images with a grid and abstract from deeper levels (points, lines, etc.) in each layer to higher-level concepts (a face, a house, etc.).

[0064] One embodiment is a method for training the device according to one of claims 2 or 3.

[0065] One step in the process is providing a training data pair consisting of a training view information and a second training top view.

[0066] A further step in the process is the generation of the second top view depending on the training view information and the noise information using the second generator unit.

[0067] Compared to the training method according to the second aspect of the invention, the training method according to the third aspect of the invention has the advantage that a so-called over-adaptation of the generator and discriminator units to the training data is prevented, since in this case the generator and discriminator units have to cope with unexpected input information generated randomly.

[0068] A further step in the procedure is to use the second discriminator unit to decide whether the second top view was generated by the second generator unit, or whether the second top view was generated by a camera whose optical axis is essentially orthogonal to a plane spanned by the vehicle's longitudinal direction and the vehicle's transverse direction.

[0069] A further step in the process is to compare the second top view with the second training top view, and to adjust the second generator unit and / or the second discriminator unit depending on the comparison of the second top view with the training top view.

[0070] In particular, the second generator unit is adjusted such that the probability increases that the second discriminator unit classifies the second top view as an image from a camera whose optical axis is essentially orthogonal to a plane spanned by the vehicle's longitudinal and transverse directions. Alternatively or additionally, the second discriminator unit is adjusted such that the probability increases that the second discriminator unit recognizes the second top view as an image generated by the second generator unit.

[0071] In particular, the procedure includes the step of adjusting the second generator unit and the second discriminator unit depending on the comparison of the second top view with the training top view.

[0072] Adjusting the second generator unit and / or the second discriminator unit means, in the case that these units each comprise a neural network, an adjustment of the edge weights of the neural networks.

[0073] One embodiment relates to a method for training the device according to one of claims 2 or 3.

[0074] One step of the procedure is to train the device by performing the procedure according to the second aspect of the invention at least once.

[0075] A further step of the procedure is the training of the device by performing the procedure according to the third aspect of the invention at least once.

[0076] In particular, the step of training the device by at least one execution of the method according to the third aspect of the invention is only carried out after the step of training the device by at least one execution of the method according to the second aspect of the invention.

[0077] The invention is described below with reference to an exemplary embodiment and the accompanying drawings. These show: Fig. 1 shows an embodiment of the device according to the invention, Fig. 2 shows an embodiment of a motor vehicle with several cameras, and Fig. 3 shows an embodiment of a neural network.

[0078] Fig. 1 shows a device for generating a first top view d1 of a motor vehicle from at least one image b from at least one camera k1, k2, k3, k4, the optical axis of which is essentially parallel to a plane spanned from the longitudinal direction of the vehicle and the transverse direction of the vehicle.

[0079] The device is configured to receive at least one image b from the camera k1, k2, k3, k4, to extract a second visual information s2 from the image b using a coding unit E, and to extract a visual-independent content information z from the image b using the coding unit E.

[0080] Furthermore, the device is set up to generate the first top view d1 depending on the first view information s1 and the view-independent content information z by means of a first generator unit G1.

[0081] The device also comprises a first discriminator unit D1, which is configured to receive the top view d1 generated by the first generator unit G1, a second generator unit G2, which is configured to generate a second top view d2 from a third view information s3 and a noise information r, and a second discriminator unit D2, which is configured to receive the top view d2 generated by the second generator unit G2.

[0082] The first discriminator unit D1 and the second discriminator unit D2 are each configured to decide whether the first top view d1, or the second top view d2, was generated by the first generator unit G1, or the second generator unit G2, respectively, or whether the first top view d1, or the second top view d2, was generated by a camera whose optical axis is substantially orthogonal to a plane spanned by the vehicle's longitudinal direction and the vehicle's transverse direction.

[0083] The first generator unit G1, the second generator unit G2, the first discriminator unit D1 and the second discriminator unit D2 each comprise a neural network.

[0084] The neural network of the first generator unit G1 and the neural network of the second generator unit G2 each have at least partially the same edge weights, and the neural network of the first discriminator unit D1 and the neural network of the second discriminator unit D2 each also have at least partially the same edge weights.

[0085] This device can be trained by the methods according to the second aspect of the invention or according to the embodiment of claims 2 or 3.

[0086] The method according to the second aspect of the invention comprises in particular the following steps: Providing a training data pair consisting of a training image b and a first training top view td1, extracting the second view information s2 from the training image b using the coding unit E, extracting the view-independent content information z from the training image b using the coding unit E, generating the first top view d1 as a function of the first view information s1 and the view-independent content information z using the first generator unit G1, deciding using the first discriminator unit D1 whether the first top view s1 was generated by the first generator unit G1, or whether the first top view s1 was generated by a camera whose optical axis is essentially orthogonal to a plane spanned by the vehicle's longitudinal and transverse directions, comparing the first top view s1 with the first training top view td1.and adjusting the coding unit E and the first generator unit G1 and the first discriminator unit D1 depending on the comparison of the first top view d1 with the first training top view td1.

[0087] The method according to the second aspect of the invention comprises in particular the following steps: Providing a training data pair consisting of a training view information s3 and a second training top view td2, generating the second top view d2 as a function of the training view information s3 and the noise information r using the second generator unit G2, deciding using the second discriminator unit D2 whether the second top view s2 was generated by the second generator unit G2, or whether the second top view s2 was generated by a camera whose optical axis is substantially orthogonal to a plane spanned by the vehicle's longitudinal and transverse directions, comparing the second top view d2 with the second training top view td2, and adjusting the second generator unit G2 and the second discriminator unit D2 as a function of the comparison of the second top view d2 with the training top view td2.

[0088] The method for training the device according to the fourth aspect of the invention advantageously combines the two training methods according to the second aspect and the third aspect of the invention.

[0089] This effect arises in particular from the fact that the first generator unit G1 and the second generator unit G2, as well as the first discriminator unit D1 and the second discriminator unit D2, are connected by sharing common edge weights.

[0090] In a preferred embodiment, the training method according to the second aspect of the invention is first carried out more intensively, before the training method according to the third aspect of the invention is carried out more intensively at a later time.

[0091] Fig. 2 Figure 1 shows an embodiment of a motor vehicle with several cameras k1, k2, k3, k4, each of which is arranged on the motor vehicle such that its optical axes are essentially parallel to a plane spanned by the longitudinal direction of the vehicle and the transverse direction of the vehicle.

[0092] Fig. 3 shows an embodiment of a neural network consisting of four layers A, B, C, D of neurons A1-A4, B1-B5, C1-C5, D1-D3.

[0093] The neurons are partially connected to each other by edges. As already explained, each of these edges is a Fig. 3 Edge weight not shown is assigned.

Claims

1. Device for generating a first top view (d1), dependent on first view information (s1), of a motor vehicle from at least one image (b) from at least one camera (k1, k2, k3, k4) the optical axis of which is substantially parallel to a plane spanned from the vehicle longitudinal direction and the vehicle transverse direction, wherein the device is configured • to receive the at least one image (b) from the camera (k1, k2, k3, k4), • to extract second view information (s2) from the image (b) by way of a coding unit (E), • to extract view-independent content information (z) from the image (b) by way of the coding unit (E), and • to generate the first top view (d1) on the basis of the first view information (s1) and the view-independent content information (z) by way of a first generator unit (G1), characterized in that the device additionally • comprises a first discriminator unit (D1) that is configured to receive the top view (d1) generated by the first generator unit (G1), • comprises a second generator unit (G2) that is configured to generate a second top view (d2) from third view information (s3) and noise information (r), and • comprises a second discriminator unit (D2) that is configured to receive the top view (d2) generated by the second generator unit (G2), wherein the coding unit (E), the first generator unit (G1), the second generator unit (G2), the first discriminator unit (D1) and the second discriminator unit (D2) each comprise a neural network.

2. Device according to Claim 1, wherein the first discriminator unit (D1) and the second discriminator unit (D2) are each configured to decide whether the first top view (d1), respectively the second top view (d2), were generated by the first generator unit (G1), respectively second generator unit (G2), or whether the first top view (d1), respectively the second top view (d2), were generated by a camera the optical axis of which is substantially orthogonal to a plane spanned from the vehicle longitudinal direction and the vehicle transverse direction.

3. Device according to either of Claims 1 and 2, wherein • the neural network of the first generator unit (G1) and the neural network of the second generator unit (G2) each have at least partially the same edge weights, and • the neural network of the first discriminator unit (D1) and the neural network of the second discriminator unit (D2) each have at least partially the same edge weights.

4. Method for training the device according to one of Claims 1, 2 or 3, wherein the method comprises the following steps: • providing a training data pair consisting of training view information (s3) and a second training top view (td2), • generating the second top view (d2) on the basis of the training view information (s3) and the noise information (r) by way of the second generator unit (G2), • deciding, by way of the second discriminator unit (D2), whether the second top view (s2) was generated by the second generator unit (G2), or whether the second top view (s2) was generated by a camera the optical axis of which is substantially orthogonal to a plane spanned from the vehicle longitudinal direction and the vehicle transverse direction, • comparing the second top view (d2) with the second training top view (td2), and • adjusting the second generator unit (G2) and / or the second discriminator unit (D2) on the basis of the comparison of the second top view (d2) with the training top view (td2).

5. Method according to Claim 4, wherein the method comprises the following step: • adjusting the second generator unit (G2) and the second discriminator unit (D2) on the basis of the comparison of the second top view (d2) with the training top view (td2).

6. Method for training the device according to one of Claims 1, 2 or 3, wherein the method comprises the following steps: • providing a training data pair consisting of a training image (b) and a first training top view (td1), • extracting the second view information (s2) from the training image (b) by way of the coding unit (E), • extracting the view-independent content information (z) from the training image (b) by way of the coding unit (E), • generating the first top view (d1) on the basis of the first view information (s1) and the view-independent content information (z) by way of the first generator unit (G1), • deciding, by way of the first discriminator unit (D1), whether the first top view (s1) was generated by the first generator unit (G1), or whether the first top view (s1) was generated by a camera the optical axis of which is substantially orthogonal to a plane spanned from the vehicle longitudinal direction and the vehicle transverse direction, • comparing the first top view (s1) with the first training top view (td1), and • adjusting the coding unit (E) and / or the first generator unit (G1) and / or the first discriminator unit (D1) on the basis of the comparison of the first top view (d1) with the first training top view (td1), • training the device by carrying out the method according to either of Claims 4 and 5 at least once.

7. Method according to Claim 6, wherein the step of training the device by carrying out the method according to either of Claims 4 and 5 at least once takes place only after the other steps.