Method for determining a positioning pose of an at least partially automated mobile platform

Abstract

A method for determining a localization pose of an at least partially automated mobile platform is proposed, wherein the mobile platform is provided for generating a ground image of an environment of the mobile platform and for receiving an aerial image of the environment of the mobile platform from an aerial image system, the method having the following steps: providing (S1) a digital ground image of the environment of the mobile platform; receiving (S2) an aerial image of the environment of the mobile platform; generating (S3) a localization pose of the mobile platform by means of a trained neural convolutional network, the neural convolutional network having a first trained neural encoding convolutional network portion and a second trained neural encoding convolutional network portion.

Description

TECHNICAL FIELD

[0001] The invention relates to a method for determining a localization pose of an at least partially automated mobile platform by means of aerial images and ground images of the surroundings of the mobile platform. BACKGROUND

[0002] Precise localization is a prerequisite for driving an at least partially automated platform, for example an autonomously operating vehicle.

[0003] For the localization of such mobile platforms by means of ground images of the surroundings of the mobile platform, various different approaches have been taken, which are typically feature-based in terms of the surroundings of the mobile platform, wherein the features are then assigned to a pose of the mobile platform by means of a high-resolution map. SUMMARY

[0004] However, the use of such high-resolution maps has economic disadvantages. In contrast, deep learning-based approaches for determining a pose by means of regression based on ground images have the advantage of a determined size of the respective map and a constant query time. By means of monocular images, video image sequences and depth images from a direct camera position, localization can be determined by means of such approaches. Here, localization in very large geographical areas, for example in larger urban areas, is challenged in terms of the capacity of such neural networks, in terms of scalability.

[0005] The invention discloses a method for determining a localization pose of an at least partially automated mobile platform, a method for generating a trained neural convolutional network for determining a localization pose, a method for maneuvering a mobile platform, a computer program and a machine-readable storage medium. Advantageous configurations are described in the description.

[0006] The invention is based on the recognition that the spatial context and perspective of the surroundings of the mobile platform can be used by means of aerial images centered on an estimated position of the mobile platform in order to train a neural network and determine a pose of the mobile platform in combination with ground images. This enables, inter alia, a correct assignment of non- uniquely unambiguous features in ground images over larger geographical areas.

[0007] According to one aspect, a method for determining a localization pose of an at least partially automated mobile platform is proposed, wherein the mobile platform is configured to generate a ground image of the surroundings of the mobile platform and to receive an aerial image of the surroundings of the mobile platform from an aerial image system. In one step of the method, a digital ground image of the surroundings of the mobile platform is provided. In another step, an aerial image of the surroundings of the mobile platform is received. In a further step, a localization pose of the mobile platform is generated by means of a trained neural convolutional network, wherein the neural convolutional network has a first trained neural encoding convolutional network portion and a second trained neural encoding convolutional network portion.

[0008] By means of the method, a localization pose can advantageously be determined by means of a visual ground image and a visual aerial image of the surroundings of the mobile platform without the use of high-resolution maps. Thus, for training the neural convolutional network, aerial images are used to pre-determine the pose of the mobile platform. Since the method is not based on handcrafted features, it can be well scaled in terms of larger geographical areas.

[0009] By means of this method based on a trained neural convolutional network, features can be learned from a larger spatial context, for example from aerial images in combination with ground images, in order to accurately determine the pose of the mobile platform on a city scale. Here, the aerial images can be present in the form of satellite images and the ground images can be generated by the mobile platform by means of a digital camera system.

[0010] The neural convolutional network essentially has alternately repeated filter (Convolutional Layer, convolution layer) and aggregation layers (Pooling Layer, pooling layer) and can have one or more layers of "normal" fully connected neurons (Dense / Fully Connected Layer, dense layer / full connection layer) at the end of the network. The first or second trained neural encoding convolutional network portion can here be configured as part of the neural convolutional network, or the network portions can each be implemented in the form of a single neural convolutional network.

[0011] The ground image and the aerial image can both exist as digital images of different perspectives of the surrounding environment from the mobile platform and can be generated, for example, by means of a digital camera system. The perspective of the aerial image on the surrounding environment of the mobile platform is a top down view. Such an aerial image can be generated, for example, by a camera system of a satellite, an aircraft or a drone. Here, such an aerial image can not only be a single finished aerial image of the surrounding environment of the mobile platform, but can also be, for example, a partial view of a larger aerial image, wherein the partial view is centered, in particular, on the estimated pose of the mobile platform. Such an aerial image can be, in particular, a satellite image tile, which can be called up for a determined satellite navigation position, for example, a GPS position.

[0012] The localization pose of the mobile platform is a pose which defines a position in three spatial dimensions and an orientation of the mobile platform in space, which can be specified, for example, by three Euler angles determined by the method.

[0013] The reference pose of the mobile platform is a pose which provides a very accurate specification of the training for the determination of the localization pose of the method, for example, by a reference system for determining the pose of the mobile platform.

[0014] Feedforward neural networks provide a framework for a variety of different algorithms for machine learning, cooperative work and processing of complex data inputs. Such neural networks learn to perform a task from examples, often without being programmed with specific rules for the task.

[0015] Such a neural network is based on a collection of connected units or nodes called artificial neurons. Each connection can transmit a signal from an artificial neuron to another. An artificial neuron that receives a signal can process it and then activate other artificial neurons connected to it.

[0016] In a conventional implementation of a neural network, the signals at the synapses of artificial neurons are real numbers, and the output of an artificial neuron is computed from the sum of its inputs through a non-linear function. The connections between artificial neurons often have weights matched to their strengths during learning procedures. The weights increase or decrease the strength of the signal at the synapse. Artificial neurons can have a threshold to fire only if the total signal exceeds the threshold.

[0017] Typically, multiple artificial neurons are integrated in layers. Different layers can perform different types of transformations on their inputs. Signals propagate from the first layer (input layer) - possibly after passing through multiple layers - to the last layer (output layer).

[0018] As a supplement to the above implementations of feedforward neural networks, the architecture of artificial convolutional neural networks can also comprise one or more convolutional layers (English convolutional layer, if necessary followed by pooling layers. The sequence of layers can be used with or without normalization layers (e.g. batch normalization), zero padding layers, exit layers and activation functions (e.g. Rectified Linear Unit (ReLU), tanh function or softmax function). In principle, these units can be repeated arbitrarily often, at sufficient repetition deep convolutional neural networks can be involved.

[0019] In order to train the thus defined architecture of a neural encoding-decoding convolutional network, each neuron obtains random initial weights, for example. Input data is then input into the network, each neuron weights the input signal with its weights, the result is then further provided to the neurons of the next layer. At the output layer the result is provided. The size of the error can be calculated as well as the share of each neuron on this error, then the weights of each neuron can be changed in the direction of minimizing the error. Then a recursive run is made, the error measurement is repeated and the weights are adapted until the error is below a predefined limit.

[0020] In the entire description of the present application, sequences of method steps are shown in order to make the method easily understandable. However, the person skilled in the art will recognize that a number of the described method steps can also be performed in a different order and produce the same result. In this sense, the order of the method steps can be changed accordingly and is therefore also disclosed.

[0021] Mobile platforms can be understood as driver assistance systems of at least partially automated mobile systems and / or vehicles. One example can be an at least partially automated vehicle or a vehicle with a driver assistance system. That is, in this case, the at least partially automated system comprises the mobile platform in terms of at least partially automated functions, but the mobile platform also comprises vehicles and other mobile machines that include driver assistance systems. Other examples of mobile platforms can be driver assistance systems with multiple sensors, mobile multi-sensor robots (such as robotic vacuum cleaners or lawn mowers), multi-sensor monitoring systems, production machines, personal assistants, short shuttles, self-driving taxis, ships, aircraft, commercial vehicles or access control systems. Each of these systems can be a fully or partially automated system.

[0022] According to one aspect it is proposed that, for generating the localization pose, in one step a ground image is input as an input signal of a first trained neural encoding convolutional network portion in order to form a first encoding vector. In another step, an aerial image is input as an input signal of a second trained neural encoding convolutional network portion in order to form a second encoding vector. In a further step, the localization pose of the mobile platform is generated by means of the fused first and second encoding vectors.

[0023] Here, inputting an input signal into a neural network means that the signal, i.e. a ground image or an aerial image, is transmitted to the input layer of the neural network.

[0024] According to one aspect it is proposed that at least one layer of the first trained neural encoding convolutional network portion has the same weights as at least one layer of the second trained neural encoding convolutional network portion.

[0025] In this method, not only a ground image, e.g. an RGB image from a front-facing camera of the mobile platform, but also an aerial image, e.g. a satellite image, is provided to the neural convolutional network. By the common weights of at least one early convolutional layer of both networks, it is made possible for the neural network to exchange different information between the two network portions at an early stage before the output signals of both network portions are fused for a later final pose regression. In the fusion of the two output signals of both network portions, the features are fused by means of a fully connected layer in order to determine the pose.

[0026] In other words, this method helps to disambiguate ground images that look very similar but are spatially far apart. Without this exchange of weights, both the aerial image and the ground image would lead to similar poses for these images. By considering aerial images of the surroundings of the mobile platform by means of this method, the neural convolutional network is trained to learn discriminative features by means of the significant spatial arrangement of the features of the aerial images, and additionally the pose can be determined more accurately.

[0027] In order to be able to determine the vehicle position or the vehicle localization pose with a higher accuracy, not the similarity of the ground images and the aerial images, e.g. at least local parts of satellite images, is compared, but the pose of the mobile platform is derived from the provided ground images in combination with the respective local aerial image or local satellite imagery.

[0028] Thus, an end-to-end learning based on ground images and aerial images is performed in order to achieve a good scalability. Thus, the advantages of the position prior experience in terms of good scalability are combined with the advantages of using a neural convolutional network.

[0029] According to one aspect it is proposed to fuse the first encoding vector with the second encoding vector in such a way that the first encoding vector and the second encoding vector are concatenated and fully connected with at least one output layer of a fusion part of the neural convolutional network, wherein an output signal of the output layer is indicative of the localization pose.

[0030] Herein, in a fully connected layer, the neurons of one layer are connected with all neurons of the next layer, thus the layer is called "fully-connected layer".

[0031] According to one aspect it is proposed to generate the aerial image of the surrounding of the mobile platform by means of a satellite, an aircraft or a drone.

[0032] According to one aspect it is proposed to select the aerial image by means of a pose of the mobile platform which has been determined by means of a navigation system which is supported by a global navigation system and / or a mobile radio. By means of this predefinition by means of the position of the navigation system, the search space for features can be reduced and the determination of the pose by means of the ground image can be estimated more precisely by means of the reduced amount of data.

[0033] According to another aspect it is proposed to generate the ground image of the surrounding of the mobile platform by means of a digital camera system. Herein, the ground image is typically generated by means of a digital camera system with a corresponding view angle by a front camera of the mobile platform.

[0034] This method for determining a localization pose of a mobile platform can be combined with different existing methods for improving the pose determination. For example, this is especially the integration of sequential information and the consideration of geometric constraints which leads to a further performance improvement.

[0035] The main advantage of the method is the scalability of the application of the method, since in the method correlation information and localization information of large areas are considered. Furthermore, a constant query time for the pose determination is derived by means of the method, which is not applicable in conventional feature-based methods. For example, in 3D-3D / 2D-3D feature matching, a good scaling in large map sizes is not achieved.

[0036] In the method a fixed "map size" is derived, since the map is implicitly represented by the set and stored weights of the network. Additionally, information which is publicly accessible is used for the first estimated pose by means of the method, for example satellite images can be used for aerial images, which is economically advantageous and does not require manual labeling.

[0037] According to an aspect, a method is proposed for generating a trained neural convolutional network for determining a localization pose of an at least partially automated mobile platform by means of a ground image of a surrounding of the mobile platform and an aerial image of the surrounding of the mobile platform. Herein, the neural convolutional network has a first neural encoding convolutional network portion and a second neural encoding convolutional network portion and a fusion portion.

[0038] The trained neural convolutional network is generated by means of a plurality of training cycles, wherein each training cycle has the following steps. In one step, a reference pose of the at least partially automated mobile platform is provided. In another step, a ground image of the surrounding of the mobile platform in the reference pose is provided. In another step, an aerial image of the surrounding of the mobile platform in the reference pose is provided. In another step, the ground image is used as an input signal for the first neural encoding convolutional network portion in order to generate a first output signal. In another step, the aerial image is used as an input signal for the second neural encoding convolutional network portion in order to generate a second output signal. In another step, the localization pose is determined by means of the fusion portion which fuses the first output signal and the second output signal. In another step, the neural convolutional network is adapted in order to minimize a deviation from the respective reference pose when determining the respective determined localization pose.

[0039] According to an aspect, it is proposed that the fusion of the first output signal and the second output signal for generating the trained neural convolutional network has the following steps. In one step, a first encoding vector is formed by means of the first output signal. In another step, a second encoding vector is formed by means of the second output signal. In another step, the first encoding vector and the second encoding vector are fused by concatenating the first encoding vector and the second encoding vector and fully connecting the concatenated encoding vectors to an output layer of the fusion portion of the neural convolutional network, wherein the output layer accounts for the localization pose.

[0040] According to an aspect, it is proposed that, when adapting (or in other words training) the neural convolutional network, at least one layer of the first neural encoding convolutional network portion and the corresponding layer of the second neural encoding convolutional network portion exchange respective weights of the corresponding layers with each other.

[0041] A method is proposed in which, on the basis of the localization pose of the at least partially automated mobile platform, a control signal is provided for maneuvering the mobile platform. Alternatively or additionally, on the basis of the localization pose of the at least partially automated mobile platform, a warning signal is provided for warning an occupant of the vehicle.

[0042] The term "based on" should be interpreted broadly in the sense of the provision of a control signal based on a positioning attitude. It is to be understood in this way that the positioning attitude can be taken into account for any determination or calculation of the control signal, wherein this does not exclude that also other input quantities can be taken into account for the determination of the control signal. This applies equally to the provision of a warning signal.

[0043] A device is described which is arranged for carrying out one of the above-mentioned methods. By means of this device the method can easily be integrated into different systems.

[0044] A computer program is described which comprises instructions which, when the computer program is executed by a computer, cause the computer to carry out one of the above-mentioned methods. This computer program makes it possible to use the described method in different systems.

[0045] A machine-readable storage medium is described on which the above-mentioned computer program is stored. BRIEF DESCRIPTION OF DRAWINGS

[0046] REFERENCE Figure 1 and Figure 2 Embodiments of the application are illustrated and hereinafter further described. The accompanying drawings illustrate:

[0047] Figure 1 A flow diagram is shown of a method for determining a positioning attitude of an at least partially automated mobile platform;

[0048] Figure 2 A flow diagram is shown of a method for generating a trained neural convolutional network for determining a positioning attitude. DETAILED DESCRIPTION

[0049] Figure 1 A method 100 for determining a positioning attitude 130 of an at least partially automated mobile platform is shown schematically in a data flow diagram. A digital ground image 110 of the surroundings of the mobile platform S1 can be provided, for example by the mobile platform itself, which is arranged for generating the ground image, for example by means of a digital camera system.

[0050] Additionally, a bird's eye view image of the surroundings of the mobile platform is received S2, for example by the mobile platform from a bird's eye view image system. Such a bird's eye view image system can generate a bird's eye view image 120 of the surroundings of the mobile platform, for example by means of a satellite, an aircraft or a drone. In order to generate or select the bird's eye view image 120 of the surroundings of the mobile platform, the pose of the mobile platform can be determined, for example by means of a global navigation system and / or a mobile radio-supported navigation system. Here, the mobile platform itself can be provided with a system which determines such a pre-determination of the pose by means of a global navigation system and / or a mobile radio-supported navigation system. Then, the localization pose of the mobile platform is generated S3 by means of a trained neural convolutional network, wherein the neural convolutional network has a first trained neural encoding convolutional network portion 112, 114, 116 and a second trained neural encoding convolutional network portion 122, 124, 126.

[0051] In order to generate S3 the localization pose 130 of the mobile platform, the ground image 110 is input S4 as an input signal of the first trained neural encoding convolutional network portion 112, 114, 116 in order to form a first encoding vector 116. In parallel, the bird's eye view image 120 is input S5 as an input signal of the second trained neural encoding convolutional network portion 122, 124, 126 in order to form a second encoding vector 126. The localization pose 130 of the mobile platform is generated S6 by means of a fusion of the first encoding vector 116 and the second encoding vector 126. For this purpose, at least one layer 112 of the first trained neural encoding convolutional network portion 112, 114, 116 and at least one corresponding layer 122 of the second trained neural encoding convolutional network portion 122, 124, 126 have identical weights. Here, the first encoding vector 116 and the second encoding vector 126 are fused in such a way that the first encoding vector 116 and the second encoding vector 126 are concatenated and fully connected with at least one output layer of the fusion portion 118 of the neural convolutional network, wherein the output signal of the output layer of the fusion portion of the neural convolutional network is indicative of the localization pose 130.

[0052] Figure 2 A method 200 for generating a trained neural convolutional network for determining a localization pose 130 of a mobile platform which is at least partially automated by means of a ground image 110 of the surroundings of the mobile platform and a bird's eye view image 120 of the surroundings of the mobile platform is illustrated schematically in a data flow diagram. The neural convolutional network has a first neural encoding convolutional network portion 112, 114, 116, a second neural encoding convolutional network portion 122, 124, 126 and a fusion portion 118.

[0053] The trained convolutional neural network 140 is generated with a plurality of training cycles, wherein each training cycle has the following steps. In a step S21, a reference pose 220 of the at least partially automated mobile platform is provided. In a further step S22, a ground image 110 of the surrounding environment of the mobile platform in the reference pose 220 is provided. In a further step S23, an aerial image of the surrounding environment of the mobile platform in the reference pose 220 is provided. In a further step S24, the ground image 220 is used as an input signal for the first neural encoding convolutional network portion 112, 114, 116 in order to generate a first output signal. In a further step S25, the aerial image 120 is used as an input signal for the second neural encoding convolutional network portion 122, 124, 126 in order to generate a second output signal. In a further step, the determined pose 130 is determined S26 by means of a fusion portion 118 which fuses the first output signal and the second output signal. In a further step S27, the neural convolutional network is adapted in order to minimize the deviation from the respective reference pose 220 when determining the respective determined pose 130. In the adaptation S27 of the neural convolutional network, at least one layer 112 of the first neural encoding convolutional network portion 112, 114, 116 and the corresponding layer 112 of the second neural encoding convolutional network portion 122, 124, 126 exchange respective weights of the corresponding layers 112, 122 with each other, so that the respective layers have the same weights after the training.

[0054] The fusion of the first output signal and the second output signal for generating the trained neural convolutional network has the following steps. In a step S28, a first encoding vector 116 is formed by means of the first output signal. In a further step S29, a second encoding vector 126 is formed by means of the second output signal. In a further step S30, the first encoding vector 116 and the second encoding vector 126 are fused in such a way that the first encoding vector 116 and the second encoding vector 126 are concatenated and the concatenated first encoding vector 116, 126 is fully connected with an output layer of the fusion portion 118 of the neural convolutional network, wherein the output layer specifies the determined pose 130.

Claims

1. A method (100) for determining the positioning attitude (130) of a mobile platform that is at least partially automated, wherein, The mobile platform is configured to generate a ground image (110) of the surrounding environment of the mobile platform, and the mobile platform is configured to receive an aerial image (120) of the surrounding environment of the mobile platform from an aerial image system. The method includes the following steps: providing (S1) a digital ground image (110) of the surrounding environment of the mobile platform; receiving (S2) the aerial image (120) of the surrounding environment of the mobile platform; generating (S3) a positioning pose (130) of the mobile platform using a trained neural convolutional network (140), the neural convolutional network having a first trained neural coding convolutional network portion (112, 114, 116) and a second trained neural coding convolutional network portion (122, 124, 126), wherein the generation of the positioning pose (130) includes the following steps: using the ground image (110) as the first trained neural coding convolutional network portion (112, 114, 116) The input signal of 6) is input (S4) to form a first encoding vector (116); the aerial image (120) is input as the input signal of the second trained neural coding convolutional network part (122, 124, 126) to form a second encoding vector (126); the positioning pose of the mobile platform is generated (S6) by means of the fused first encoding vector and second encoding vector (116, 126), the first encoding vector (116) and the second encoding vector (126) are fused in such a way that the first encoding vector (116) and the second encoding vector (126) are concatenated and fully connected to at least one output layer of the fusion part (118) of the neural convolutional network (140), wherein the output signal of the output layer describes the positioning pose (130), wherein the similarity between the ground image (110) and the aerial image (120) is not compared.

2. The method (100) according to claim 1, wherein, At least one layer (112) of the first trained neural coding convolutional network portion (112, 114, 116) and at least one corresponding layer (122) of the second trained neural coding convolutional network portion (122, 124, 126) have the same weights.

3. The method (100) according to claim 1 or 2, wherein, Aerial images (120) of the surrounding environment of the mobile platform have been generated by means of satellites, aircraft or drones.

4. The method (100) according to claim 1 or 2, wherein, The aerial imagery (120) is selected using the attitude of the mobile platform, which has been determined by means of a global navigation system and / or a navigation system supported by a mobile radio.

5. The method (100) according to claim 1 or 2, wherein, Ground images (110) of the surrounding environment of the mobile platform have been generated using a digital camera system.

6. The method according to claim 1 or 2, wherein, Based on the positioning attitude (130), control signals are provided for operating at least partially automated mobile platforms; and / or, based on the positioning attitude (130), warning signals are provided for warning occupants of the at least partially automated mobile platforms.

7. A method (200) for generating a trained neural convolutional network (140) using ground images (110) of the surrounding environment of a mobile platform and aerial images (120) of the surrounding environment of the mobile platform, the trained neural convolutional network being used to determine the localization pose (130) of a mobile platform that is at least partially automated, wherein, The neural convolutional network (140) has a first neural coding convolutional network portion (112, 114, 116), a second neural coding convolutional network portion (122, 124, 126), and a fusion portion (118), and is generated by means of multiple training cycles to train the neural convolutional network (140), wherein each training cycle has the following steps: providing (S21) a reference pose (220) of the at least partially automated mobile platform; providing (S22) a ground plane of the surrounding environment of the mobile platform in the reference pose. A ground image (110); providing (S23) an aerial image (120) of the surrounding environment of the mobile platform in the reference pose; using (S24) the ground image (110) as the input signal of the first neural coding convolutional network portion (112, 114, 116) to generate a first output signal; using (S25) the aerial image (120) as the input signal of the second neural coding convolutional network portion (122, 124, 126) to generate a second output signal; determining (S26) the positioning pose (130) by means of the fusion portion (118) that fuses the first output signal and the second output signal; adapting the neural convolutional network (140) (S27) to minimize the deviation from the corresponding reference pose (220) when determining the corresponding determined positioning pose (130), the fusion of the first output signal and the second output signal having the following steps: forming (S28) a first coding vector (116) by means of the first output signal; forming (S29) a second coding vector by means of the second output signal. Code vector (126); the first encoding vector (116) and the second encoding vector (126) are fused (S30) by concatenating the first encoding vector (116) and the second encoding vector (126) and fully connecting the concatenated encoding vector (116, 126) to the output layer of the fusion part (118) of the neural convolutional network (140), the output layer describing the localization pose (130), wherein the similarity between the ground image (110) and the aerial image (120) is not compared.

8. The method (200) according to claim 7, wherein, When adapting the neural convolutional network (140) (S27), at least one layer (112) of the first neural coding convolutional network portion (112, 114, 116) and the corresponding layer (122) of the second neural coding convolutional network portion (122, 124, 126) exchange the corresponding weights of the corresponding layers (112, 122).

9. The method according to claim 7 or 8, wherein, Based on the positioning attitude (130), control signals are provided for operating at least partially automated mobile platforms; and / or, based on the positioning attitude (130), warning signals are provided for warning occupants of the at least partially automated mobile platforms.

10. An apparatus for determining the positioning attitude (130) of a mobile platform that is at least partially automated, the apparatus being configured to perform the method according to any one of claims 1 to 9.

11. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 9.

12. A machine-readable storage medium on which a computer program product according to claim 11 is stored.

Images