Method for self-localization of a vehicle in a specific environment

Abstract

The invention relates to a method (100) for self-localization of a vehicle (1) in a specific environment (2), comprising the following steps: - detecting (101) the specific environment (2) of the vehicle (1) on the basis of sensor data, wherein the sensor data result from a detection of at least one sensor (3) of the vehicle (1), - initiating (102) a transmission of a machine learning model (4) trained on the basis of the specific environment from an external data processing device (5) to the vehicle (1), - extracting (103) features (6) in the sensor data using the trained machine learning model (4), - transmitting (104) the extracted features (6) to the external data processing device (5), - providing (105) localization information (7) by the external data processing device (5), wherein the localization information (7) is determined on the basis of the transmitted extracted features (6), - performing (106) self-localization based on the provided localization information (7). Furthermore, the invention relates to a computer program, a device and a storage medium for this purpose.

Description

The invention relates to a method for self-locating a vehicle in a specific environment. The invention further relates to a computer program, an apparatus and a storage medium for this purpose.Prior ArtA video-based self-localization of vehicles is based, for example, on the fact that image features from a current image recorded by a vehicle camera are found back in a previously recorded map of the environment. These so-called correspondences between the current view of the vehicle and the map representation of the environment can enable the position determination of the camera. For image feature extraction, hand-designed features, such as edge and corner detectors or revised algorithms, have been used in the conventional way. These have been successful in the past but have failed to be used in many situations where larger variations have occurred between the images taken at the mapping time and the images taken at the localization time. These variations occur, for example, in the case of drastic changes in the angle of view, or in the case of changes in the time of day or in the weather conditions or in the season.Another approach from the prior art is, for example, that classic algorithms or deep neural networks are used for image feature extraction for vehicle localization. However, the vehicle always uses the same method for image feature extraction, or it uses one of a few available methods, for example one that is used during the day and one that is used at night.Another prior art approach is to offload the localization task to a server. In this case, the relevant data processing takes place completely on the server, in which the images are sent directly thereto from the vehicle. As a result, the image feature extraction is also transferred to the server and the latter can correspondingly use a location-specific method for this purpose. However, the disadvantage here is the high volume of data, since images have to be transmitted substantially completely to the server in real time.An alternative approach to using a classical map of 3D points with associated image feature descriptors is to use a learned map in which the local visual information is implicitly integrated into a deep neural network; these are skipped by definition to a particular local environment. However, they have the disadvantage compared to the classic variant that they can be expanded less flexibly, can be interpreted more difficultly, and currently do not allow the same precision in the position determination.It is also part of the prior art, in particular in automated valve parking, that the position determination is not necessarily carried out with the aid of the sensors of the vehicle, since the vehicle can be located from the outside, for example by sensors integrated into the infrastructure. Especially in cases in which large areas (parking spaces) are to be used as the field of application, it may be disadvantageous if sensors integrated into the infrastructure have to be used.Disclosure of the InventionThe subject matter of the invention is a method with the features of claim 1, a computer program with the features of claim 8, an apparatus with the features of claim 9 and a computer-readable storage medium with the features of claim 10. Features and details which are described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, so that with regard to the disclosure reference is or can always be made to the individual aspects of the invention in a mutually alternating manner.The invention relates in particular to a method for self-locating a vehicle in a specific environment, comprising the following steps, wherein the steps can be carried out repeatedly and / or successively. Within the scope of the present invention, the specific environment is in particular a local environment in which the vehicle is located. This could be, for example, a parking garage or a parking lot. The vehicle can be designed as an at least partially automated or autonomous vehicle or also as a robot, which locates itself, for example, in a specific environment such as a factory hall.In a first step, the specific environment of the vehicle is preferably detected on the basis of sensor data, wherein the sensor data result from a detection of at least one sensor of the vehicle. The sensor data can comprise, for example, image data, radar data, ultrasonic data and / or LiDAR data. The at least one sensor can accordingly be designed as a camera, a radar sensor, an ultrasonic sensor and / or a LiDAR sensor. The specific environment can also be detected, for example, on the basis of a GPS of a navigation system of the vehicle. For example, the navigation system of the vehicle could report that the vehicle is in the vicinity of a parking garage, possibly this was also input as a navigation destination, or it can be predicted that this is the destination of the vehicle.In a further step, a transmission of a machine learning model trained on the basis of the specific environment from an external data processing device to the vehicle is preferably initiated. It is conceivable that an indicator is first determined by the vehicle, which is specific to the specific environment. Based on the indicator that may be communicated to the external computing device to initiate the communication of the machine learning model, the external computing device may determine an appropriate machine learning model and then communicate it to the vehicle. The indicator could be, for example, at least one detected object in the specific environment or a combination of at least two detected objects. A form, such as a size or shape of objects, can also serve as an indicator in order to detect the specific environment. Alternatively, the specific environment can also be determined on the basis of current position information of the vehicle. This position information can be provided by a navigation system or a GNSS unit or can also be derived on the basis of the mobile radio cell in which a mobile radio unit of the vehicle is located.In a further step, features in the sensor data are preferably extracted using the trained machine learning model. These features may indicate, for example, certain patterns or objects in the sensor data.In a further step, the extracted features are preferably transmitted to the external data processing device. This can be carried out, for example, via a wired or a wireless connection, wherein a wireless connection is preferred. Since only the extracted features are transmitted to the external data processing device instead of the sensor data themselves, a communication effort between the vehicle and the external data processing device can be advantageously reduced.In a further step, localization information is preferably provided by the external data processing device, wherein the localization information is ascertained on the basis of the transmitted extracted features. The localization information may indicate, for example, where the vehicle is located in the specific environment, and preferably further an orientation of the vehicle within the specific environment. In particular, with such a localization on the basis of sensor data of a sensor which monitors the environment of the vehicle, localization information can be determined with significantly higher precision than the localization information which is typically determined by a navigation system installed in the vehicle on the basis of GNSS, IMU and odometry.In a further step, the self-localization is preferably carried out on the basis of the provided localization information. Expressed simply, the vehicle thus knows, on the basis of the provided localization information, where it is located within the specific environment and, for example, further how it is oriented therein. Based on this information, for example, navigation of the vehicle within the specific environment can be carried out.It is conceivable that, as part of performing the self-localization, an incremental calculation of a position, and preferably also an orientation, of the vehicle takes place, wherein at least the steps of extracting the features, transmitting the extracted features and providing the localization information are performed at defined intervals. Accordingly, a current position and preferably also a current orientation of the vehicle can be advantageously determined in each case at the defined time intervals, such as for example every 100 μs, on the basis of the provided localization information.The method may further comprise the step of:recording sensor data in the specific environment, wherein the recording is performed with at least two different viewpoints to provide the recorded sensor data as training data for the machine learning model trained based on the specific environment.This allows a detailed training data base to be determined for the machine learning model trained based on the specific environment. It is conceivable that the training data can be continuously expanded by further recorded sensor data in order to advantageously provide a precise and currently trained machine learning model. For example, a change in the specific environment, for example due to a modification, can thereby be taken into account more quickly. It is possible that the respective vehicle in the specific environment acquires further sensor data during self-localization, from which no features are extracted, in order to provide these further sensor data as training data for the machine learning model.It is also possible that the method further comprises the following step:training a machine learning model for extracting features based on the recorded sensor data to provide the machine learning model trained based on the specific environment.It is possible that the machine learning model is continuously updated based on other recorded sensor data to adapt to changes in the specific environment and thus improve the accuracy of the feature extraction.In another example, the method further comprises the step of:creating a map representation of the specific environment based on features of the recorded sensor data extracted by the trained machine learning model.This step is preferably performed by the external data processing device. Expressed in simplified terms, the map representation represents in particular which objects and structures of the specific environment are located at which location, for example in a coordinate system.According to a further possibility, the provision of the localization information comprises the following step:matching the transmitted extracted features with the created map representation in order to ascertain a position and an orientation of the vehicle for the localization information.Expressed simply, it is determined which objects or structures of the specific environment are represented by the extracted features in order to be able to determine at which position and in which orientation the vehicle is located in the specific environment.It may be advantageous if the external data processing device is a cloud server and the transmission of the machine learning model trained on the basis of the specific environment is carried out via a mobile radio network or a local WLAN network of the specific environment.It is possible for the method according to the invention to be used in a vehicle. The vehicle may be designed, for example, as a motor vehicle and / or passenger motor vehicle and / or autonomous vehicle. The vehicle may have a vehicle device, for example for providing an at least partially automated or autonomous driving function, and / or a driver assistance system. The vehicle device can be designed to at least partially automatically control and / or accelerate and / or brake and / or steer the vehicle.The machine learning model is trained in particular for classification and / or object detection. Accordingly, the training can result in a trained machine learning model which can be used for classification and / or object detection. The deployment and thus the inference may be provided in a vehicle, for example. The data points of the input data can be, for example, pixels of image data or can be based thereon, in order to carry out the classification and / or object detection of the data points on the basis of the pixels. The input data can comprise sensor and / or image data which at least partially result from a detection with a sensor, preferably camera sensor, and / or which have been at least partially synthesized, i.e. in particular simulate the real data of a sensor. Specifically, it can be provided that the values of image points, preferably pixels, of the image data represent an environment of a sensor and / or of a vehicle and / or a traffic scene, in particular the specific environment. A classification, preferably image classification and / or object detection, based on these values can be provided. This makes it possible, for example, to detect objects of the traffic scene. The classification can also be provided in the form of a semantic segmentation (i.e. a pixel or region classification) and / or an object detection. The image data can be images of a radar sensor and / or an ultrasonic sensor and / or a LiDAR sensor and / or a thermal imaging camera, for example. Accordingly, the images can also be embodied as radar images and / or ultrasonic images and / or thermal images and / or LiDAR images.The invention likewise relates to a computer program, in particular a computer program product, comprising instructions which, when the computer program is executed by a computer, cause the computer program to execute the method according to the invention. Thus, the computer program according to the invention brings with it the same advantages as have been described in detail with reference to a method according to the invention.The invention likewise relates to a device for data processing which is set up to carry out the method according to the invention. The device can be, for example, a computer which executes the computer program according to the invention. The computer may include at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program can be stored and from which the computer program can be read out by the processor for execution.The invention can likewise be a computer-readable storage medium which has the computer program according to the invention and / or comprises instructions which, when executed by a computer, cause the computer to execute the method according to the invention. The storage medium is embodied, for example, as a data memory such as a hard disk and / or a nonvolatile memory and / or a memory card. The storage medium may be integrated into the computer, for example.In addition, the method according to the invention can also be embodied as a computer-implemented method.Further advantages, features and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can be essential to the invention individually or in any combination. The following are shown: FIG. 1 shows a schematic visualization of a method, a device, a storage medium and a computer program according to exemplary embodiments of the invention, FIG. 2 shows a schematic illustration of a method according to exemplary embodiments of the invention.FIG. 1 schematically illustrates a method 100, a device 10, a storage medium 15 and a computer program 20 according to exemplary embodiments of the invention.FIG. 1 shows in particular an exemplary embodiment of a method 100 for self-localization of a vehicle 1 in a specific environment 2. in a first step 101, the specific environment 2 of the vehicle 1 is detected on the basis of sensor data, wherein the sensor data result from a detection of at least one sensor 3 of the vehicle 1. In a second step 102, a transmission of a machine learning model 4 trained on the basis of the specific environment is initiated from an external data processing device 5 to the vehicle 1. In a third step 103, features 6 in the sensor data are extracted using the trained machine learning model 4. In a fourth step 104, the extracted features 6 are transmitted to the external data processing device 5. In a fifth step 105, localization information 7 is provided by the external data processing device 5, wherein the localization information 7 is ascertained on the basis of the transmitted extracted features 6. In a sixth step 106, the self-localization is carried out on the basis of the provided localization information 7.The present invention describes in particular a video-based method for self-localization of an autonomous vehicle 1, in which parts of the data and the processing thereof are swapped out to an external data processing device 5, such as a server, for example into the cloud. Artificial deep neural networks (deep learning) can be particularly well suited for self-localization, since these can be trained specifically for their robustness with respect to variations in image data. In particular, deep neural networks can be trained (over-fitting) specifically for use in a limited local environment, so that even in low-detail environments, such as a parking lot or parking garage, they can reliably recognize image features of the same physical points, for example a spot in the asphalt or a specific branch of a tree, even under difficult conditions. According to exemplary embodiments, the present invention is based on the fact that such a deep neural network 4, specifically trained for a specific local environment, can be loaded from a server 5 onto a vehicle 1, in particular an autonomous or at least partially automated vehicle 1, as soon as said vehicle is intended to be located in this specific local environment 2, for example because it is intended to park itself automatically (automated value parking).According to exemplary embodiments, one aspect of the invention is a transmission of a location-specific deep neural network 4 from a server 5 to a vehicle 1, in particular an autonomous or at least partially automated vehicle 1, which uses this neural network 4 for extracting image features 6, which in turn are sent to the server 5 for localization.The vehicle 1 preferably uses an integrated sensor 3, such as a camera, for capturing the specific local environment 2. in particular, it uses a specialized deep neural network 4, sent by a server, for image feature extraction from these images and then sends the extracted image features 6 back to the server 5. the server 5 can determine the corresponding image features 6 to the features obtained by the vehicle 1 in a map representation 8 of the local environment and can thus determine the position and orientation of the vehicle 1 in the map coordinate system. This localization information 7 is preferably sent in a last step to the vehicle 1, which can use this to plan automated navigation, for example.This procedure can advantageously be particularly data-saving, in the sense that as little data transmission as possible takes place between vehicle 1 and server 5. At the same time, it can advantageously allow the use of highly specialized deep neural networks 4, so that optimum localization performances can be achieved, since the computational load of the localization is divided between vehicle 1 and server 5.In comparison with the prior art, the present invention thus enables the use of highly specialized, location-specific deep neural networks for image feature extraction, since these are loaded into the system of the vehicle 1 only when required, i.e. for example as soon as a vehicle 1 is intended to be able to locate itself at a specific environment 2. At the same time, the method according to exemplary embodiments can be data-saving, since it is avoided that complete images have to be exchanged between server 5 and vehicle 1. Instead, the (pre)trained deep neural network 4 is transmitted from the server 5 to the vehicle 1 in particular only once, and in subsequent operation the compact extracted image features 6 are transmitted from the vehicle 1 to the server 5, such that these features 6 on the server 5 can be matched to the features stored in a local map representation 8 for the position determination. It should be taken into account that deep neural networks with very compact representations can be used. For example, it may be possible that the size of deep neural networks with powers that can be maintained with the other method of the current state of the art may be brought to significantly below 1 MB, which corresponds approximately to the size of a single image. In addition, this constellation has the advantage that a part of the calculations-namely the map-based localization-can be swapped out to the server for the autonomous driving function of the vehicle, while at the same time the extracted image features are available directly on the vehicle for other functions of the vehicle, for example for obstacle detection or for visual odometry.A method according to an exemplary embodiment proceeds as follows: in a first step, a local environment 2 in which autonomous vehicles 1 are subsequently intended to be able to localize is recorded in a mapping trip, so that said mapping trip is recorded in detail on video from a plurality of viewing angles. In a further step, a deep neural network 4 is trained (overlaid) specifically for the use of image feature extraction of images of this local environment and stored on the server 5. In a further step, a map representation 8 of the recorded local environment 2 is created with the aid of the image features 6 extracted by the deep neural network 4 and is likewise stored on the server 5. In a further step, as soon as a vehicle 1 is now intended to be able to locate itself in the mapped local environment 2, the specialized deep neural network 4 is loaded from the server 5 onto the vehicle 1. This can be done, for example, via the mobile radio network or a local WLAN network. The vehicle 1 is preferably equipped with at least one sensor such as a camera 3 and can use the transmitted deep neural network 4 for image feature extraction on its images. The extracted image features 6 of the part of the local environment 2 currently observed by the vehicle 1 are preferably sent from the vehicle 1 to the server 5. The server 5 then preferably uses the obtained image features 6 for matching with the local map representation 8 in order to determine the position and orientation of the vehicle 1 in the map coordinate system. The localization information 7 can then be sent from the server 5 to the vehicle 1, so that the latter can plan its route, for example, on the basis of this information.In addition to the deep neural network 4, the map representation 8 of the local environment 2 could also be sent from the server 5 to the vehicle 1. Then, the calculation of the own position and orientation with the aid of the extracted image features 6 must preferably be carried out by the vehicle 1 itself, i.e. more effort is incurred for the computer systems integrated into the vehicle 1. If the map representation 8 is small, this procedure can be worthwhile, since the communication effort for the sending of the image features 6 is saved. This is in particular a trade-off between the additional computing load for the vehicle 1 and the possibly saved communication outlay, which could turn out to be different depending on the local environment 2.Instead of a camera, a radar or LiDAR sensor, for example, can also be used in a corresponding manner.If there is sufficient bandwidth at a point in time for sending data, or if the vehicle 1 parks, for example, images previously recorded during operation can also be sent from the vehicle 1 to the server 5. These images can then in turn be used for the further training of the specialized deep neural network 4, so that this can be updated again and again with current recordings of the local environment 2 from the relevant perspective of the vehicles 1 traveling there. This is helpful, for example, in environments that greatly change their visual appearance, for example depending on the season or simply over longer periods of time.A method according to an embodiment is illustrated in FIG. 2. In particular, a interaction between an external data processing device in the form of a server 5 and a vehicle 1 is shown. The server 5 shares an instance of a machine learning model in the form of a deep neural network 4 with the vehicle 1. The vehicle 1 uses the neural network 4 to extract image features 6 from an image of its camera 3, on which a current section of the specific local environment 2 can be seen. The extracted image features 6 are sent to the server 5. The server 5 uses its map representation 8 of the local environment 2 and the image features 6 obtained from the vehicle 1 for determining the position of the vehicle 1. the server 5 sends the localization information 7 to the vehicle 1 in the form of its current position and orientation.The foregoing explanation of the embodiments describes the present invention solely by way of examples. Of course, individual features of the embodiments can be freely combined with one another, insofar as technically expedient, without departing from the scope of the present invention.

Claims

Method (100) for self-locating a vehicle (1) in a specific environment (2), comprising the following steps: - detecting (101) the specific environment (2) of the vehicle (1) on the basis of sensor data, wherein the sensor data result from a detection of at least one sensor (3) of the vehicle (1), - initiating (102) a transmission of a machine learning model (4) trained on the basis of the specific environment from an external data processing device (5) to the vehicle (1), - extracting (103) features (6) in the sensor data using the trained machine learning model (4), - transmitting (104) the extracted features (6) to the external data processing device (5), - providing (105), by the external data processing device (5), locating information (7) being determined on the basis of the transmitted extracted features (6), performing (106) the self-localization on the basis of the provided localization information (7).Method (100) according to Claim 1, characterized in that, as part of the performance (106) of the self-localization, an incremental calculation of a position of the vehicle (1) takes place, wherein at least the steps of extracting (103) the features (6), of transmitting (104) the extracted features (6) and of providing (105) the localization information (7) are performed at defined intervals.The method (100) according to any of the preceding claims, characterized in that the method (100) further comprises the step of: - recording sensor data in the specific environment (2), wherein the recording is performed with at least two different viewpoints to provide the recorded sensor data as training data for the machine learning model (4) trained based on the specific environment.The method (100) according to claim 3, characterized in that the method (100) further comprises the step of: - training a machine learning model for extracting features (6) based on the recorded sensor data to provide the machine learning model (4) trained based on the specific environment.Method (100) according to claim 4, characterized in that the method (100) further comprises the step of: - creating a map representation (8) of the specific environment (2) based on features (6) of the recorded sensor data extracted by the trained machine learning model (4).Method (100) according to claim 5, characterised in that the provision (105) of the locating information (7) comprises the following step: - matching the transmitted extracted features (6) with the created map representation (8) in order to determine a position and an orientation of the vehicle (1) for the locating information (7).Method (100) according to one of the preceding claims, characterized in that the external data processing device (5) is a cloud server and the transmission of the machine learning model (4) trained on the basis of the specific environment is carried out via a mobile radio network or a local WLAN network of the specific environment (2).A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer program to execute the method (100) according to any one of the preceding claims.Data processing device (10) configured to carry out the method (100) according to any one of claims 1 to 7.A computer readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer to carry out the steps of the method (100) of any one of claims 1 to 7.

Images