Method, system, and computer program product for automatically positioning a vehicle

Abstract

A method for automatically positioning a vehicle (10) at a geographical location, having the steps: recording (S10) first images and sensor data (30) of the surroundings of the vehicle (10) by means of a camera and sensor arrangement (20) of the vehicle (10) while the vehicle (10) is driving along a route, assigning geographical coordinates to the images and sensor data (30), sending (S20) the first images and sensor data (30) to a data evaluation unit (50), creating a digital surroundings map (70) by the data evaluation unit (50), recording (S40) second images and sensor data (30) of the surroundings by means of the camera and sensor arrangement (20) of the vehicle (10) while the vehicle (10) is driving along the same route, sending (S50) the second images and sensor data (30) to the data evaluation unit (50), comparing (S60) the second images and sensor data (30) with the digital surroundings map (70) by the data evaluation unit (50), determining (S70) the position of the vehicle (10).

Description

Technical Field

[0001] This invention relates to a method, system, and computer program product for automatically locating vehicles. Background Technology

[0002] The trend towards driver assistance systems (FAS) and highly automated driving functions (HAF) in motor vehicles, as well as air and water vehicles, necessitates comprehensive safeguards. This is because the responsibility for driving a vehicle no longer rests entirely with the driver, but rather with the computer units within the vehicle taking over active functions. Therefore, it is essential to ensure that automatically moving objects exhibit a very low error rate in driving behavior. This also includes the need for functions that provide accurate and up-to-date vehicle positioning at all times.

[0003] For the localization of a single vehicle, processed and abstracted map data can be used. It is known that the vehicle's surroundings are acquired using a camera and the data is compared with existing maps in order to enable localization, in particular by identifying salient objects on the terrain in the vehicle's surroundings.

[0004] However, such positioning becomes difficult when salient objects in the surrounding environment are absent or cannot be seen in camera recordings due to weather conditions. Furthermore, the third-party maps used are not always up-to-date, either because new map data is unavailable or updates require payment. Alternatively, the map data format may not meet the requirements for higher resolution. Additionally, GPS location information may be incorrect, making it impossible for automated driving assistance functions to rely solely on this positioning method.

[0005] US 10,816,654 B1 describes a system for locating ground vehicles or moving objects within their surrounding environment. The system acquires radar map data from a radar system and compares the radar map data with reference map data. The location of the vehicle or the moving object is then determined by matching the radar map data with the reference map data.

[0006] DE 10 2016 224329 A1 describes a system for vehicle positioning, the system having a first sensor unit for determining the relative motion of the vehicle with respect to its surrounding environment and a second sensor unit for acquiring radar data of the surrounding environment.

[0007] DE 10 2017 22169 A1 describes a method for enabling a vehicle to self-localize in its environment. Here, during vehicle operation, the surrounding environment is acquired using environmental sensors, and this environment is compared with a previously created map to determine the vehicle's position on the map.

[0008] DE 10 2019 133703 A1 describes a method for locating a vehicle. Here, radar data is received by the vehicle's radar, a local map is created based on the radar data, and the local map is compared with a invoked map to determine the vehicle's position. Summary of the Invention

[0009] Therefore, the present invention is based on the objective of providing a method, system, and computer program product for automatically locating vehicles, which can achieve accurate positioning of vehicles along a route and is characterized by a high degree of safety and reliability.

[0010] This objective is achieved in terms of method by the features of claim 1, in terms of system by the features of claim 12, and in terms of computer program product by the features of claim 15. The other claims relate to preferred embodiments of the invention.

[0011] According to a first aspect, the present invention relates to a method for automatically locating a vehicle at a geographic location. The method includes the following steps:

[0012] - As the vehicle travels along a certain route, first images and sensor data of the vehicle's surrounding environment are recorded using the vehicle's cameras and sensor devices, wherein geographic coordinates are assigned to the images and sensor data accordingly.

[0013] - Send the recorded first image and sensor data to the data evaluation unit;

[0014] - A digital environment map is created by the data evaluation unit;

[0015] - As the vehicle travels along the same route, a second image and sensor data of the surrounding environment are recorded by means of the vehicle's camera and sensor devices;

[0016] - Send the recorded second image and sensor data to the data evaluation unit;

[0017] - The data evaluation unit compares the recorded second image and sensor data with the digital environment map;

[0018] - Determine the vehicle's location.

[0019] One proposed improvement involves a second vehicle using a second camera and sensor devices to record second images and sensor data.

[0020] In another embodiment, a digital environment map is updated using second image and sensor data.

[0021] In one advantageous embodiment, the camera and sensor device includes a radar system.

[0022] In particular, the camera and sensor device includes a LiDAR (Light detection and ranging) system, and / or an ultrasonic system, and / or at least one RGB camera in the visible light range, and / or at least one UV camera in the ultraviolet wavelength range, and / or at least one IR camera in the infrared range.

[0023] In another embodiment, the digital environment map is designed as a three-dimensional 3D map and is based on radar mapping.

[0024] Advantageously, digital environment maps are designed as semantic graphs with nodes and graph edges, where graph edges form connections between nodes.

[0025] In particular, create sub-environment maps for semantic categories that can be related to each other.

[0026] One proposed improvement involves a data evaluation unit that uses artificial intelligence algorithms to process and evaluate image and sensor data, as well as to create digital environment maps.

[0027] In particular, it is noted that artificial intelligence algorithms include neural networks, especially convolutional neural networks.

[0028] In another embodiment, it is proposed to compare and combine the environmental maps of the vehicles in the fleet, in particular using server and / or cloud computing infrastructure 90.

[0029] According to a second aspect, the present invention relates to a system for automatically locating a vehicle at a geographic location. The system includes cameras and sensor devices arranged on the vehicle, as well as a data evaluation unit. The system is designed to implement the method according to the first aspect.

[0030] In one improved embodiment, the camera and sensor device includes a radar system, and / or a LiDAR (light detection and ranging) system, and / or an ultrasonic system, and / or at least one RGB camera in the visible light range, and / or at least one UV camera in the ultraviolet wavelength range, and / or at least one IR camera in the infrared wavelength range.

[0031] In one implementation, the digital environment map is designed as a three-dimensional 3D map and is based on radar mapping.

[0032] In particular, the digital environment map is designed as a semantic graph with nodes and graph edges, where the graph edges constitute the connections between nodes.

[0033] According to a third aspect, the present invention provides a computer program product comprising executable program code that implements the method according to the first aspect. Attached Figure Description

[0034] The invention will now be described in more detail with the aid of embodiments shown in the accompanying drawings.

[0035] In the attached diagram:

[0036] Figure 1 A block diagram illustrating an embodiment of the system according to the present invention is shown;

[0037] Figure 2 A flowchart illustrating the various method steps of the method according to the present invention is shown;

[0038] Figure 3 A computer program product according to an embodiment of the third aspect of the present invention is illustrated schematically. Detailed Implementation

[0039] Additional features, aspects, and advantages of the invention or its embodiments can be seen from the detailed description of the claims.

[0040] Driver assistance systems intervene semi-automatically or automatically in the vehicle's drives, controllers, or signaling devices, or warn the driver shortly before or during a critical situation via an appropriate human-machine interface. In this context, the vehicle's positioning within its surrounding environment is crucial for the functionality of multiple driver assistance systems (FAS) and highly automated driving functions (HAF).

[0041] Figure 1 A system 100 according to the invention is shown for determining the position of a vehicle 10 moving along a certain route. The moving vehicle 10 can be a motor vehicle such as a passenger vehicle or a freight vehicle, an air vehicle or a water vehicle, wherein the air vehicle is in particular an unmanned air vehicle (drone), and the water vehicle is in particular a ship, sailboat and ferry. The obtained position of the vehicle can be used in particular for driver assistance systems (FAS) and highly automated driving functions (HAF).

[0042] Vehicle 10 includes a camera and sensor device 20 for acquiring images and sensor data 30 of the vehicle 10's surrounding environment. The camera and sensor device 20 particularly includes a radar system 22, which in turn may include multiple radar sensors. Furthermore, the camera and sensor device 20 may have a LiDAR (Light Detection and Ranging) system 23 and an ultrasonic system 24 to, for example, measure the distance to and speed of a vehicle traveling ahead.

[0043] Furthermore, the camera and sensor device 20 may include at least one RGB camera 25 covering the visible light range with the primary colors "blue, green, and red". It may also additionally include at least one UV camera 26 covering the ultraviolet wavelength range and / or at least one IR camera 27 covering the infrared wavelength range. Thus, cameras 25, 26, and 27, with their different absorption spectra, can represent different lighting and weather conditions in the recording area. In particular, cameras 25, 26, and 27 can be arranged on the vehicle 10 in this way to achieve 360° panoramic image display in all spatial directions. A stereo optical camera system may also be provided. In addition, the camera and sensor device 20 is equipped with an acoustic sensor 28 for acquiring acoustic signals using a microphone. This allows the recording of tire rolling noise or engine noise, which can characterize specific geographical areas, such as specific lane surfaces.

[0044] In addition, a GPS connection (Global Positioning System) 40 is advantageously provided to acquire geographic location using a relatively coarse resolution and assign that geographic location to the recorded image and sensor data 30.

[0045] The recording frequency of the camera and sensor device 20 is designed for the relatively high speed of the vehicle 10 and is capable of recording images and sensor data 30 at a high data recording frequency. Furthermore, it can be proposed that when a large, significant change occurs in the recording area of ​​the camera and sensor device 20, such as when a prominent object like a building appears in the recording area of ​​the camera and sensor device 30, the camera and sensor device 20 automatically initiates the data recording process. This allows for selective data acquisition, and the data processing unit 50 processes only the relevant image and sensor data 30. This enables more efficient use of computing power.

[0046] The camera and sensor device 20 transmits the recorded images and sensor data 30 to the data processing unit 50 for further processing. Preferably, the images and sensor data 30 recorded by the camera and sensor device 20 are transmitted to the data processing unit 50 via a wireless mobile communication connection.

[0047] The data processing unit 50 preferably includes a processor 51 that processes image and sensor data 30. The processor 51, or another processor, is also designed to control the camera and sensor device 30. The data processing unit 50 and processor 51 may be integrated into the vehicle 10 and / or into the camera and sensor device 50, or designed as a cloud-based solution with a cloud computing infrastructure 90 connected to the vehicle 10 via a mobile communication connection. However, it is also conceivable that the image and sensor data 30 may first be stored in a memory unit 55 for subsequent processing by the data processing unit 50.

[0048] For the purposes of this invention, "processor" can be understood, for example, as a machine, an electronic circuit, or a powerful computer. A processor can be, in particular, a central processing unit (CPU), a microprocessor, or a microcontroller, such as an application-specific integrated circuit (ASIC) or a digital signal processor, which may be combined with memory units for storing program instructions. A processor can also be understood as a virtual processor, a virtual machine, or a soft-core CPU. A processor can also be, for example, a programmable processor equipped with configuration steps for implementing the methods according to the invention, or configured by means of configuration steps to enable the programmable processor to implement the methods, components, modules, or other aspects and / or parts of the features of the invention according to the invention. Furthermore, highly parallel computing units and powerful graphics modules may be provided.

[0049] For the purposes of this invention, "memory cell" or "memory module" can be understood, for example, as volatile memory (RAM) or permanent memory in the form of working memory, such as a hard disk or data carrier, or, for example, a replaceable memory module. However, a memory module can also be a cloud-based storage solution.

[0050] For the purposes of this invention, "module" can be understood, for example, as a processor and / or a memory unit for storing program instructions. For instance, a module is specifically configured to implement program instructions, thereby causing the module to perform or implement the method or method steps according to the invention.

[0051] For the purposes of this invention, the recorded image and sensor data 30 can be understood as raw data as well as prepared recording results from the camera and sensor device 20. The data format of the image and sensor data 30 is preferably designed as a Tensor. However, other data formats may also be used.

[0052] In particular, the camera and sensor device 20 can have a 5G standard mobile communication module. 5G is the fifth-generation mobile communication standard, and compared to the 4G mobile communication standard, its advantages include: higher data rates, up to 10 Gbit / sec; utilization of a wider frequency range, such as 2100, 2600, or 3600 MHz; increased frequency capacity and therefore increased data throughput; and real-time data transmission, as up to one million devices can respond simultaneously per square kilometer. Latency is from a few milliseconds to less than 1 ms, enabling real-time transmission of data and computation results. The images and sensor data 30 acquired by the camera and sensor device 20 are sent in real-time to the cloud computing infrastructure 90, where corresponding analysis and computation are performed. The analysis and computation results, along with the location determination, are sent back in real-time to the camera and sensor device 20 or another control device of the vehicle 10, and thus the analysis and computation results, along with the location determination, can be quickly integrated into operating instructions for the driver or into automated driving functions. This speed of data transmission is necessary when using a cloud-based solution to process the images and sensor data 30. The advantage offered by cloud-based solutions is their high and therefore fast computing performance.

[0053] Furthermore, the data processing unit 50 can access one or more additional databases 57. Databases 57 may store, for example, extraction parameters (e.g., specific object or road features used to analyze the recorded images and sensor data 30) or other image and / or feature parameters. Additionally, databases 57 may store target parameters and target values ​​that define safety standards. Furthermore, a user interface 80 may be provided for inputting other data or for displaying an environmental map 70 created by the data processing unit 50. The user interface 80 is particularly designed as a touchscreen display. This display is especially relevant to the navigation system of the vehicle 10. However, it is also conceivable that a user's mobile terminal device, such as a smartphone, may be used alone or additionally as an output module of the user interface 80. It is also conceivable that a computer may be used as an output module, for example, in a testing and research center.

[0054] The data processing unit 50 or processor 51 creates a digital environment map 70 based on the recorded image and sensor data 30. For this purpose, the data processing unit 50 or processor 51 uses artificial intelligence algorithms such as machine learning. In particular, an artificial neural network can be provided to extract relevant image features from the acquired image and sensor data 30. Convolutional neural networks can be used, in particular, as the neural network.

[0055] Neural networks are composed of neurons arranged in multiple layers and interconnected in various ways. A neuron receives information from external sources or other neurons at its input, evaluates the information in a specific way, and forwards it in a modified form to another neuron at its output or outputs it as the final result. Hidden neurons are arranged between the input and output neurons. Depending on the network type, there can be multiple layers of hidden neurons. These hidden neurons are responsible for forwarding and processing information. Finally, the output neurons provide the result and output it to the outside world. Various types of neural networks are created by arranging and connecting neurons, such as feedforward networks, recurrent networks, or convolutional neural networks. Networks can be trained using unsupervised or supervised learning.

[0056] Convolutional neural networks (CNNs) are a special type of artificial neural network. CNNs have multiple convolutional layers and are well-suited for machine learning and artificial intelligence applications (KünstlicherIntelligenz, KI) in the fields of image and speech recognition.

[0057] The digital environment map 70 of the vehicle 10 created by the data processing unit 50 is specifically designed as a three-dimensional 3D map. Information in three-dimensional space can be obtained, in particular, using the radar system 22, and this information can be stored as a three-dimensional representation of the surrounding environment in the form of a point cloud. The advantage of the radar mapping method is that 3D points can be used as low-level signals to characterize the surrounding environment.

[0058] Furthermore, a semantic graph is used to display and store the environmental map 70. Here, the selected image and sensor data 30, as well as the radar measurement data 32 obtained, particularly by the radar system 22, are combined at specific nodes 72 of the semantic graph. The connections between nodes 72 are called graph edges 74 and specifically illustrate the semantic and physical relationships between the various nodes 72 of the semantic graph. The semantic graph provides a basis for efficiently achieving optimization objectives through changes in the graph edges 74. Thus, map accuracy can be improved while taking into account the physical accuracy of the measurement results from the camera and sensor devices 20. In particular, the creation of the environmental map 70 can be designed as an iterative process, in which the accuracy and resolution of the environmental map 70 are improved based on newly acquired image and sensor data 30 each time the route is traveled. If, for example, the area of ​​the route has changed, the route is updated in the environmental map 70 or additional conditions are formulated for the map. Thus, for example, the outline of a parking lot can be identified based on ground markings in a parking garage and drawn on the environmental map 70. Furthermore, the status of the parking lot can be indicated, such as recording the status of "available" or "full" on the map. Alternatively, areas in environment map 70 that were previously obscured by parked vehicles during the drive into the parking garage could be added.

[0059] Compared to known distance measurement methods such as LiDAR or ultrasonic methods, displaying the environment map 70 as a semantic map can significantly improve positioning accuracy. Furthermore, because the environment map 70 is stored as a 3D point cloud, efficient use of storage space can be achieved.

[0060] Furthermore, the 3D points of the digital environment map 70 can be equipped with additional attributes and features, such as specific semantic categories for specific objects in the surrounding environment. Therefore, in addition to the 3D representation of the surrounding environment on the environment map 70, each point in the 3D point cloud is annotated with semantic information. This could include, for example, vehicle type, specific building type, vegetation type, pedestrian overpass, streetlight installation, etc. With the additional semantic information in the environment map 70, areas in the surrounding environment that change relatively quickly over time can be safely and accurately identified, such as vegetation types that change seasonally. Based on these identified changes, the environment map 70 can then be adapted and updated.

[0061] In particular, the semantic information of the environmental map 70 can be used to generate sub-environment maps 72, which contain semantic information of at least one specific semantic category. Semantic categories could be, for example, buildings, factory facilities, bridges, highway sections, etc. Sub-environment maps 72, each specifically including one or more semantic categories, can then be combined. In particular, semantic categories can be classified according to their temporal change potential; for example, buildings that remain stable over a longer period differ from vegetation that changes with the seasons.

[0062] To better identify semantic categories, semantic graphs can include multiple layers. Specifically, there is a first layer called a "single edge," a second layer called a "skip edge," and a third layer called a "loop closure." These layers can be distinguished from each other by using different weights for various semantic categories or different algorithms.

[0063] Furthermore, it can be proposed that the data processing unit 40 can also access image and sensor data 30 recorded by other vehicles 10 in the convoy and use it to create an environmental map 70. To this end, each vehicle 30 in the convoy is equipped with mutually compatible image and sensor devices 30, and can exchange, compare, combine, and correlate their respective graph-based environmental maps 70 created independently by each vehicle 10. This allows for the generation of a holistic environmental map 70 with improved resolution and accuracy when multiple vehicles 10 repeatedly travel a certain route. Because the environmental map 70 is represented as a graph, individual map parts and specific map points can be correlated with each other using appropriate auxiliary conditions. Optimization and processing of sub-maps of the environmental map 70 can be performed in the data processing unit 40 of the vehicle 10 or externally to the vehicle 10, such as on a backend server or cloud computing infrastructure 90. Therefore, all vehicles 10 in the convoy can be interconnected via a backend server or cloud computing infrastructure 90.

[0064] An exemplary application of the invention is parking in a trained manner at locations where GPS functionality is unavailable (e.g., in a specific parking garage). According to the invention, the driver of vehicle 10 drives a specific route as a training drive. Here, an environmental map 70 is created by data processing unit 50 based on this training drive, in which the driving trajectory is stored. The 3D points stored on the environmental map 70 can now be used when driving the route again to locate the position of vehicle 10 on the map using the latest images acquired by camera and sensor device 20 and sensor data 30.

[0065] exist Figure 2 The method steps according to the present invention are shown in the figure.

[0066] In step S10, while the vehicle 10 is traveling along a certain route, the vehicle's camera and sensor device 20 records first images and sensor data 30 of the vehicle 10's surrounding environment, wherein geographical coordinates are assigned to the images and sensor data 30 accordingly.

[0067] In step S20, the recorded first image and sensor data 30 are sent to the data evaluation unit 50.

[0068] In step S30, the data evaluation unit 50 creates a digital environment map 70.

[0069] In step S40, while the vehicle 10 is traveling along the same route, the camera and sensor device 20 records a second image and sensor data 30 of the surrounding environment of the vehicle 10.

[0070] In step S50, the recorded second image and sensor data 30 are sent to the data evaluation unit 50.

[0071] In step S60, the data evaluation unit 50 compares the recorded second image and sensor data 30 with the digital environment map 70.

[0072] In step S70, the position of vehicle 10 is determined.

[0073] Figure 3 A computer program product 200 is schematically shown, which includes executable program code 250 configured to implement the method according to the first aspect of the invention.

[0074] The method according to the present invention allows for better positioning of vehicles. This is particularly important for autonomous driving functions based on accurate knowledge of the vehicle's geographical location.

[0075] List of reference numerals

[0076] 10 vehicles

[0077] 20 Camera and sensor devices

[0078] 22 Radar Systems

[0079] 23 LiDAR systems

[0080] 24. Ultrasonic System

[0081] 25 RGB Camera

[0082] 26 UV camera

[0083] 27 IR camera

[0084] 30 Image and sensor data

[0085] 40 GPS connection

[0086] 50 Data Evaluation Units

[0087] 51 processor

[0088] 55 memory units

[0089] 57 Database

[0090] 70 Environmental Map

[0091] 90 Cloud computing infrastructure

[0092] 100 System

[0093] 200 computer program products

[0094] 250 Program Code

Claims

1. A method for automatically locating a vehicle (10) at a geographic location, the method comprising the following steps: - As the vehicle (10) travels along a certain route, the vehicle (10) records (S10) first images and sensor data (30) of the surrounding environment of the vehicle (10) by means of the camera and sensor device (20) of the vehicle (10), wherein geographical coordinates are assigned to the images and sensor data (30) accordingly. - The recorded first image and sensor data (30) are sent (S20) to the data evaluation unit (50); - A digital environment map (70) is created by the data evaluation unit (50); - While the vehicle (10) is traveling along the same route, the camera and sensor device (20) of the vehicle (10) records (S40) a second image and sensor data (30) of the surrounding environment; - The recorded second image and sensor data (30) are sent (S50) to the data evaluation unit (50); - The data evaluation unit (50) compares the recorded second image and sensor data (30) with the digital environment map (70) (S60); - Determine the position of the vehicle (10) as described in (S70); - The analysis and calculation results, as well as the determination of the position, are sent back in real time to the camera and sensor device (20) or another control device of the vehicle (10), so that the analysis and calculation results, as well as the determination of the position, are incorporated into the operation instructions given to the driver or into the automated driving function.

2. The method according to claim 1, wherein, The second vehicle (10) uses a second camera and sensor device (20) to record a second image and sensor data (30).

3. The method according to claim 1 or 2, wherein, The digital environment map (70) is updated with second image and sensor data (30).

4. The method according to any one of claims 1 to 3, wherein, The camera and sensor device (20) includes a radar system (22).

5. The method according to claim 4, wherein, The camera and sensor device includes a LiDAR (light detection and ranging) system (23), and / or an ultrasonic system (24), and / or at least one RGB camera (25) in the visible light range, and / or at least one UV camera (26) in the ultraviolet wavelength range, and / or at least one IR camera (27) in the infrared wavelength range.

6. The method according to any one of claims 1 to 5, wherein, The digital environment map (70) is designed as a three-dimensional 3D map and is based on radar mapping.

7. The method according to any one of claims 1 to 6, wherein, The digital environment map (70) is designed as a semantic graph with nodes (72) and graph edges (74), wherein the graph edges (74) form connections between the nodes (72).

8. The method according to claim 7, wherein, Create sub-environment maps for semantic categories that can be related to each other (70).

9. The method according to any one of claims 1 to 8, wherein, The data evaluation unit (50) uses artificial intelligence algorithms to process and evaluate the image and sensor data (30) and to create the digital environment map (70).

10. The method according to claim 9, wherein, The artificial intelligence algorithms include neural networks, especially convolutional neural networks.

11. The method according to any one of claims 1 to 10, wherein, The environmental maps (70) of the vehicles (10) of the convoy are compared and combined with each other, which in particular utilizes server and / or cloud computing infrastructure (90).

12. A system (100) for automatically locating a vehicle (10) at a geographic location, the system having a camera and sensor device (20) arranged on the vehicle (10) and a data evaluation unit (50), wherein, The system is designed to implement the method according to any one of claims 1 to 3.

13. The system (100) according to claim 12, wherein, The camera and sensor device (20) includes a radar system (22), and / or a LiDAR (light detection and ranging) system (23), and / or an ultrasonic system (24), and / or at least one RGB camera (25) in the visible light range, and / or at least one UV camera (26) in the ultraviolet wavelength range, and / or at least one IR camera (27) in the infrared range.

14. The system (100) according to claim 12 or 13, wherein, The digital environment map (70) is designed as a three-dimensional 3D map and is based on radar mapping, and the digital environment map (70) is designed as a semantic graph with nodes (72) and graph edges (74), wherein the graph edges (74) form connections between the nodes (72).

15. A computer program product (200) having executable program code (250) configured to implement the method according to claim 1 or 2.

Images