Drive device, vehicle, and method for autonomous driving and / or assisted driving

Abstract

The present invention relates to a vehicle, the vehicle comprising: a drivable road detection unit (32) configured to receive map data from a storage device (33) and to receive localization data from a localization device (22) and configured to create a first occupancy grid, each cell of which represents a first degree of confidence that the surrounding environment is drivable, based on the map data and / or the localization data; an optical sensor-based drivable road detection unit (32c) configured to receive image data from an optical sensing device (20) and configured to create a second occupancy grid, each cell of which represents a second degree of confidence that the surrounding environment is drivable, based on the image data; and a fusion unit (32d) configured to create a third occupancy grid, each cell of which represents a third degree of confidence that the surrounding environment is drivable, by fusing the first occupancy grid and the second occupancy grid.

Description

Technical Field

[0001] The present invention relates to a drive device for automatic driving and / or assisted driving of a vehicle. The drive device comprises a storage device configured to store map data, a positioning input port configured to receive positioning data of the vehicle, and an optical input port configured to receive image data and / or geometric data indicating the surroundings of the vehicle.

[0002] The present invention further relates to a vehicle, which comprises a storage device configured to store map data, a positioning device configured to output positioning data of the vehicle, and an optical detection device configured to output image data and / or geometric data indicating the surroundings of the vehicle.

[0003] The present invention also relates to a computer-implemented method for driving a vehicle in an automatic mode and / or a driving assistance mode. The method comprises the steps of generating positioning data of the vehicle using a positioning device, generating image data and / or geometric data indicating the surroundings of the vehicle using an optical detection device, and receiving map data from a storage device and receiving positioning data from a positioning device.

Background Art

[0004] Vehicles operating in autonomous mode (e.g., driverless) or semi-autonomous mode (using driving assistance) free the driver from some driving-related tasks. When operating in autonomous mode, the vehicle can navigate to various locations, enabling the vehicle to travel with minimal human interaction or, in some cases, without passengers. Similarly, in the assisted driving mode, some of the driver's tasks are performed by the driver assistance system.

[0005] Vehicles that operate autonomously or semi - autonomously are typically navigated based on routes provided by route and map services. The configuration of roads and lanes within the roads is important when planning a route for a vehicle. Therefore, the accuracy of maps is very important. However, in some cases, due to various factors such as map - making errors, road damage, or new road construction, the road boundaries may be different from those obtained from the map. Such a discrepancy between the road according to the map and the actual road conditions can cause errors in vehicle planning and control.

[0006] U.S. Patent Application Publication No. 2019 / 0078896 discloses a data - driven map update system for autonomous vehicles. U.S. Patent Application Publication No. 2017 / 0297571 relates to methods and apparatuses for monitoring and adapting the performance of an autonomous vehicle's fusion system. U.S. Patent Application Publication No. 2019 / 0384304 discloses route detection for autonomous machines using deep neural networks. U.S. Patent Application Publication No. 2020 / 0160068 relates to automatically detecting drivable road surfaces for which mapping for autonomous vehicles has not been done. U.S. Patent Application Publication No. 2016 / 0061612 discloses apparatuses and methods for recognizing the driving environment of autonomous vehicles. U.S. Patent Application Publication No. 2020 / 0183011 relates to a method for creating an occupancy grid map.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

[0008] An object of the present invention is to provide a drive device, a vehicle, and a computer-implemented method for predicting the boundary of a drivable road so as to be able to plan a safe and comfortable route. [Means for Solving the Problems]

[0009] This object is solved by the subject matter of the independent claims. The dependent claims describe preferred embodiments of the invention.

[0010] A drive device for automatic driving and / or assisted driving of a vehicle includes a storage device configured to store map data, a positioning input port configured to receive positioning data of the vehicle, an optical input port configured to receive image data and / or geometric data indicating the surroundings of the vehicle, and a drivable road detection unit. The drivable road detection unit includes a map-based drivable road detection unit, an optical sensor-based drivable road detection unit, and a fusion unit. The map-based drivable road detection unit is configured to receive the map data from the storage device and the positioning data from the positioning input port. The map-based drivable road detection unit is further configured to create a first occupancy grid in which each cell represents a first reliability that the surrounding environment is drivable based on the map data and / or the positioning data. The optical sensor-based drivable road detection unit is configured to receive the image data and / or the geometric data from the optical input port and to create a second occupancy grid in which each cell represents a second reliability that the surrounding environment is drivable based on the image data and / or the geometric data. The fusion unit is configured to create a third occupancy grid in which each cell represents a third reliability that the surrounding environment is drivable by fusing the first occupancy grid and the second occupancy grid.

[0011] Optionally, the drive device includes a control unit configured to generate a drive signal for automatic driving and / or assisted driving based on the third occupancy grid, and the drive signal is output to the vehicle for control purposes.

[0012] The vehicle includes a storage device configured to store map data, a positioning device configured to output positioning data of the vehicle, an optical detection device configured to output image data and / or geometric data indicating the surroundings of the vehicle, and a drivable road detection unit. The drivable road detection unit includes a map-based drivable road detection unit, an optical sensor-based drivable road detection unit, and a fusion unit. The map-based drivable road detection unit is configured to receive the map data from the storage device and receive the positioning data from the positioning device. The map-based drivable road detection unit is further configured to create a first occupancy grid in which each cell represents a first reliability that the surrounding environment is drivable based on the map data and / or the positioning data. The optical sensor-based drivable road detection unit is configured to receive the image data and / or the geometric data from the optical detection device and create a second occupancy grid in which each cell represents a second reliability that the surrounding environment is drivable based on the image data and / or the geometric data. The fusion unit is configured to create a third occupancy grid in which each cell represents a third reliability that the surrounding environment is drivable by fusing the first occupancy grid and the second occupancy grid.

[0013] Optionally, the vehicle includes a control unit configured to drive the vehicle in an autonomous driving mode and / or an assisted driving mode based on the third occupancy grid.

[0014] The present invention is based on a general technical idea of fusing real-time AI (Artificial Intelligence) and a map-based approach in order to enhance the accuracy, precision, redundancy, and / or safety of a drivable road identification system. In other words, there is provided a system for redundantly / securely recognizing a drivable road, which includes a unit for fusing information from high-definition lane map data and data generated by real-time semantic segmentation of drivable roads from data regarding the surroundings of a vehicle. According to the present invention, a drivable road can be predicted with high accuracy and high precision.

[0015] The vehicle and / or the drive device of the vehicle may be an autonomous or self-driving vehicle, sometimes called a robocar. Alternatively or additionally, the vehicle and / or the drive device may be a semi-autonomous vehicle. Thus, the drive device may be regarded as a control device of an advanced driver assistance system. According to a classification system having six levels issued in 2021 by SAE International, an automotive standards organization, as J3016_202104 "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles", an autonomous vehicle may be regarded as level 4 or level 5, and a semi-autonomous or assisted driving vehicle may be regarded as levels 1 to 3.

[0016] The vehicle can be any type of self-propelled automobile, preferably configured to travel on a road. For example, the vehicle includes an engine and / or an electric motor for driving the wheels of the vehicle. However, the present invention is not limited to vehicles traveling on the ground. The vehicle can be a marine vehicle such as a boat or a ship. Generally, the present invention relates to vehicles that need to be navigated in real time in a route / lane while avoiding obstacles.

[0017] The memory device may include one or more memories that can be implemented via a plurality of memory devices to provide a memory of a given capacity. The memory device may include one or more volatile memory (or memory) devices, such as random access memory (RAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), or other types of memory devices. The memory device may also include a solid state device (SSD). However, in other embodiments, the memory device functions as an SSD cache that enables non-volatile storage of context states and other such information during a power-off event so that fast power-on can occur upon resumption of system activity, and may include a hard disk drive (HDD) with or without a smaller capacity SSD storage.

[0018] The memory device can be configured to store any type of information or map data. The map data is data that can reconstruct a map around the vehicle. The map data can be updated periodically or intermittently. For this purpose, the memory device can be electronically connected or coupled to a communication unit that enables wired or wireless communication with a network, and thus with a server, with other types of memory devices external to the vehicle, and / or with other vehicles.

[0019] The communication unit may be regarded as a network interface device that can include a wireless transceiver and / or a network interface card (NIC). The wireless transceiver may be a WiFi transceiver, an infrared transceiver, a Bluetooth transceiver, a WiMax transceiver, a wireless cellular phone transceiver, or other radio frequency (RF) transceiver, or a combination thereof.

[0020] The location identification device may be a satellite transceiver (e.g., a Global Positioning System (GPS) transceiver) for determining the current position of the vehicle. The location identification device may include a Visual Positioning System (VPS) that analyzes the surrounding images, compares the images with the data bank images, and determines the position of the user taking the images. The current position of the vehicle is processed and / or output as vehicle position identification data by the location identification device. Therefore, the current position of the vehicle can be determined based on the position identification data. In other words, the position identification data includes information indicating the current position of the vehicle.

[0021] The optical detection device may use electromagnetic radiation in various wavelength ranges such as the visible wavelength range and / or the radio frequency (RF) wavelength range to sense and / or explore the surroundings of the vehicle. The optical detection device unit may be configured to detect and / or emit electromagnetic radiation in a single wavelength range or multiple wavelength ranges. The optical detection device may be a sensor unit for detecting and / or emitting electromagnetic radiation using optical means.

[0022] The optical detection device may include sensors that can determine the surroundings of the vehicle in three dimensions. The optical detection device may include multiple sensors for expanding the field of view by adding / merging data generated by the multiple sensors. For example, the optical detection device may include a monocular camera and / or a stereo camera, i.e., two cameras spaced apart from each other to obtain a stereo image of the surroundings of the vehicle. The camera may be a still camera and / or a video camera. The camera may be mechanically movable, for example, by attaching the camera to a rotating and / or tilting platform. The camera can generate image data.

[0023] However, other imaging or scanning techniques using electromagnetic radiation may be used and may in turn form part of the optical sensing device. The optical sensing device may alternatively or additionally include a radar device and / or a light detection and ranging (LiDAR) device. The LiDAR device can sense objects around the vehicle using lasers. The LiDAR device can include, among other system components, one or more laser sources, a laser scanner, and one or more detectors. The radar device can be a system that utilizes radio signals to sense objects within the local environment of the vehicle. In some embodiments, in addition to sensing objects, the radar unit can further sense the speed and / or direction of travel of other objects such as other vehicles on the road. The LiDAR device and / or the radar device can generate geometric data.

[0024] The map-based drivable road detection unit is electronically and / or communicatively connected or coupled to a storage device and / or a positioning device. The map-based drivable road detection unit is configured to receive map data from the storage device and / or positioning data from the positioning device. The drivable road detection unit, the map-based drivable road detection unit, the optical sensor-based drivable road detection unit, and / or the fusion unit may be part of a computer or processor that performs the tasks outlined below. The computer includes the hardware (e.g., processor, memory, storage) and software (e.g., operating system, program) necessary to perform the tasks outlined below.

[0025] The map-based drivable road detection unit is configured to generate, create, and / or calculate a grid representing the area around the vehicle based on the map data and the positioning data. Thus, the map-based drivable road detection unit processes the map data and the positioning data. The positioning data indicates at which point within the map represented by the map data the vehicle is located. The grid indicates the area around the vehicle.

[0026] The grid may be a first occupancy grid having or composed of a plurality of cells. Each cell corresponds to a respective area around the vehicle. Each cell of the grid is associated with a first confidence level indicating the likelihood that the area represented by the cell is drivable. The confidence level may be regarded as a confidence level. The combination of the grid and the respective confidence levels or confidence levels for each cell may be regarded as a first occupancy grid. When the vehicle is moving, since the position of the vehicle changes and the positioning data changes, the map-based drivable road detection unit needs to update the first occupancy grid.

[0027] The first and second occupancy grids can be representations around the vehicle, can include a plurality of cells, and each of the plurality of cells is associated with a respective confidence level that the surrounding environment is drivable. Thus, the confidence level includes information on whether the area corresponding to each cell of the grid is occupied. This is because the fact that an area in the real world is unoccupied is a prerequisite for the area to be drivable. The confidence level can be a value between a minimum value (e.g., 0 or 0%) and a maximum value (e.g., 1 or 100%).

[0028] The higher the confidence level of each cell, the higher the likelihood that the area corresponding to each cell of the grid is drivable. For example, a confidence level of 0 indicates that the area corresponding to each cell is not drivable, and a confidence level of 1 indicates that each area in the real world is definitely drivable.

[0029] The optical sensor-based drivable road detection unit may be part of the drivable road detection unit and can be electronically and / or communicatively connected or coupled to an optical detection device so as to be able to receive image data and / or geometric data. The optical sensor-based drivable road detection unit may also be part of a computer as described above. The optical sensor-based drivable road detection unit is configured to generate, create, and / or calculate a grid representing the surroundings of the vehicle based on the image data and / or geometric data. Similar to the grid generated by the map-based drivable road detection unit, the grid includes a plurality of cells representing corresponding regions of the real world. Each cell of the grid is associated with a second confidence level indicating whether the region represented by the cell is drivable. The combination of the grid and the respective confidence levels for each cell may be regarded as a second occupancy grid.

[0030] The optical sensor-based drivable road detection unit may include a deep learning solution (for example, using a deep neural network (DNN) such as a convolutional neural network (CNN)). The neural network or artificial intelligence (AI) is used to identify drivable roads or parts of the surroundings of the drivable vehicle based on the image data and / or geometric data. The optical sensor-based drivable road detection unit may be configured to identify other vehicles or objects on the road. These functions of the optical sensor-based drivable road detection unit may be trained.

[0031] The second confidence level may be calculated based on the characteristics or capabilities of the neural network for identifying drivable roads, or may be generated by other methods. For example, when the image data is recognized as being of low quality, such as when the vehicle is surrounded by fog or other environmental conditions that degrade the quality of the image data, the optical sensor-based drivable road detection unit is configured to lower the confidence level for each cell. However, other criteria for evaluating the confidence level of the identification of drivable roads may also be applied.

[0032] The grid generated from the image data and / or geometric data may be composed of a plurality of cells. Each cell corresponds to a respective area around the vehicle. Each cell of the grid is associated (by the method described above) with a second confidence level indicating the possibility that the area represented by the cell is drivable. The combination of the grid and the respective confidence levels for each cell may be regarded as a second occupancy grid. When the vehicle is moving, since the position of the vehicle changes and the image data and / or geometric data change, the optical sensor-based drivable road detection unit needs to update the second occupancy grid.

[0033] The fusion unit may be part of the drivable road detection unit and may be electronically and / or communicably connected or coupled to the map-based drivable road detection unit and the optical sensor-based drivable road detection unit to receive the first occupancy grid and the second occupancy grid. The fusion unit may be part of or a functional unit of the computer described above.

[0034] The fusion unit is configured to fuse the first occupancy grid and the second occupancy grid to create a new third occupancy grid. This fusion process corresponds to the processing of the first occupancy grid and the second occupancy grid. In particular, the first confidence level of each cell of the first occupancy grid can be associated with the second confidence level of the corresponding cell of the second occupancy grid and then fused. However, the fusion step is not limited to this. The first confidence level of each cell of the first occupancy grid can be associated with the second confidence levels of a plurality of corresponding cells of the second occupancy grid and then fused, or vice versa. Also, the first confidence levels or first confidence levels of a plurality of cells of the first occupancy grid can be associated with the second confidence levels or second confidence levels of a plurality of corresponding cells of the second occupancy grid and then fused. The fusion process may have an initial stage for aligning the first and second occupancy grids. For example, it may be beneficial to spatially align the occupancy grids. This will be described in more detail below.

[0035] The third reliability or the third confidence level can be a (mathematical) function of the first confidence level and the second confidence level. The third reliability is thus based on the information of the first reliability and the second reliability. The fusion process may be performed individually for each cell or by combining / fusing the first occupancy grid and the second occupancy grid.

[0036] The third occupancy grid thus includes information based on the map data and the positioning data, as well as information from the image data and / or the geometric data. Since more information is used to generate the third occupancy grid, the prediction of the drivable road is more likely to be highly accurate.

[0037] The control unit is not essential for the present invention. The control unit can be implemented by a known control unit for autonomously or semi-autonomously driving the vehicle. The present invention can be considered to be in providing the information / data based on which the control unit operates.

[0038] The control unit can be electronically and / or communicably connected or coupled to the fusion unit to receive the third occupancy grid. The control unit may be a part or a functional unit of the above-described computer. The control unit is configured to generate signals for controlling a steering device, a throttle device (also referred to as an acceleration device), and a brake device for driving the vehicle on a drivable road. The steering device, the throttle device, and the brake device may be a part of a control device for (mechanically) navigating the vehicle.

[0039] The steering device can be a part of the vehicle for adjusting the direction or the traveling direction of the vehicle. The throttle device may also be a part of the vehicle for controlling the speed of the motor or the engine and thus the speed and acceleration of the vehicle. The braking device can be a part of the vehicle for decelerating the vehicle by providing friction to slow down the speed of the wheels or tires of the vehicle. The steering device, the throttle device, and the braking device may be controlled based on signals output by the control unit.

[0040] The control unit may execute an algorithm for navigating the vehicle based on information of the drivable road (third occupancy grid), and / or may include such a neural network (AI).

[0041] The drive device may be a part of the vehicle and is electronically and / or communicably connected or coupled to the positioning device and the optical detection device by a positioning input port and an optical input port respectively. In other words, the positioning device outputs positioning data input to the drive device via the positioning input port, and the optical detection device outputs image data and / or geometric data input to the drive device via the optical input port. The map-based drivable road detection unit is electronically connected or coupled to the positioning input port. The optical sensor-based drivable road detection unit is electronically and / or communicably connected or coupled to the optical input port.

[0042] In an optional embodiment, the map-based drivable road detection unit is configured to create the first reliability calculated based on the positioning accuracy of the positioning data and the update date of the map data. Optionally, the optical sensor-based drivable road detection unit is configured to create the second reliability based on the uncertainty of the processing of the semantic segmentation of the image data.

[0043] The map-based drivable road detection unit may include a function for determining the accuracy of vehicle position determination. For example, the signal strength of the GPS signal, the number of satellites from which the GPS signal is received, and / or other characteristics may be used to determine the vehicle position determination or the accuracy of the position. The map-based drivable road detection unit may combine information on the vehicle position determined using GPS or the Global Navigation Satellite System (GNSS) and data from the Inertial Measurement Unit (IMU) to improve the accuracy of the position determination data. The map-based drivable road detection unit may include a processing system and / or function for calculating or computing a reliability based on the accuracy of the position determination. For example, the map-based drivable road detection unit includes a (mathematical) function that associates a first reliability with the position determination accuracy.

[0044] The first reliability may also be based on the update date of the map data. For example, the longer the period until the last update, the lower the first reliability may be. In other words, the older the version of the map, the lower the likelihood that the map data is accurate. For example, the longer the time elapsed between the last update and the calculation of the first reliability, the lower the likelihood that the map data is accurate. Another cause for the map data not being up-to-date may be semantic information, for example, separate labels such as drivable roads, paved roads, railway tracks, etc. Therefore, the first reliability needs to be lower compared to the state where the map data is up-to-date. For example, the map-based drivable road detection unit includes a (mathematical) function that associates the first reliability with the period elapsed since the last update of the map data.

[0045] The optical sensor-based drivable road detection unit may include a function for determining a second reliability in consideration of the uncertainty in the processing of semantic segmentation of image data and / or geometric data. For example, the optical sensor-based drivable road detection unit calculates or computes the second reliability based on the uncertainty in the processing of semantic segmentation of the image data. The optical sensor-based drivable road detection unit may include a (mathematical) function that associates the second reliability with the uncertainty in the processing of semantic segmentation of the image data. The semantic information may refer to drivable roads, paved roads, vehicles, and / or other information regarding the surroundings of the vehicle.

[0046] The uncertainty in the processing of semantic segmentation of the image data may be determined by the optical sensor-based drivable road detection unit. For this purpose, the optical sensor-based drivable road detection unit may include statistical or other types of information indicating the uncertainty in the processing of semantic segmentation of the image data. For example, the statistical or other types of information may be collected by simulating the process of semantic segmentation of the image data that can determine the uncertainty in the processing.

[0047] In an optional embodiment, the first resolution of the first occupancy grid is different from the second resolution of the second occupancy grid, and optionally, the fusion unit is configured to correct the lower of the resolutions of the first occupancy grid or the second occupancy grid to match the higher of the resolutions of the first occupancy grid and the second occupancy grid, and further optionally, the fusion unit is configured to fuse the first occupancy grid corrected by the grid resolution update unit and the second occupancy grid.

[0048] The first resolution may be determined by the coarseness of the map data. The coarseness of the map data determines the resolution of the grid. For example, the resolution of the grid that can correspond to the size of each cell (in other words, the area covered by each cell in the real world) can be determined by the number of data points per unit area.

[0049] The second resolution may be determined by the resolution of the optical detection device. For example, the pixel resolution of the camera of the optical sensor device can determine the resolution of the surrounding image, and thus the coarseness of the image data and / or geometric data. The coarseness of the image data and / or geometric data can determine the size of the unit cell, and thus the real-world area corresponding to the unit cell. In other words, the real-world area corresponding to one pixel is equal to the second resolution. The same discussion applies to the resolution of the optical detection device when the optical detection device includes LiDAR or radar.

[0050] In addition, the second resolution is determined by the distance of the real-world object from the camera. The farther the object is from the camera, the fewer pixels are required to image the object. Therefore, the second resolution can change due to the movement of the vehicle. As a result, the first resolution is usually different from the second resolution.

[0051] The grid resolution update unit may be part of or a functional unit of the computer described above. The grid resolution update unit may be provided to adjust or match the first resolution to the second resolution, or the second resolution to the first resolution. The grid resolution update unit may change to a lower resolution of the first resolution and the second resolution.

[0052] The modification of the first resolution or the second resolution may be performed by interpolation, averaging, and / or other mathematical methods for increasing the resolution of the grid. The fusion may consist of the application of a discrete Gaussian averaging of the input. For example, a cell is divided into a plurality of sub-cells to increase the resolution. The number of sub-cells is selected to match the number of cells of the occupancy grid having a higher resolution. The reliability of the sub-cells may have the value of the previous cell, the average value between adjacent cells, and / or is interpolated such that there is a smooth transition from one adjacent cell to the sub-cell, between sub-cells, and from the sub-cell to another adjacent cell. One way to modify the reliability is based on their proximity to other cells. For example, the value of the resulting cell is defined by the values of the surrounding cells. This is an example of interpolation.

[0053] The fusion unit may fuse the first and second occupancy grids based on the modified resolution. This makes it possible to match each cell from the first occupancy grid to the corresponding cell of the second occupancy grid.

[0054] In an optional embodiment, the first resolution of the first occupancy grid is lower than the second resolution of the second occupancy grid, and optionally, the grid resolution update unit is configured to modify the first resolution of the first occupancy grid to match the second resolution of the second occupancy grid.

[0055] Due to the high resolution of the optical detection device, the resolution of the first occupancy grid is lower compared to the second occupancy group due to the coarseness of the map data. In this case, the grid resolution update unit increases the resolution of the first occupancy grid as described above. After this step, the resolution of the first occupancy grid matches the resolution of the second occupancy grid. Therefore, the fusion unit can fuse each cell of the first occupancy grid with the corresponding cell of the second occupancy grid.

[0056] The grid resolution update unit can be aggregated as a part configured to modify the first occupancy grid and the second occupancy grid such that each cell of the first occupancy grid has a corresponding cell in the second occupancy grid having the same size and / or position. For this purpose, the grid resolution update unit may add, delete, and / or divide cells within the first occupancy grid and / or the second occupancy grid. Optionally, the deletion of cells is performed to align the size of the occupancy grid having a larger dimension to the occupancy grid having a smaller dimension. For example, the second occupancy grid includes dimensions determined by the area that the optical detection device can image. This area is usually smaller than the area covered by the map data. Therefore, the grid resolution update unit may crop the first occupancy grid and / or the map data such that the first occupancy grid matches the second occupancy grid. This may be done by deleting cells from the first occupancy grid that do not have corresponding cells in the second occupancy grid.

[0057] In an optional embodiment, the missing values of the first confidence level and the second confidence level are set between the maximum value of the first confidence level and the second confidence level and the minimum value of the first confidence level and the second confidence level, and optionally, the fusion unit further includes a handling unit configured to set the first missing confidence value or the second missing confidence value to a predetermined value between the maximum value and the minimum value.

[0058] Map data, image data, and / or geometric data may each be missing specific data points that would be required to fully cover the first occupancy grid and the second occupancy grid. These missing data points can be regarded as missing values of the first and / or second confidence levels. Therefore, a "missing value" may refer to a missing entry in the first occupancy grid and / or the second occupancy grid. In other words, the cells of the first occupancy grid and / or the second occupancy grid may not be associated with their respective confidence levels.

[0059] Alternatively, the first occupancy grid and the second occupancy grid are complete, but some cells of the occupancy grid may lack reliability. In either case, the values of the first and second reliabilities are missing. These missing values may be due to measurement artifacts and / or other inconsistencies when acquiring and / or processing image data, geometric data, and / or map data.

[0060] The handling unit may fill these missing values. The handling unit may be part of or a functional unit of the computer described above. The handling unit may be configured to set the missing values between the maximum and minimum values of the reliability. For example, the maximum and minimum values may refer to the boundaries of the possible reliability range, the maximum value may be 1, and / or the minimum value may be zero. The maximum and minimum values can be the reliability of nearby grids or the average of the reliability of surrounding grids.

[0061] The handling unit may be programmed to adaptively, i.e., according to the situation, set the missing values. However, in a preferred embodiment, the handling unit is configured to set the missing values of the first and / or second reliabilities to a predetermined value. This predetermined value may be set in advance considering the normal or expected value of the missing reliability. The handling unit may be configured to set a predetermined value for the first missing reliability value and another predetermined value for the second missing reliability value.

[0062] In an optional embodiment, the predetermined value is the average of the maximum and minimum values. The handling unit may set the missing value to a fixed value that is the average of the maximum and minimum values, for example, 0.5.

[0063] In an optional embodiment, the fusion unit is configured to create the third occupancy grid by calculating the average of the first reliability and the second reliability.

[0064] In this embodiment, the fusion of the first occupancy grid and the second occupancy grid is performed by calculating the average of the reliability of a specific cell in the first occupancy grid and the reliability of the corresponding cell in the second occupancy grid. This means that the third, i.e., the fused reliability, of a specific real-world area (corresponding to a cell in the third occupancy grid) is the average value of the first reliability of the same real-world area (corresponding to the corresponding cell in the first occupancy grid) and the second reliability of the same real-world area (corresponding to the corresponding cell in the second occupancy grid). In other words, the third occupancy grid is obtained by averaging the first and second reliabilities of the respective cells of the first and second occupancy groups. This fusion method requires little computational effort and is thus calculated in a short period of time. This fusion approach may be considered deterministic.

[0065] In alternative and / or additional optional embodiments, the fusion unit is configured to create the third occupancy grid by using Bayes' rule, and optionally, the drivable road detection unit further includes a likelihood calculation unit configured to calculate the likelihood that the second reliability is true by using a map matching algorithm with the first occupancy grid. This fusion approach may be regarded as probabilistic.

[0066] Bayes' rule (or Bayes' law or Bayes' theorem) represents the probability of an event based on prior knowledge of the conditions that may be related to the event. Here, the likelihood p(M x,y |AI x,y ) that the road is drivable is the likelihood p(M x,y ) that the road is drivable based on map data (the first reliability or the first confidence level), the likelihood p(AI x,y ) that the road is drivable based on image data and / or geometric data (the second reliability or the second confidence level), and the likelihood p(AI x,y ) of how true the second confidence level (likelihood p(AI x,y )) is when the initial belief or the first confidence level (likelihood p(M x,y |M x,y) is determined by. The indices x and y indicate the individual cell indices of cell M of the first occupancy grid and cell AI of the second occupancy grid. Specifically, these likelihoods are related by the following equation. TIFF0007699241000001.tif13131 likelihood p(AI x,y |M x,y ) is calculated / computed using a likelihood calculation unit that may be part of or a functional unit of the aforementioned computer. The likelihood calculation unit may be configured to execute well-known map matching algorithms such as ICP (Iterative Closest Point) and NDT (Normal Distributions Transform). ICP is an algorithm used to minimize the difference between two point clouds. In NDT, a normal distribution is assigned to each cell, and each cell locally models the probability of measuring a point. The result of the transformation is a piecewise continuous differentiable probability density. ICP and NCP are known to those skilled in the art, and further explanation of these transformation techniques is substantially unnecessary. Likelihood p(AI x,y |M x,y ) is inversely proportional to the uncertainty of the map matching result. The uncertainty can be estimated from the covariance matrix of the map matching algorithm.

[0067] The storage device, the positioning device, the drive unit, the drivable road detection unit, the map-based drivable road detection unit, the optical sensor-based drivable road detection unit, the fusion unit, the control unit, the grid resolution update unit, the countermeasure unit, and / or the likelihood calculation unit may be communicatively and / or electronically coupled to each other via an interconnection, a network, a bus, and / or a combination thereof. For example, these components may be coupled to each other via a Controller Area Network (CAN) bus. The CAN bus is a vehicle bus standard designed so that microcontrollers and devices can communicate with each other in an application without a host computer.

[0068] A computer-implemented method for driving a vehicle in an automatic mode and / or a driving assistance mode comprises the following steps. a) Generating positioning data of the vehicle using a positioning device; b) Generating image data and / or geometric data around the vehicle using an optical detection device; c) A drivable road detector receives map data from a storage device, receives the positioning data from the positioning device, and creates a first occupancy grid in which each cell represents a first confidence level that the surrounding environment is drivable based on the map data and / or the positioning data; d) An optical sensor-based drivable road detector receives the image data and / or the geometric data from the optical detection device, and creates a second occupancy grid in which each cell represents a second confidence level that the surrounding environment is drivable based on the image data and / or the geometric data; and e) A fusion unit creates a third occupancy grid in which each cell represents a third confidence level that the surrounding environment is drivable by fusing the first occupancy grid and the second occupancy grid.

[0069] Optionally, the method further includes driving the vehicle by a control unit based on the third occupancy grid.

[0070] The above comments, annotations, and optional embodiments regarding the drive unit and the vehicle apply equally to a computer-implemented method for driving a vehicle in an automatic mode and / or a driving assistance mode. The method may be executed by a computer that performs the functions of the drivable road detector, the optical sensor-based drivable road detector, the fusion unit, and / or the control unit.

[0071] In an optional embodiment, the step of creating the first occupancy grid includes creating the first confidence level calculated based on the location identification accuracy of the location identification data and the update date of the map data. Optionally, the step of creating the second occupancy grid includes creating the second confidence level calculated based on the uncertainty of the semantic segmentation process of the image data.

[0072] In an optional embodiment, the first resolution of the first occupancy grid is different from the second resolution of the second occupancy grid. Optionally, the step of creating the third occupancy grid includes using a grid resolution update unit to correct the lower of the first occupancy grid and the second occupancy grid to match the higher of the resolutions of the first occupancy grid and the second occupancy grid. Optionally further, the step of creating the third occupancy grid further includes fusing the first occupancy grid corrected by the grid resolution update unit and the second occupancy grid.

[0073] In an optional embodiment, the first resolution of the first occupancy grid is lower than the second resolution of the second occupancy grid. Optionally, the step of creating the third occupancy grid includes correcting the first resolution of the first occupancy grid to match the second resolution of the second occupancy grid.

[0074] In an optional embodiment, missing values of the first and second confidence levels are set between the maximum value and the minimum value. Optionally, the step of creating the third occupancy grid further includes setting a first or second missing confidence value to a predetermined value between the maximum value and the minimum value.

[0075] In an optional embodiment, the predetermined value is the average of the maximum value and the minimum value.

[0076] In an optional embodiment, the step of creating the third occupancy grid further includes creating the third occupancy grid by calculating an average of the first confidence level and the second confidence level.

[0077] In an optional embodiment, the step of creating the third occupancy grid further includes creating the third occupancy grid by using Bayes' rule, and optionally, the step of creating the first occupancy grid further includes calculating a likelihood that the second confidence level is true by using a map-matching algorithm with the first occupancy grid.

[0078] The present invention further relates to a computer program including instructions that, when executed by a computer, cause the computer to perform the steps of the above-described method.

[0079] The present invention also relates to a computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform the steps of the above-described method.

[0080] Optional embodiments of the present invention will be described in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0081]

Figure 1

Figure 2

Figure 3

Figure 4a

Figure 4b

Figure 4c

Best Mode for Carrying Out the Invention

[0082] FIG. 1 shows a vehicle 10 electronically connected to servers 12 and one or more other vehicles 14 by a network 16. The vehicle 10 can be any type of self-propelled motor vehicle and is preferably configured to travel on a road. For example, the vehicle 10 includes an engine and / or an electric motor for driving the wheels of the vehicle 10.

[0083] The server 12 may be a computer or a computer system that enables access to its storage. The server 12 may store map data indicating a map of drivable roads on which the vehicle 10 or other vehicles 14 can travel. The server 12 can be configured to update the map data. The update of the map data can be achieved by an external input and / or the server 12 may receive updated map data from the vehicle 10 and / or other vehicles 14 via the network 16.

[0084] The other vehicles 14 may travel on the same road as the vehicle 10. The other vehicles may be of the same type or model as the vehicle 10 or may be of different types or models. The network 16 may include a mobile communication network and / or a wireless local area network (WLAN).

[0085] The vehicle 10 includes an optical detection device 20, a positioning device 22, a control device 24, a communication device 26, and / or a drive device 30. The optical detection device 20, the positioning device 22, the control device 24, the communication device 26, and / or the drive device 30 are communicatively and / or electronically connected to each other to exchange data or other types of information.

[0086] The optical detection device 20 may include one or more cameras, a LiDAR device, and / or a radar device. The camera may be a stereo camera. The optical detection device 20 can image the surroundings of the vehicle 10. In particular, the optical detection device 20 is configured to provide a 3D representation of the surroundings of the vehicle 10. The optical detection device 20 outputs the surroundings of the vehicle 10 as image data and / or geometric data.

[0087] The position determination device 22 may be a device for determining the position of the vehicle 10. The position determination device 22 can be a GPS (Global Positioning System) transceiver. The position determination device 22 is configured to output the position of the vehicle 10 as position determination data.

[0088] The control device 24 includes the (mechanical) components of the vehicle 10 that need to be controlled to drive or navigate the vehicle 10. The control device 24 may include a steering device, a throttle device (also referred to as an acceleration device), and a brake device for driving the vehicle 10 on a drivable road.

[0089] The steering device can be a part of the vehicle 10 for adjusting the orientation or traveling direction of the vehicle 10. The throttle device can also be a part of the vehicle 10 for controlling the speed of the motor or engine, and thus the speed and acceleration of the vehicle 10. The brake device can be a part of the vehicle 10 for decelerating the vehicle 10 by providing friction to slow down the speed of the wheels or tires of the vehicle 10. The steering device, the throttle device, and the braking device may be controlled based on signals output by the control unit 34.

[0090] The communication device 26 may be any component that enables communication of the vehicle 10 via the network 16. The communication device 26 may include a wired or wireless transceiver for exchanging data with the network 16. The communication device 26 may be regarded as an interface through which the vehicle 10 can communicate with the server 12. The communication device 26 may also facilitate direct communication with other vehicles 14.

[0091] The drive device 30 can be regarded as a computer or computer system including a plurality of processors (not shown) and a storage device 33. The drive device 30 is configured to execute a plurality of algorithms that can be stored in the storage device 33. The plurality of algorithms processed by the drive device 30 enable the vehicle 10 to navigate autonomously and / or semi-autonomously. The drive device 30 may be regarded as an autopilot or a driving assistance system of the vehicle 10.

[0092] For this purpose, the drive device 30 can execute various functions that can be associated with the drivable road detection unit 32 and / or the control unit 34. Each of these components can be regarded as part of or a functional unit of the drive device 30 that executes specific algorithms to achieve autonomous and / or semi-autonomous navigation of the vehicle 10. The drivable road detection unit 32 can include a map-based drivable road detection unit 32a, a likelihood calculation unit 32b, an optical sensor-based drivable road detection unit 32c, and / or a fusion unit 32d. Accordingly, the component 32 (particularly, the components 32a, 32b, 32c, and / or 32d), and / or 34 can be regarded as an implementation of computer software or a program.

[0093] Algorithms or instructions for components 32, 32a, 32b, 32c, 32d, and / or 34 can be stored in the storage device 33. The drive device 30 can receive location-specific data from the location-specific device 22 via the location-specific input port 35. Similarly, the drive device 30 can receive image data and / or geometric data from the optical detection device 20 via the optical input port 36. The location-specific input port 35 and / or the optical input port 36 can each be regarded as an interface that enables communication between the drive device 30 and the location-specific device 22 and the optical detection device 20.

[0094] The map-based drivable road detection unit 32a is configured to receive map data from the storage device 33 and receive location-specific data from the location-specific device 22 via the location-specific input port 35.

[0095] The map-based drivable road detection unit 32a is further configured to create a first occupancy grid based on the map data and / or the location-specific data. The first occupancy grid is a representation around the vehicle 10 and includes a plurality of cells, and each of the plurality of cells is associated with a first reliability that the surrounding environment is drivable. Therefore, the first reliability of each cell in the first occupancy grid indicates the possibility that the real-world area corresponding to the cell in the first occupancy grid is drivable. The first reliability can be regarded as the likelihood p(M x,y ). Therefore, p(M x,y ) is a relationship or table that associates the reliability p with each cell M x,y , and x, y indicate the individual cell indices of the occupancy grid.

[0096] The map-based drivable road detection unit 32a generates a grid including a plurality of cells M x,y , and each cell corresponds to a specific area around the vehicle 10. Then, the map-based drivable road detection unit 32a associates each cell M x,y with a reliability p indicating the possibility whether the real-world area corresponding to each cell is drivable. This association is the first occupancy grid p(Mx,y ) results in.

[0097] The likelihood p(M x,y ) is calculated or determined based on the positioning accuracy of the positioning data determined by the positioning device 22 and / or the last update date of the map data. The lower the positioning accuracy of the positioning data, the lower the likelihood p(M x,y ) that the real-world area corresponding to a specific cell is drivable. Similarly, the more past the last update of the map data is, the smaller the likelihood p(M x,y ) that the real-world area corresponding to a specific cell is drivable. On the other hand, the relationship between the positioning accuracy and / or the last update time and the likelihood p(M x,y ) can be a mathematical function, a table, or any other type of relationship that can be stored in the storage device 33.

[0098] The optical sensor-based drivable road detection unit 32c is configured to receive image data and / or geometric data from the optical detection device 20 via the positioning input port 35. The optical sensor-based drivable road detection unit 32c is further configured to create a second occupancy grid based on the image data and / or geometric data. The second occupancy grid is a representation around the vehicle 10, includes a plurality of cells, and each of the plurality of cells is associated with a second confidence level that the surrounding environment is drivable. Therefore, each cell of the second occupancy grid indicates the possibility that the real-world area corresponding to the cell of the second occupancy grid is drivable. The second confidence level can be regarded as the likelihood p(AI x,y ). Therefore, p(AI x,y ) is a relationship, a table, etc. that associates the confidence level p with each cell AI x,y , and x, y indicate the individual cell indices of the occupancy grid.

[0099] The optical sensor-based drivable road detection unit 32c thus has a plurality of cells AI x,yGenerate a grid including [the relevant content], and each cell corresponds to a specific area around the vehicle 10. Then, the optical sensor-based drivable road detection unit 32c [relates to each cell M] x,y with the likelihood p indicating whether the real-world area corresponding to each cell is drivable or not. This association gives the second occupancy grid p(AI x,y ).

[0100] The likelihood p(AI x,y ) is calculated or computed based on image data and / or geometric data using a neural network or other forms of artificial intelligence (AI). To determine the second occupancy grid p(AI x,y ), techniques known to those skilled in the art can be used.

[0101] The likelihood calculation unit 32b uses well-known map matching algorithms such as ICP (Iterative Closest Point) and NDT (Normal Distributions Transform) to calculate or compute the likelihood p(AI x,y |M x,y ). The likelihood p(AI x,y |M x,y ) indicates how true or likely the observed value p(AI x,y ) is considering the likelihood p(M x,y ). The likelihood p(AI x,y |M x,y ) is inversely proportional to the uncertainty of the map matching result. The uncertainty can be estimated from the covariance matrix of the map matching algorithm.

[0102] The fusion unit 32d fuses the first occupancy grid and the second occupancy grid to create a new third occupancy grid. Each cell of the third occupancy grid is associated with a third confidence level indicating whether the real-world area corresponding to this cell is drivable or not. In particular, the fusion unit 32d fuses the first confidence level of each cell in the first occupancy group with the second confidence level of the corresponding cell in the second occupancy grid.

[0103] The third reliability is the likelihood p(p(AI x,y ); p(M x,y )) that indicates the possibility of whether the real-world area corresponding to each cell is drivable or not. The likelihood p(p(AI x,y ); p(M x,y )) is determined by the likelihoods p(M x,y ), p(AI x,y ), and / or p(AI x,y |M x,y ).

[0104] In one embodiment, the fusion of the first occupancy group in the second occupancy grid to create the third occupancy grid is performed using the following formula. TIFF0007699241000002.tif14131 This formula is based on Bayes' rule and corresponds to a probabilistic fusion approach.

[0105] In another embodiment, the fusion of the first occupancy group in the second occupancy grid to create the third occupancy grid is performed using the following formula. TIFF0007699241000003.tif12131 In this embodiment, the likelihood p(p(AI x,y ); p(M x,y )) is the average of p(M x,y ) and p(AI x,y ). In this case, the likelihood p(AI x,y |M x,y ) is unnecessary, and the likelihood calculation unit 32b can be omitted. This formula corresponds to a deterministic fusion approach.

[0106] The fusion unit 32d may include a grid resolution update unit 32d1 and / or a handling unit 32d2. The grid resolution update unit 32d1 addresses situations where the resolutions of the first occupancy grid and the second occupancy grid do not match. This means that one cell of the two occupancy grids does not have a corresponding cell in the other of the two occupancy grids. In other words, a specific area of the real world does not have corresponding cells in both the first occupancy grid and the second occupancy grid.

[0107] These differences in resolution can be based on the different coarseness of the image data, geometric data, location identification data, and map data. In this case, it is impossible to fuse each cell of the first occupancy grid with the corresponding cell of the second occupancy grid. In order to align the number of cells of one of the two occupancy grids with the number of cells of the other of the two occupancy grids, the occupancy grid with a lower resolution (a smaller number of cells) is processed by the grid resolution update unit 32d1. In particular, the grid resolution update unit 32d1 divides the cells into sub-cells to increase the resolution. The reliability of the sub-cells can be selected to correspond to the reliability of the divided cells. However, interpolation or averaging methods may be used to assign reliability to the sub-cells. The fusion unit 32d fuses the two occupancy grids based on the occupancy grid updated by the grid resolution update unit 32d1.

[0108] The handling unit 32d2 functions in a situation where the specific cells of the first occupancy grid and / or the second occupancy grid cannot be attributed to the corresponding reliability. This reason can be artifacts in the determination of the image data, geometric data, map data, and / or errors in the processing of the first and second reliabilities. The handling unit 32d2 sets the missing reliability to a predetermined value between the minimum value and the maximum value of the reliability. In an optional embodiment, the handling unit 32d2 sets the predetermined value to the average of the maximum value and the minimum value. The minimum value may be zero indicating that the real-world area is not drivable, and the maximum value may be 1 indicating that the real-world area is drivable with 100% reliability.

[0109] The control unit 34 may include a known neural network or other types of known artificial intelligence (AI) to generate a drive signal for navigating the vehicle 10. The drive system may be used for autonomous driving and / or assisted driving (semi-autonomous driving). The control unit 34 can include an output port for outputting the drive signal to a control device 24 that controls a steering device, a throttle device, and / or a brake device based on the drive signal.

[0110] A method for autonomously and / or semi-autonomously navigating vehicle 10 will be described with reference to FIG. 3.

[0111] The first step is a map-based detection step. The drivable road ahead of vehicle 10 is detected in consideration of information regarding the positioning of vehicle 10 using a positioning device 22 (e.g., GPS) and map data obtained from a storage device 33. The reliability of the map-based drivable road detection p(M x,y ) can be a function of the positioning accuracy and the map update date. The resulting drivable road is represented in the form of a first occupancy grid having a predetermined dimension and resolution. Each cell of the first occupancy grid represents the likelihood that the surrounding environment is drivable (e.g., 0 is non-drivable, 0.5 is drivable with 50% reliability, 1 is drivable with 100% reliability).

[0112] The second step is an AI-based detection step that can be executed in parallel with the first step. The drivable road ahead of vehicle 10 is detected in real time using an optical sensing device 20 (e.g., a camera) and AI for semantic segmentation executed by an optical sensor-based drivable road detection unit 32c. The reliability of the drivable road detection p(AI x,y ) is a function of the uncertainty of the AI (e.g., a Bayesian neural network). The resulting drivable road is represented in the form of a second occupancy grid having a predetermined dimension and resolution. Each cell of the second occupancy grid represents the likelihood that the surrounding environment is drivable (0 is non-drivable, 0.5 is drivable with 50% reliability, 1 is drivable with 100% reliability).

[0113] The third step is the grid dimension and resolution update step. To enable the fusion of the first occupancy grid and the second occupancy grid, it is necessary to ensure that the dimensions and resolutions of the occupancy grids are the same. The occupancy grid with a lower resolution is corrected (e.g., using interpolation) to match the number of cells per meter of the occupancy grid with a higher resolution. The occupancy grid with a larger dimension may be cropped to match the occupancy grid with a smaller dimension. This is executed by the grid resolution update unit 32d1. Further, the handling unit 32d2 fills in the missing reliability of the first occupancy grid and / or the second occupancy grid as necessary.

[0114] The fourth step is the likelihood step of AI being true. The likelihood p(AI x,y ) being true, p(AI x,y |M x,y ), is calculated considering the map-based detection p(M x,y ) using well-known map matching algorithm approaches such as ICP and NDT. The likelihood p(AI x,y |M x,y ) is inversely proportional to the matching error, which is one of the outputs of the map matching algorithm. This is executed by the likelihood calculation unit 32b.

[0115] The fifth step is the final result step. The final output of the fusion process in the case of the new belief p(M x,y |AI x,y ) of the driving road detection using Bayes' rule. This is a probabilistic fusion approach. A simpler alternative would be to fuse by calculating the average reliability between the map-based p(M x,y ) detection and the AI-based p(AI x,y ) detection. This is a deterministic fusion approach.

[0116] A method for autonomously and / or semi-autonomously navigating the vehicle 10 of FIG. 3 will be described in more detail in conjunction with FIGS. 4a to 4c.

[0117] Figure 4a illustrates the second step described above. First, image data and / or geometric data are received from the optical detection device 20. The image data and / or geometric data are data that enable the generation of a three-dimensional image around the vehicle 10. The image data or the three-dimensional image around the vehicle 10 is then projected onto a common coordinate system such as the coordinate system of the vehicle 10. If the optical detection device 20 includes a plurality of optical sensors (e.g., a stereo camera and a LiDAR device) that generate image data and / or geometric data, the image data and / or geometric data of each optical sensor are fused into a common three-dimensional representation around the vehicle 10 in the common coordinate system. For this purpose, different types of image data and / or geometric data are fused using interpolation, averaging, and / or other types of fusion techniques, so that the fused image data and / or geometric data in the common coordinate system have the same resolution and dimensions.

[0118] In the next step, the fused image data and / or geometric data are segmented into drivable road sections and non-drivable road sections using a pre-trained neural network. Thereby, each section is associated with an estimated confidence indicating the likelihood that each section is drivable. This is done in the common coordinate system.

[0119] Thereafter, a second occupancy grid with a resolution M is created, which includes a plurality of cells each associated with the confidence or likelihood that the road section corresponding to the cell is drivable.

[0120] In another optional step, to train the neural network, fused image data and / or geometric data from all sensors may be collected. The drivable road sections are accordingly labeled and fed into a training session of the neural network to segment the drivable road sections from the sensor data in the common coordinate system.

[0121] Figure 4b illustrates the first step described above. First, the location data is received from the location device 22. The location data is in the world coordinate system, that is, not in the common coordinate system. At the same time, access is made to the storage device 33 to receive the map data. The map data is also in the world coordinate system, that is, not in the common coordinate system.

[0122] In an optional offline process, satellite images of the area of interest are found and these satellite images are each aligned with the real-world coordinates. Next, a detailed road network is drawn on the satellite images to create a geotagged map database.

[0123] The location data is used to find the nearest waypoint or node within the map database. This is still done in the world coordinates. Based on the selected node, the drivable roads within the area in front of the node (that is, in front of the vehicle 10) are extracted and a confidence level is assigned to each section of the drivable road. Since this step is still done in the world coordinates, as the next step, the drivable roads are projected onto the common coordinate system, that is, the vehicle coordinate system. Based on the projected drivable roads, a second occupancy grid having a resolution N is created.

[0124] Figure 4c illustrates the above-described third to fifth steps. In the first step, it is checked whether the first occupancy grid having a resolution N and the second occupancy grid having a resolution M include cells that are not associated with a confidence or likelihood that the road is drivable. If so, the missing confidence can be assigned to be an average between a minimum value and a maximum value, for example, 0.5. Additionally, it is checked whether the resolutions and dimensions of the first occupancy grid and the second occupancy grid match. If not, the resolution and / or dimension of the occupancy grid having a higher resolution and / or a larger dimension is cropped. Alternatively or additionally, the resolution of the occupancy grid having a lower resolution is increased by dividing the cells into sub-cells. The confidence of the sub-cells is set with reference to the confidence of the divided cells and / or the confidence of the cells adjacent to the divided cells.

[0125] Thereafter, a new third occupancy grid is created by fusing the first occupancy grid and the second occupancy grid. Each cell of the third occupancy group has a third confidence created by fusing the respective first confidence and the respective second confidence. The fusion method has been described above. The third occupancy grid has a resolution that is the maximum of the first resolution N and the second resolution M.

[0126] In a final optional step, the vehicle 10 is navigated based on the third occupancy grid.

Claims

1. A drive device for the automatic driving and / or assisted driving of a vehicle (10), comprising: a storage device (33) configured to store map data; a positioning input port (35) configured to receive positioning data of the vehicle (10); an optical input port (36) configured to receive image data and / or geometric data indicating the surroundings of the vehicle (10); a drivable road detection unit (32) including a map-based drivable road detection unit (32a), the map-based drivable road detection unit (32a) being configured to receive the map data from the storage device (33) and the positioning data from the positioning input port (35), and configured to create a first occupancy grid in which each cell represents a first reliability that the surrounding environment is drivable based on the map data and / or the positioning data; the drivable road detection unit (32) further includes an optical sensor-based drivable road detection unit (32c), the optical sensor-based drivable road detection unit (32c) being configured to receive the image data and / or the geometric data from the optical input port (36), and configured to create a second occupancy grid in which each cell represents a second reliability that the surrounding environment is drivable based on the image data and / or the geometric data; the drivable road detection unit (32) includes a fusion unit (32d) configured to create a third occupancy grid in which each cell represents a third reliability that the surrounding environment is drivable by fusing the first occupancy grid and the second occupancy grid; the drive device (30) includes a control unit (34) configured to generate a drive signal for automatic driving and / or assisted driving based on the third occupancy grid, the drive signal being output to the vehicle (10); the first resolution of the first occupancy grid is different from the second resolution of the second occupancy grid; the fusion unit (32d) further includes a grid resolution update unit (32d1) configured to correct the lower of the resolutions of the first occupancy grid or the second occupancy grid so as to match the higher of the resolutions of the first occupancy grid and the second occupancy grid. The fusion unit (32d) is a drive device configured to fuse the first occupancy grid corrected by the grid resolution update unit (32d1) and the second occupancy grid.

2. A vehicle, a storage device (33) configured to store map data, a position specifying device (22) configured to output position specifying data of the vehicle (10), an optical detection device (20) configured to output image data and / or geometric data indicating the surroundings of the vehicle (10), a drivable road detection unit (32) including a map-based drivable road detection unit (32a), the map-based drivable road detection unit (32a) being configured to receive the map data from the storage device (33) and the position specifying data from the position specifying device (22), and to create a first occupancy grid in which each cell represents a first reliability that the surrounding environment is drivable based on the map data and / or the position specifying data, and a drivable road detection unit (32). The drivable road detection unit (32) further includes an optical sensor-based drivable road detection unit (32c), the optical sensor-based drivable road detection unit (32c) being configured to receive the image data and / or the geometric data from the optical detection device (20), and to create a second occupancy grid in which each cell represents a second reliability that the surrounding environment is drivable based on the image data and / or the geometric data. The drivable road detection unit (32) includes a fusion unit (32d) configured to create a third occupancy grid in which each cell represents a third reliability that the surrounding environment is drivable by fusing the first occupancy grid and the second occupancy grid. The vehicle (10) includes a control unit (34) configured to drive the vehicle (10) in an automatic driving mode and / or an assisted driving mode based on the third occupancy grid. The first resolution of the first occupancy grid is different from the second resolution of the second occupancy grid. The fusion unit (32d) further includes a grid resolution update unit (32d1) configured to correct the lower resolution of the first occupancy grid or the second occupancy grid so as to match the higher resolution of the first occupancy grid and the second occupancy grid. The vehicle, wherein the fusion unit (32d) is configured to fuse the first occupancy grid corrected by the grid resolution update unit (32d1) and the second occupancy grid. **Claim 3** The map-based drivable road detection unit (32a) is configured to create the first reliability calculated based on the position specification accuracy of the position specification data and the update date of the map data. The drive device or vehicle according to claim 1 or 2, wherein the optical sensor-based drivable road detection unit (32c) is configured to create the second reliability based on the uncertainty of the semantic segmentation process of the image data. **Claim 4** The first resolution of the first occupancy grid is lower than the second resolution of the second occupancy grid. The drive device or vehicle according to any one of claims 1 to 3, wherein the grid resolution update unit (32d1) is configured to correct the first resolution of the first occupancy grid so as to match the second resolution of the second occupancy grid. **Claim 5** The missing values of the first reliability and the second reliability are set between the maximum values of the first reliability and the second reliability and the minimum values of the first reliability and the second reliability. The drive device or vehicle according to any one of claims 1 to 4, wherein the fusion unit (32d) further includes a countermeasure unit (32d2) configured to set the first missing reliability value or the second missing reliability value to a predetermined value between the maximum value and the minimum value. **Claim 6** The drive device or vehicle according to claim 5, wherein the predetermined value is the average of the maximum value and the minimum value. **Claim 7** The drive device or vehicle according to any one of claims 1 to 6, wherein the fusion unit (32d) is configured to create the third occupancy grid by calculating the average of the first reliability and the second reliability. **Claim 8** The fusion unit (32d) is configured to create the third occupancy grid by using Bayes' rule. The travelable road detection unit (32) further includes a likelihood calculation unit (32b) configured to calculate the likelihood that the second reliability is true by using a map matching algorithm with the first occupancy grid. The drive device or vehicle according to any one of claims 1 to 6.

9. A computer-implemented method for driving a vehicle (10) in an automatic mode and / or a driving assistance mode, comprising: generating position identification data of the vehicle (10) using a position identification device (22); generating image data and / or geometric data showing the surroundings of the vehicle (10) using an optical detection device (20); receiving map data from a storage device (33), receiving the position identification data from the position identification device, and creating a first occupancy grid in which each cell represents a first reliability that the surrounding environment is travelable based on the map data and / or the position identification data; receiving the image data and / or the geometric data from the optical detection device (20), and creating a second occupancy grid in which each cell represents a second reliability that the surrounding environment is travelable based on the image data and / or the geometric data; creating a third occupancy grid in which each cell represents a third reliability that the surrounding environment is travelable by fusing the first occupancy grid and the second occupancy grid; driving the vehicle (10) based on the third occupancy grid, and wherein a first resolution of the first occupancy grid is different from a second resolution of the second occupancy grid; the step of creating the third occupancy grid includes modifying the lower of the first occupancy grid or the second occupancy grid to match the higher of the resolutions of the first occupancy grid and the second occupancy grid using a grid resolution update unit (32d1); the step of creating the third occupancy grid further includes fusing the first occupancy grid modified by the grid resolution update unit (32d1) and the second occupancy grid.

10. The step of creating the first occupancy grid includes creating the first reliability calculated based on the position identification accuracy of the position identification data and the update date of the map data. The step of creating the second occupancy grid includes creating the second confidence level calculated based on the uncertainty of the semantic segmentation process of the image data, the method according to claim 9.

11. The first resolution of the first occupancy grid is lower than the second resolution of the second occupancy grid, The step of creating the third occupancy grid includes modifying the first resolution of the first occupancy grid to match the second resolution of the second occupancy grid, the method according to claim 9 or 10.

12. The step of creating the third occupancy grid further includes creating the third occupancy grid by calculating an average of the first confidence level and the second confidence level, the method according to any one of claims 9 to 11.

13. The step of creating the third occupancy grid further includes creating the third occupancy grid by using Bayes' rule, The step of creating the first occupancy grid further includes calculating the likelihood that the second confidence level is true by using a map matching algorithm with the first occupancy grid, the method according to any one of claims 9 to 11.

Images