Method and system for generating a surrounding environment model

Abstract

To provide a method and a system for supplying a surrounding environment model at a low cost.SOLUTION: A method for generating a surrounding environment model in an own vehicle (FE) comprises the steps of: recording a surrounding environment of the own vehicle (FE) by using at least one surrounding environment detection sensor; generating (S2) a first surrounding environment model (U1) based on recording of the at least one surrounding environment detection sensor; processing (S3) the first surrounding environment model (U1) in a trained neural network by using a surrounding environment model (UF) generated based on sensor data; and generating (S4) a second surrounding environment model (U2) based on processing of the first surrounding environment model (U1).SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a method and a system for generating a surrounding environment model.

Background Art

[0002] In the prior art, in an ADAS or AD system, it is known that different externally receptive surrounding environment sensors, such as cameras, lidars, radars, or ultrasonic sensors, are used to generate a sufficient vehicle surrounding environment model. At that time, in an experimental prototype of an automated vehicle, in particular, expensive and high-resolution 360° lidar sensors are installed. Although these 360° lidar sensors enable excellent surrounding environment model results, they are still costly. Therefore, especially in mass-produced vehicles, they have been replaced by low-cost solutions.

[0003] Artificial intelligence (AI), especially deep neural networks, are currently widely spread and outperform human results in certain applications, such as in the fields of image processing, language processing, etc. By using this method, it is particularly possible to solve inverse problems that are actually considered "impossible", such as colorizing black-and-white images or estimating depth maps based purely on monocular camera images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Therefore, an object of the present invention is to provide a method and a system that can supply an improved surrounding environment model at low cost.

Means for Solving the Problems

[0006] This problem is solved by the subject matters of claims 1 and 5. Further advantageous configurations and embodiments are the subject matters of the dependent claims.

[0007] In a first consideration, in the area of computer graphics, it has been found that there is an approach of first preprocessing a low-resolution image and then upscaling it to a high resolution in a post-processing step to optimize the image quality and calculation time of video games. This is no longer done by fixed filter technology, which is currently preferred, but by a neural network pre-trained based on a large number of video games. On the other hand, this makes it possible to generate a higher image quality than directly calculating the image at a high native resolution. For example, Patent Document 1 describes an example of a network for upscaling a low-resolution image.

[0008] This approach is applied to the level of the surrounding environment model.

[0009] Therefore, according to the present invention, in a method for generating a surrounding environment model in a host vehicle, - recording the surrounding environment of the host vehicle using at least one surrounding environment detection sensor; - generating a first surrounding environment model based on the recording of at least one surrounding environment detection sensor; - processing the first surrounding environment model in a trained neural network; - generating a second surrounding environment model based on the processing of the first surrounding environment model. A method comprising these steps is proposed.

[0010] The at least one surrounding environment detection sensor may be, for example, a monocular camera, a stereo camera, a radar sensor, a lidar sensor, and / or an ultrasonic sensor. Preferably, for recording the vehicle surrounding environment, not only a single sensor but also a plurality of similar and / or different sensors are used in the host vehicle.

[0011] Create a first surrounding environment model from the data recorded by the surrounding environment detection sensor. The surrounding environment model may include, for example, all detected objects / free spaces / lane division lines / traffic signs. Therefore, the surrounding environment model may be created as a grid map having, for example, occupied or unoccupied grid cells (or additional cell classes). Alternatively, it is also conceivable to create a free space map, an object-based description, or a combination of different models. When using a plurality of different sensors, the surrounding environment model may be created based on information from different surrounding environment detection sensors.

[0012] The created first surrounding environment model is processed in a trained neural network to create a second surrounding environment model. Through the processing in the neural network, the first surrounding environment model is extended or improved using information from the neural network, and as a result, a surrounding environment model containing more accurate information that is undetectable by some sensors provided on the host vehicle or undetectable at a high level of detail (LOD) is created. Here, the first and second surrounding environment models have substantially the same (but not necessarily identical) information, such as objects, free spaces, etc., while mainly differing in terms of the LOD of the information description.

[0013] The neural network is, a priori, trained using sensor data from high-precision surrounding environment sensors before being used in the host vehicle, and as a result, an AI-based virtual surrounding environment model supersensor is created that can improve the quality of the surrounding environment model generated by simpler sensors. This can be regarded as similar to the above network that upscales images in a video game, which learns the ideal appearance of an image based on high-quality and valuable pixel material (e.g., a game scene depicted at 16K resolution). Therefore, in order to train the neural network, sensor RAW data is generated using high-precision reference sensors of fleet vehicles around a real vehicle. Based on these high-precision sensor RAW data, an accurate and high-precision surrounding environment model is generated using labeling, post-processing, etc. as needed. Here, the neural network is trained offline to learn the general surrounding environment model representation around the vehicle.

[0014] In a particularly preferred configuration, the resolution of the generated first surrounding environment model is improved by the processing in the neural network. Improving the resolution of the first surrounding environment model, in light of the present invention, is understood to mean improving the LOD of the generated surrounding environment model using the neural network. Therefore, the first surrounding environment model is a surrounding environment model with a low resolution, and the second surrounding environment model after processing is a surrounding environment model with a high resolution. This is advantageous because in this way, there is no need to use additional or expensive sensors to create a high-detail or high-precision surrounding environment model.

[0015] In a further preferred configuration, the neural network uses information from the learned high-resolution semantic grid map for the processing of the first peripheral environment model. Particularly preferably, in this case, it is a semantic dynamic grid map that globally represents the surroundings of the stationary and dynamic vehicle in an object-free form. This is advantageous because such an intermediate representation within a peripheral environment model with a low level of abstraction contains more details than a general abstract peripheral environment model output representation, for example, a list of traffic participants, a road model, a free space model, etc. In this way, the neural network learns, as described above, how an ideal type of high-resolution, semantic dynamic grid map generated based on a high-accuracy reference sensor looks like. The neural network uses this knowledge to, for example, upscale and thereby improve a low-resolution grid map generated in the host vehicle online in a post-processing step.

[0016] Also, particularly preferably, due to the improvement in resolution, geometry improvement, classification improvement, completion of unobserved appearances, and / or improvement of higher-order states are performed in the first surrounding environment model. By geometry improvement, for example, the size of other traffic participants can be determined better. Also, in more detail, the classification of individual objects existing in the surrounding environment model can be improved, and thus, for example, more accurate discrimination between different traffic participants can be achieved. The completion of unobserved appearances is advantageous because the neural network can, for example, complete the lane markings using the knowledge based on the high-resolution surrounding environment model, because the network knows how the solid lane markings ideally look and where such markings usually appear in the lane. In this way, for example, lane markings erroneously detected as non-solid markings can be completed and corrected, thereby obtaining an improved surrounding environment model and improving the safety in the host vehicle because more accurate and complete data is supplied to the driving assistance system using the surrounding environment model. The improvement of higher-order states may be understood, for example, as an improvement in cell speed estimation. Using a high-accuracy surrounding environment model and determining additional traffic participants with high accuracy, these motion profiles can be determined more accurately together with possible acceleration capabilities, direction-changing capabilities, etc., and thus how each traffic participant moves can be predicted better. For example, a cyclist has a lower acceleration capability than a car but a faster direction-changing capability.

[0017] Also, according to the present invention, a system for generating a surrounding environment model is proposed. This system includes at least one surrounding environment detection sensor that records the surrounding environment of the host vehicle, and data recorded by the surrounding environment detection sensor Analysis and organizationAn evaluation unit, and a calculation unit configured to create a first surrounding environment model based on the records of at least one surrounding environment detection sensor, process the first surrounding environment model using a neural network, and generate a second surrounding environment model by processing the first surrounding environment model. Here, the calculation unit may be an ECU, an ADCU (Assisted & Automated Driving Control Unit), or a calculation unit integrated on the sensor side. A data connection is provided for data exchange between the surrounding environment detection sensor and the evaluation unit or between the evaluation unit and the calculation unit. Here, the data connection may also be configured as wired or wireless, for example, Bluetooth, a mobile network, Wifi, etc.

[0018] As the neural network, preferably, a recurrent neural network that can take into account the temporal relationship may be used. For upscaling, in particular, a fully convolutional network may be used. In the case of an exemplary grid-based intermediate representation, the grid may be understood as an image, and a similar network architecture may be used, for example, when upscaling computer graphics.

[0019] Further advantageous configurations and embodiments can be obtained from the drawings.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Modes for Carrying Out the Invention

[0021] FIG. 1 schematically shows a flowchart of a method according to an embodiment of the present invention. In step S1, the surrounding environment of the host vehicle FE is recorded using at least one surrounding environment detection sensor 2. In a further step S2, a first surrounding environment model U1 is generated based on the data of the recording of at least one surrounding environment detection sensor. This first surrounding environment model U1 is processed in a trained neural network in step S3. Thereafter, in step S4, a second surrounding environment model U2 is generated based on the processing of the first surrounding environment model U1.

[0022] FIG. 2 schematically shows a system according to an embodiment of the present invention. In this case, the system 1 comprises at least one surrounding environment detection sensor 2, an evaluation unit 3 and a calculation unit 4. The at least one surrounding environment detection sensor 2, the evaluation unit 3 and the calculation unit 4 are connected using a data connection D. In this case, the data connection D may be constituted by wire or wirelessly.

[0023] Figure 3 schematically shows the flow for generating a surrounding environment model. As shown in the figure, a neural network is trained using a surrounding environment model UF generated based on the sensor data of the fleet vehicle FF. In this case, the fleet vehicle FF may be equipped with a high-precision sensor. Therefore, the surrounding environment model UF has high-precision data. Here, the surrounding environment model UF may be a semantic dynamic grid map as shown in the figure, and for example, dynamic and static objects, free space, lanes (and additional information potentially related to driving tasks) are input into this grid map. Then, this trained neural network is used in the host vehicle FE. The host vehicle FE records the surrounding environment using at least one surrounding environment detection sensor and generates a first surrounding environment model U1. Here, the host vehicle FE does not have a high-resolution sensor. Therefore, the first surrounding environment model U1 is a low-resolution surrounding environment model U1. In step S4, after processing the first surrounding environment model U1 in the trained neural network, in step S5, a second surrounding environment model U2 is generated. The first surrounding environment model U1 has been extended and optimized using high-precision data in the neural network and is thus upscaled. As a result, the second surrounding environment model U2 is a high-resolution surrounding environment model. Note that although this application relates to the invention described in the claims, it also includes the following from other perspectives. 1. In a method for generating a surrounding environment model in a host vehicle (FE), - a step (S1) of recording the surrounding environment of the host vehicle (FE) using at least one surrounding environment detection sensor (2); - a step (S2) of generating a first surrounding environment model (U1) based on the recording of the at least one surrounding environment detection sensor; - a step (S3) of processing the first surrounding environment model (U1) in a trained neural network; - a method comprising a step (S4) of generating a second surrounding environment model (U2) based on the processing of the first surrounding environment model (U1). 2. The method according to item 1 above, characterized in that the resolution of the generated first surrounding environment model (U1) is improved by the processing in the neural network. 3. The method according to item 1 or 2 above, characterized in that the neural network uses information from a learned high-resolution semantic grid map (UF) for the processing of the first surrounding environment model (U1). 4. The method according to any one of items 1 to 3 above, characterized in that geometry improvement, classification improvement, completion of unobserved appearances, and / or improvement of higher-order states are performed in the first surrounding environment model (U1) by improving the resolution. 5. A system (1) for generating a surrounding environment model, comprising at least one surrounding environment detection sensor (2) for recording the surrounding environment of a host vehicle (FE), an evaluation unit (3) for evaluating the recording by the surrounding environment detection sensor (2), and a calculation unit (4) configured to create a first surrounding environment model (U1) based on the recording of the at least one surrounding environment detection sensor, process the first surrounding environment model (U1) using a neural network, and generate a second surrounding environment model (U2) by processing the first surrounding environment model (U1).

Description of Symbols

[0024] 1 System 2 Peripheral Environment Detection Sensor 3 Evaluation Unit 4 Calculation Unit D Data Connection FE Own Vehicle FF Fleet Vehicle S1~S4 Method Steps UF Peripheral Environment Model of Fleet Vehicle U1 First Peripheral Environment Model U2 Second Peripheral Environment Model

Claims

1. A method for generating a surrounding environment model in a host vehicle (FE), comprising: - recording the surrounding environment of the host vehicle (FE) using a plurality of different surrounding environment detection sensors (2) (step S1); - generating a first surrounding environment model (U1) based on the recordings of the plurality of different surrounding environment detection sensors (step S2); - processing the first surrounding environment model (U1) in a neural network trained using a surrounding environment model (UF) generated based on high-precision sensor data generated using a high-precision reference sensor of a fleet vehicle (FF) (step S3); - generating a second surrounding environment model (U2) with an improved resolution of the generated first surrounding environment model (U1) based on the processing of the first surrounding environment model (U1) (step S4).

2. The method according to claim 1, characterized in that the neural network learns how a high-resolution semantic grid map (UF) looks and uses this knowledge for the processing of the first surrounding environment model (U1).

3. The method according to claim 1 or 2, characterized in that the improvement of the resolution performs geometry improvement, classification improvement, completion of unobserved appearances, and improvement of prediction of the movement of each traffic participant in the first surrounding environment model (U1).

4. A system (1) for generating a surrounding environment model, comprising: a plurality of different surrounding environment detection sensors (2) for recording the surrounding environment of a host vehicle (FE); an evaluation unit (3) for analyzing and arranging the recordings by the surrounding environment detection sensors (2); a calculation unit (4) configured to create a first surrounding environment model (U1) based on the recordings of the plurality of different surrounding environment detection sensors, process the first surrounding environment model (U1) using a neural network trained using a surrounding environment model (UF) generated based on high-precision sensor data generated using a high-precision reference sensor of a fleet vehicle (FF), and generate a second surrounding environment model (U2) with an improved resolution of the generated first surrounding environment model (U1) by the processing of the first surrounding environment model (U1).

Images