Providing a view of the surroundings of a vehicle

Abstract

A method (200) for providing a view of the surroundings of a vehicle (105) comprises steps of driving along a test track (110) by means of the vehicle (105); scanning the surroundings of the vehicle (105); wherein the test track (110) is separated from an area behind it by means of a screen (115); providing a view of an area virtually behind the screen (115); mounting the view in the scan so that the view appears to adjoin the test track (110); and providing the combined scan.

Description

[0001] The present invention relates to the provision of a view of the surroundings of a vehicle. In particular, the invention relates to the provision of training data for a control device of a vehicle.

[0002] A machine learning method can be used for autonomous vehicle control. For example, the vehicle's surroundings can be scanned using a camera, and a scanned object can be automatically detected. An artificial neural network (ANN) can be used for this purpose. However, such a network must be trained with a large amount of test data before it can detect an object with sufficient accuracy.

[0003] Providing test data for this purpose can be critical, as the quality and quantity of the test data can be crucial for the quality of the subsequent detection of an object. Test data can be provided by conducting numerous test drives in which the environment is scanned. However, it can be time-consuming to select views from the collected views that support a training process. Redundant data, meaningless views, or views whose content is not conducive to training can be sorted out. Such a process often involves a lot of manual work, which can make it error-prone and expensive. If the test data is to contain information about an object visible on it so that it can be used for supervised learning, additional manual work may be required to provide this data (labels).

[0004] Alternatively, views of the environment can also be generated synthetically. However, it has been shown that the generated data is often not realistic enough to enable good training results. Furthermore, it is often difficult to generate convincing and representative synthetic test data.

[0005] An object underlying the present invention is to provide an improved technology for providing a view of a vehicle's surroundings, particularly for training or testing purposes for automatic vehicle control. The invention solves this problem by means of the subject matter of the independent claims. Subclaims specify preferred embodiments.

[0006] According to a first aspect of the present invention, a method for providing a view of the surroundings of a vehicle comprises steps of driving along a test track by means of the vehicle; scanning the surroundings of the vehicle; wherein the test track is separated from an area behind it by means of a screen; providing a view of an area virtually behind the screen; mounting the view in the scan so that the view appears to connect to the test track; and providing the combined scan.

[0007] The virtual area behind the screen can be freely designed so that suitable views can be created for a predetermined purpose. The view can be mounted into the scan in the manner of a photomontage, so that the view replaces the screen. The view of the virtual area behind the screen can originate from any source. For example, a library of predefined views can be provided from which a view can be selected to provide a variety of different combined scans based on a single drive on the test track.

[0008] Combined scanning in the test track area can be very realistic, as it can reflect a real situation unchanged and supplement it with a synthetic one. The ability to combine different views with the scanning allows this situation to be placed in different contexts. This allows a multitude of different combined scans to be provided with little effort, which can be used, in particular, for training or testing automatic vehicle control.

[0009] A technique for blending the view into the scan can be borrowed from film or video production, for example. The privacy screen can have a predetermined color, and areas of the scan that have this color can be replaced by the view. This technique is known as chroma keying and is used, for example, to display a weather map behind a meteorologist presenting it.

[0010] It is preferred that the view is created based on a three-dimensional model of an area adjacent to the privacy screen. Any 3D modeling technique can be used for this purpose. The 3D model is preferably a digital model that can be created in a computer. The view can be generated based on the model with a perspective effect, so that the combined scan can appear even more realistic. Conditions prevailing in the model can be adjusted to those prevailing during driving on the test track, for example with regard to lighting or weather conditions. Conversely, however, predetermined conditions can also be simulated in the model so that they can be reflected in the combined scan. For example, combined scans of the surroundings can be created during the same drive under different weather conditions.A combined scan can be created, for example, during the day or at night, during rain, snowfall, or when the sun is high or low in the sky.

[0011] It is further preferred that the model comprises a representation of the test track. In particular, the model can be large enough to encompass the test track and an area visible or scannable from there. Different models can, for example, simulate an urban or rural environment, a highway, a factory road, or an underground car park. The representation can primarily serve to align the model and reality, or a view and a scan. For this purpose, the representation can reproduce features or landmarks in the area of ​​the test track, for example, a lane marking or a traffic sign. These features can be aligned with corresponding features of the scan in order to perform the assembly. The level of detail of the representation can be less important.

[0012] In one embodiment, the view is determined depending on the vehicle's trajectory. The correct perspective of the vehicle's surroundings can be adopted at any time to correctly create or integrate the view. This allows for an improved optical depth effect of the surroundings. The apparent movement of the vehicle relative to a distant object may be slower than relative to a nearby object. Certain details of the model may only be visible from predetermined positions. Depending on the vehicle's position on the trajectory, objects in the model's surroundings may or may not overlap or obscure each other.

[0013] In one embodiment, the view shows an object; a reference to the object is added to the combined scan. The reference can in particular comprise a property or additional information regarding the object. If the object is to be recognized, a label or class of the object can be added. Furthermore, a bounding box around the object can be displayed in the scan so that an object to be recognized is highlighted. In this way, the combined scan can be used in an improved manner for supervised learning. In particular, an ANN can be trained to recognize the object. The properties and information of the object can already be known within the model, so that manual post-processing can be eliminated.In addition, object properties can be added that are inaccurate or difficult to obtain from real data, such as the mass of an object or the texture of its surface.

[0014] It is preferred that the model comprises at least one moving object. The object can, in particular, comprise a person, a vehicle, or another road user. A movement of the object can be independent of a movement of the vehicle. However, it is also possible to model an interaction between the object and the vehicle. In particular, it can be simulated that an object reacts to the vehicle. For example, the object can be a pedestrian who is about to cross the test track but then stops if the vehicle does not decelerate.

[0015] The method can be carried out continuously, resulting in an image data stream or video data stream from combined scans. It is particularly preferred that the method is carried out in real time. The chroma keying already mentioned herein can then be called live keying. The combined scan can be provided to an automatic control device on board the vehicle. This makes it possible to test how the control system controls the vehicle in an environment that is partly real and partly virtually generated. Such a test can provide realistic results in different contexts. For the purposes of the present application, real time is understood to mean that only a predetermined maximum delay elapses between the scanning of the environment and the provision of the combined scan. Such a delay is typically in the range of a few milliseconds.

[0016] According to a further aspect of the present invention, an apparatus for providing a view of the surroundings of a vehicle comprises a scanning device on board the vehicle; wherein the scanning device is configured to scan the surroundings of the vehicle while the vehicle travels along a test track that is separated from an area behind it by a screen; an output device; and a processing device. The processing device is configured to assemble a view of an area virtually behind the screen into a scan of the scanning device so that the view appears to connect to the test track; and to provide a combined scan by means of the output device.

[0017] The processing device can be configured to carry out a method described herein in whole or in part. For this purpose, the processing device can be implemented electronically and comprise a programmable microcomputer or microcontroller, and the method can be in the form of a computer program product with program code means. The computer program product can also be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to the device, or vice versa.

[0018] In one embodiment, the device can be arranged outside the motor vehicle. For example, the device can be provided in the area of ​​the test track, with wireless communication being established between the vehicle and the device. In another embodiment, the device can be implemented at any location, for example, in a server or as a service in a cloud. The device can be connected to the vehicle via a data network. The output device can be located on board the vehicle or at another location.

[0019] In another embodiment, a vehicle, in particular a motor vehicle, comprises a device described herein. The motor vehicle may in particular comprise a passenger car, a truck, or a bus.

[0020] According to yet another aspect of the present invention, a system comprises a test track; a screen separating the test track from an area behind it; and a device as described herein. Optionally, the device may be mounted on board a motor vehicle as described herein. In another embodiment, the system may comprise a motor vehicle having a scanning device connected to the device as described herein via a wireless communication link.

[0021] The screen preferably has a predetermined color. For contrast with the colors of other objects in the area of ​​the test track, blue or green, for example, can be used. The screen can be implemented, for example, in the form of a fence or building paint. In another embodiment, the screen can form a type of tunnel through which the test track runs. The test track and the screen can also be installed in a building, in particular in a hall. For this purpose, the screen can comprise, for example, partition walls, boards, or curtains.

[0022] In a further development of the invention, a section of a roadway on the test track can also be covered by the privacy screen. For example, the section of the roadway can be colored in a predetermined color, so that, with the installation described herein, for example, driving through a water feature, driving over debris or a lost load, or avoiding a pothole or a missing manhole cover can be practiced.

[0023] The invention will now be described in more detail with reference to the accompanying figures, in which: Fig. 1 a system; and Fig. 2 shows a flow chart of a procedure.

[0024] Fig. 1 shows a system 100 with a vehicle 105. The vehicle 105 travels on a test track 110, which is shown as a straight line by way of example. A privacy screen 115 is located to the right and left of the test track 110, preventing the vehicle 105 from seeing an area located behind the privacy screen 115. For clarity, an object 120 is shown on the left side of the test track 110, of which, from the perspective of the vehicle 105, only an upper section can be seen. The privacy screen 115 typically extends vertically at least as far as the vehicle 105 is tall.

[0025] On board the vehicle 105 is a device 125 with at least one scanning device 130. A scanning device 130 preferably operates optically and can, in particular, provide two-dimensional images. In one embodiment, the scanning device 130 comprises a camera. The scanning device can also operate three-dimensionally and, for example, comprise a stereo camera, a depth camera, or a LiDAR sensor. The scanning device 130 can also operate differently than optically and, for example, comprise a radar sensor or an ultrasonic sensor. The scanning device 130 is configured to provide an image scan of the surroundings of the vehicle 105. Depending on the location and scanning angle of a scanning device 130 on the vehicle 105, the size and, in particular, the height of the privacy screen 115 can be determined such that the scanning area is covered as completely as possible by the privacy screen 115.In one embodiment, the privacy screen 115 is so high that it projects above an upper end of a scanning range of a scanning device 130. A scanning range of a scanning device 130 typically comprises a first section, in which the test track 110 or a roadway traveled by the vehicle 105 is located, and a second section, in which the privacy screen 115 is located.

[0026] Scans from a scanning device 130 can be processed by a processing device 135. Further processing can take place on board the vehicle 105 or off-board. In the case of local processing, a processing result can be provided via an interface 140 on board the vehicle 105. Provided data can be stored or processed by a system or device on board the vehicle 105. In one embodiment, the device 125 on board the vehicle 105 provides information to a control device configured to control the vehicle 105. In another embodiment, provided information can be stored and further processed later.

[0027] By means of an optional wireless communication device 145, scans, partial results, or final results of processing by the device 125 can be transmitted to an external location 150. In one embodiment, the external location 150 can process scans from a scanning device 130 on board the vehicle 105. In another embodiment, the external location 150 can receive intermediate or final results of processing scans. Processed results can be stored or otherwise provided. With reference to a technique described herein, both processing on board the vehicle 105 using the device 125 and off-board using the external location 150 is possible.

[0028] It is proposed to determine a virtual view of a three-dimensional model and to mount this view in a scan of a scanning device 130 such that the view of the model replaces the view of the privacy screen 115. The model is preferably designed such that a realistic impression is created, similar to that experienced during a journey with the vehicle 105.

[0029] In a first embodiment, a combined scan can be used to train an artificial neural network to recognize an object depicted in the combined scan. If the object is part of the view, i.e., generated from the model, information about the object can be added to the combined scan. Such information can include, for example, a category, a label, or a property of the object.

[0030] In a second embodiment, the combined sample may be provided to the control device onboard the vehicle 105. The control device may treat the combined sample as an unprocessed sample using a sampling device 130 and control the vehicle 105 depending on the combined sample.

[0031] In both cases, a combined scan in the area of ​​the test track 110 can be unchanged and thus highly realistic. At the same time, different situations, contexts, or environmental influences can be played or overlaid in the area of ​​the privacy screen 115. For example, it can be tested whether a machine learning technique works satisfactorily even if different scenarios are visible in an area adjacent to the test track 110. A first scenario can, for example, encompass a wide area, a second scenario an urban space, and a third scenario a sporting or cultural event. The same immediate scan results from the area of ​​the test track 110 can thus be placed in different frames or contexts.

[0032] Fig. 2 shows a flowchart of a method 200 for providing an environment view of a vehicle 105.

[0033] In a step 205, the vehicle 105 can perform a drive on the test track 110. Of course, the static case in which the vehicle 105 is stationary on the test track 110 is also covered.

[0034] In a step 210, a trajectory of the vehicle 105 relative to the test track 110 can be determined. The trajectory can be determined, for example, using a GNSS satellite navigation system, an acceleration sensor, or a motion sensor on board the vehicle 105.

[0035] In a step 215, an environment of the vehicle 105 can be scanned. The scanning is preferably carried out by means of at least one scanning device 130 attached to the vehicle 105.

[0036] Optionally, a sample may be recorded in a step 220. Recorded samples may be collected and processed at a later time, or immediate processing may occur.

[0037] In a step 225, an environmental model of the vehicle 105 can be created. The environmental model can, in particular, comprise a three-dimensional model of a drivable area. The model can, for example, comprise an urban area, a factory road, or any other section of a road network. It is preferred that the model comprise the test track 110 or a representation thereof.

[0038] In a step 230, the trajectory of the vehicle 105 can be determined. For this purpose, the trajectory data determined on board the vehicle 105 from step 210 can be used. Alternatively, the trajectory can also be determined retroactively, for example, based on scans.

[0039] In a step 235, a view of the environment model from the perspective of the vehicle 105 can be determined. Preferably, both the position and orientation of the vehicle 105 are taken into account. The model can include a plurality of objects, not all of which may be visible from the selected perspective. Which object is visible at which point in a view can be determined, for example, using ray tracing methods.

[0040] In a step 240, the created view can be merged with the created scan. In this case, in the manner of a photomontage, an area of ​​the scan in which the privacy screen 115 is visible can be replaced with a corresponding part of the view created based on the environment model. The result can be an image in which the scan and the view merge as seamlessly as possible.

[0041] The view may be provided in a step 245. The method 200 may run under real-time conditions, so that the view is only slightly delayed compared to the scanning. The view may be provided to a control device that guides the vehicle 105 over the test track 110. Accordingly, the journey may be further controlled in step 205 based on the provided combined view.

[0042] The combined scan can also be provided as training data in a step 250. The training data can be configured to train a machine learning technique to recognize an object depicted in the combined scan or to guide a vehicle 105 over the test track 110 based on the combined scan. Information about a depicted object, which information can be taken from the environment model, can be attached to the training data. Such information can include, for example, an object category, a designation, a speed, or a condition. Reference symbol 100 systems 105 vehicles 110 test track 115 Privacy Screen 120 objects 125 device 130 scanning device 135 Processing facility 140 interface 145 wireless communication device 150 external positions 200 procedures 205 Carry out a trip 210 Capture trajectory 215 Scan the environment Record 220 scans 225 Create environment model 230 Determine trajectory 235 Create view 240 Merge scan with view 245 Provide view Create 250 training data

Claims

[1] Method (200) for providing a view of the surroundings of a vehicle (105), the method (200) comprising the following steps: - driving (205) along a test track (110) by means of the vehicle (105); - scanning (215) an environment of the vehicle (105); - wherein the test track (110) is separated from an area behind it by means of a screen (115); - providing (220) a view of an area virtually located behind the privacy screen (115); - mounting (240) the view into the scan so that the view appears to connect to the test section (110); and - Providing (245, 250) the combined scan. [2] The method (200) of claim 1, wherein the view is created based on a three-dimensional model of an area adjacent to the screen (115). [3] The method (200) of claim 2, wherein the model comprises a representation of the test track (110). [4] Method (200) according to one of the preceding claims, wherein the view is determined as a function of a trajectory of the vehicle (105). [5] The method (200) of any preceding claim, wherein the view shows an object; wherein an indication of the object is appended to the combined scan. [6] The method (200) of any one of claims 2 to 5, wherein the model comprises a movable object. [7] Method (200) according to one of the preceding claims, wherein the method (200) is carried out in real time. [8] Method (200) according to one of the preceding claims, wherein the motor vehicle is controlled autonomously in dependence on the determined view. [9] Device (125, 150) for providing a view of the surroundings of a vehicle (105), the device comprising the following elements: - a scanning device (130) on board the vehicle (105); - wherein the scanning device is configured to scan an environment of the vehicle (105) while the vehicle (105) travels along a test track (110) which is separated from an area behind it by means of a screen (115); - a processing device (135, 150) configured to mount a view of a region virtually located behind the screen (115) into a scan of the scanning device (130) so that the view appears to connect to the test section (110); and - an output device (140, 145) for the combined scanning. [10] Motor vehicle (105) comprising a device (125) according to claim 10. [11] A system (100) comprising a test track (110); a screen (115) separating the test track (110) from an area behind it; and a device according to claim 9. [12] The system (100) of claim 11, wherein the privacy screen (115) has a predetermined color.

Images