METHOD AND ARRANGEMENT FOR PRODUCEING A TEXTURED ENVIRONMENTAL MAP OF A VEHICLE WITH IMAGE INFORMATION AND COMPLETING SUCH ARRANGEMENT OF A VEHICLE

Description

[0001] Method and arrangement for generating an environment map of a vehicle textured with image information and vehicle comprising such an arrangement

[0002] The invention relates to a method and an arrangement for generating an environment map of a vehicle textured with image information, in particular in connection with parking processes, and to a vehicle comprising such an arrangement.

[0003] The use of image information to generate environmental representations in vehicles is well known. Environmental representations are generally used to inform a driver about the current operating situation of the vehicle and, in particular, to provide the driver with a representation of the current vehicle environment. This can be done, for example, using a display device in the vehicle interior on which the environmental representation is displayed.

[0004] Approaches exist in the prior art for creating environmental representations using vehicle-mounted cameras. The environmental representation can be created and displayed in a so-called top-down or bird's-eye view, in which a model of the vehicle itself is often displayed for orientation purposes (for example, in the center of the image). In other words, environmental representations are known in which a model of the vehicle in top view and the vehicle's surroundings are displayed, with the vehicle's surroundings being mapped based on the captured camera images. The applicant offers such a solution under the name "Area View."

[0005] The environment representation is typically compiled from images captured by multiple vehicle-mounted camera devices. For this purpose, approaches are known in which a (virtual) projection screen is created and the images are projected onto this projection screen. The images from multiple cameras are combined to form the environment representation when projected simultaneously. The projection is performed, for example, based on the known installation positions and / or viewing angles of the camera devices.

[0006] In this context, DE 10 2010 051 206 A1 discloses the merging of several partial images into a larger area extending beyond the individual partial images, whereby additional symbols (so-called artificial image elements) can also be displayed. Further technological background exists in the form of DE 10 2015 002 438 A1, DE 10 2013 019 374 A1, DE 10 2016 013 696 A1, and DE 603 ​​18 339 T2.

[0007] WO 2007 / 108221 A1 discloses the generation of a plurality of video streams using a plurality of video cameras of a vehicle, wherein the video streams are combined to generate a virtual camera viewpoint. If areas in the image of one camera are obscured by external objects, these obscured areas can be replaced with information from an image of another camera.

[0008] Further technological background can be found in US 2016 / 049078 A1 and DE 10 2013 214 369 A1.

[0009] It is also known to create so-called environment maps for a vehicle, in which information about the vehicle's surroundings is stored and, in particular, is stored in a location-specific manner (i.e., relates to specific locations within the environment map). The information can be recorded by sensor devices of the vehicle, for example so-called environment sensors. These can generally be designed to record the environment optically, acoustically, via ultrasound and / or radar. Examples of such sensor devices are (optical) distance measuring devices, radar sensors, laser sensors or ultrasonic sensors. The environment map can be stored as a file and / or digital information collection, for example in a control device of the vehicle, and continuously updated. Such an environment map could, in principle, also be used to display a representation of the environment to the driver of a vehicle.

[0010] However, it has been shown that the known approaches do not always achieve the desired quality (i.e., realism) of the environment representation.

[0011] The task is therefore to improve the quality of an environment representation for a vehicle.

[0012] This object is achieved by a method according to the appended claim 1, an arrangement according to the appended claim 6, and a vehicle according to claim 7. It is understood that the variants and features discussed in the introduction can also be provided in the present invention alone or in any combination, unless otherwise stated or apparent.

[0013] A basic concept of the invention is to create an environment map, for example, using the sensor devices explained above, and to texture it with image information captured by at least one camera device. The textured environment map can be used as an environment representation and displayed to a driver. In particular, if the environment map contains and / or depicts information regarding a three-dimensional shape of the environment and / or an elevation profile of the environment, a realistic environment representation can be generated.

[0014] Furthermore, a basic idea of ​​the invention is to select suitable image information for texturing the environment map as needed or in accordance with the situation. For example, it was recognized that during a journey (i.e. a vehicle movement in any direction at a speed other than 0 km / h and preferably while generating a traction force) and in particular during parking maneuvers, environmental areas can be positioned differently within the detection areas of the vehicle's camera devices or can be moved within the detection areas. Since the detection areas of a camera device can usually be divided into high-resolution and low-resolution areas, in which correspondingly more or less qualitative image information can be generated, both high-quality and lower-quality image information can be available for one and the same environmental area during a journey.According to an alternative, the invention therefore proposes to take into account different quality levels of image information relating to one and the same environmental area when texturing the environment map and, in particular, to give preference to high-quality image information for texturing.

[0015] According to the invention, the more recent image information is used (at the latest) when the vehicle falls below a predetermined distance threshold to at least one area of ​​the environment map. In particular, the distance threshold can be considered in the direction of travel. This is based on the idea that vehicle-mounted camera devices generally have a high-resolution detection range close to the vehicle and can only capture low-resolution image information as the distance from the vehicle increases. Thus, if a distance threshold is undershot, it can be assumed with a high degree of probability, for example by selecting the distance threshold accordingly, that the corresponding area has entered the high-resolution detection range.This makes it possible to automatically ensure that image information with a higher resolution is used than the image information already used for texturing, for example without the need for an explicit comparison of the resolutions of the image information mentioned.

[0016] However, it was also recognized that in certain operating situations, the use of current image information may be preferable when changes in the environment are detected and the environment representation must therefore be updated in order to realistically depict the environment.

[0017] Furthermore, the invention can provide for both of the above-mentioned selection criteria to be considered, but for one of them to be preferred by default. For example, high-quality image information can be preferred and selected for texturing by default, unless more recent image information is available for the corresponding area to be textured.

[0018] In general, the textured environment map can be an environmental representation or be used as such. In particular, it can be an environmental representation generated in the form of or based on image and / or video files. This environmental representation can be displayed on a display device, particularly in a vehicle interior.

[0019] As described above, the environment map can depict properties of the environment, in particular their structure and / or shape. According to one variant, the environment map comprises an elevation profile of the environment, which can be a complete three-dimensional profile of the vehicle's surroundings. At the very least, however, an extension of structures in the vehicle's surroundings in the vertical spatial direction should be recorded or stored and / or depicted in the environment map, with the vertical spatial direction preferably running orthogonally to the vehicle's ground. It is understood that the dimensions of the recordable elevation profile (or even the complete three-dimensional profile of the vehicle's surroundings) can be limited, for example, by the sensor devices used and their detection ranges.

[0020] In general, the environment map can be a file and / or a digital collection of information, for example, depicting the environment in the form of a collection of (measured) points and / or a grid model. Texturing can be understood as the mapping of image information onto the environment map (for example, via virtual computer-aided projection) and / or the filling of areas or surfaces of the environment map with the image information. The environment map can thus function in the sense of the projection screen explained in the introduction.

[0021] The acquired image information can be video information and / or a video image data stream. Accordingly, the texturing of the environment map can be continuously updated and / or generated, for example by continuously feeding in new (i.e. currently acquired) image information. When using image information acquired with different camera devices, image information acquired or recorded at the same acquisition times is preferably used to generate the texturing. The image information is preferably acquired with at least one vehicle-mounted camera device. According to one variant, at least three or at least four camera devices can be provided, which can be distributed around the vehicle, for example, so that cameras are preferably provided on each side of the vehicle (i.e. at the front and rear and on the two outer sides surrounding the entry doors).Consequently, the surrounding map can be displayed to the driver in a so-called "surround" or 360° view. As mentioned, the image information can be provided in the form of video data and / or as an image data stream, and the camera devices can be configured as video cameras.

[0022] The environment map can be generated at least partially based on information acquired by at least one vehicle-mounted sensor device, wherein the vehicle-mounted sensor device is preferably a sensor device different from the camera devices. In particular, the sensor device can be configured not to capture the vehicle environment based on measured ambient light intensities, as is common with camera devices. For example, the sensor device can be one of the following: a distance measuring device, a radar device, a lidar device, an ultrasonic device, or an optical distance measuring device (e.g., based on laser radiation).

[0023] The environment map can generally be generated during a journey and, for example, cover a route of at least 5 m, at least 10 m, at least 15 m, or at least 20 m (i.e., the environment map can extend along such a route). Using a circular buffer or the so-called FIFO (First In First Out) principle, areas of the environment map that are preferably located opposite to the current direction of travel or behind the vehicle relative to the current direction of travel can be deleted, and new areas of the environment map that are preferably located in the direction of travel can be added. This can be done by deleting and adding information for these areas accordingly.

[0024] Furthermore, a position of the vehicle within the surrounding map can generally also be determined. For this purpose, odometry information can be used in a conventional manner, which, for example, records or maps wheel movements. Furthermore, a steering angle and / or vehicle inclination can be recorded and taken into account. Information from navigation systems (in particular location information) can also be used, whereby such information can be generated, for example, based on GPS signals. However, a variant without the aid of such information and, in particular, GPS signals is preferred, for example in order to be independent of current reception conditions for such signals.

[0025] This can also be exploited to assign corresponding location information to the image information captured by a camera device. It is advantageous if the camera device is positioned and remains the same within the vehicle (i.e., positioned at a consistent location and / or consistently aligned, thus maintaining a constant viewing angle). Based on this, for example, a vehicle position can be converted into a position of the camera devices and / or corresponding location information can be assigned to the captured image information.

[0026] In a similar way, corresponding location information can also be assigned to areas of the environment map. For example, a position of the sensor device (e.g., distance sensors) can be used to generate the environment map, and a determined vehicle position can be converted into a position of the sensor device. The information acquired from this can then also be provided with location information.

[0027] In general, both areas of the environment map and captured image information can be provided with location information, so that it can also be determined which image information relates to which areas of the environment map.

[0028] In general, the solution presented here can relate to or be used for parking processes. Consequently, the solution can be selectively activated or executed when a driver, for example, activates a parking space search or other assistance functions related to parking processes, or generally makes inputs that indicate a desire to park. At the very latest, the surrounding area map can also be generated and / or textured in the manner described here. In particular, at the very latest, vehicle location within the surrounding area map can be achieved at the very latest.

[0029] If a plurality of camera devices is provided, it can be provided to consider and preferably store the image information acquired by all these camera devices, and then to make a selection regarding the texturing of the environment map based on various criteria. In this case, it can thus be considered irrelevant which camera device provides which image information; instead, only the time of acquisition of the image information (timeliness criterion) or the resolution of the image information (resolution criterion) can be considered.

[0030] As described in the introduction, the aforementioned criteria can also be taken into account, but in a hierarchical form. In other words, a criterion can be prioritized unless a certain operating situation occurs and / or other conditions are met, and then image information is selected according to the other corresponding criterion. For example, the resolution criterion can be considered by default, unless an operating situation is identified in which taking more current image information into account is preferred. This can, for example, relate to the entry of an object (e.g., a pedestrian or car) into the detection area of ​​at least one of the vehicle's camera devices, which can be detected using known image analysis algorithms.

[0031] More specifically, a further development provides that the more current image information is used (at the latest) when a change in the vehicle environment and / or a predetermined object in the vehicle environment is detected within the area.

[0032] If the distance threshold is preferably viewed in the direction of travel, it can also be ensured that only those areas of the environment map that were previously located far from the vehicle and were therefore likely captured only at a low resolution are overwritten with new image information. In particular, it can be provided that the distance threshold or general distance dependencies are viewed exclusively in the direction of travel and, above all, not against the direction of travel. In the latter case, the opposite situation can occur: initially positioned close to the vehicle, thus positioning areas captured with high resolution at an increasing distance from the vehicle.It can therefore be prevented that the surrounding area map is overwritten with low-resolution image information acquired later, since the corresponding areas are then likely to be located at a greater distance from the vehicle and also in a low-resolution detection range of the camera devices.

[0033] According to a further embodiment, a plurality of camera devices can be present and the first and second image information can be captured by different camera devices. In this context, the respective capture areas of the camera devices can be less than 50% congruent or spatially overlap by less than 50%. In particular, the camera devices can be positioned on different and in particular opposite sides of a vehicle. According to the invention, however, it is possible that, for example, when one and the same environmental area is captured by different camera devices due to maneuvering maneuvers during a parking process, the image information available for this can be viewed in combination or can be omitted from the overall information available according to the above criteria.

[0034] Furthermore, it can be provided that the environment map is larger than a section of the (textured) environment map that can be displayed on a display device. In other words, the environment map can thus be stored as a digital model and / or image, preferably continuously updated, but only displayed in sections on the display device. This has the advantage that the environment map can also be evaluated in areas beyond the displayed section, or its information can be used for other purposes.

[0035] The invention further relates to an arrangement for generating an environment map of a vehicle textured with image information, comprising: a detection device for detecting at least first and second pieces of image information from a vehicle's surroundings; and an environment map generation device configured to generate an environment map of the vehicle (for example, based on information detected by the above-mentioned sensor device) and, for at least one area of ​​the environment map for which both first and second pieces of image information are present, either the first or the second pieces of image information are selected for texturing the environment map, the more recent image information being selected when the vehicle falls below a predetermined distance threshold from the area.

[0036] All of the features and developments explained above and below in connection with the method can also be provided in the arrangement. In particular, the arrangement can be configured to carry out a method according to any of the above and below variants.

[0037] Furthermore, the invention relates to a vehicle comprising an arrangement of the above type.

[0038] Finally, it should be noted that, unless otherwise stated or apparent, the creation of a map can also be understood as an update of an already existing map of the environment (i.e., a creation of a current map of the environment).

[0039] An exemplary embodiment of the invention is explained below with reference to the attached schematic figures. They depict: Fig. 1 is a schematic representation of a vehicle comprising an arrangement according to an embodiment of the invention, wherein the arrangement carries out a method according to the invention.

[0040] In Figure 1 A vehicle 100 is schematically indicated, which comprises an arrangement 1 according to the invention. A front side of the vehicle 100 points in the direction of an area B and a rear side in the direction of the area A. Side mirrors 102 are schematically indicated on the sides of the vehicle 100 comprising the entry doors.

[0041] On all sides, the vehicle comprises 100 (surrounding) sensor devices 14, whereby Figure 1a corresponding sensor device 14 is shown only for the front. Furthermore, the vehicle 100 comprises camera devices 16 on all sides, in particular also in both side mirrors 102. The vehicle's surroundings can thus be completely captured in the sense of 360° capture, both visually and by means of the sensor devices 14. The camera devices 16 and also the sensor devices 14 are further arranged such that the capture areas only partially overlap and, in particular, are spatially congruent to less than 50% and, in particular, less than 20%.

[0042] The sensor devices 14 are configured to capture environmental information to create an environment map. In the illustrated case, these are ultrasonic sensors, although any other sensor types of the type mentioned above, and combinations thereof, can also be used within a vehicle 100. The environment map is continuously updated as a file containing digital information and stored in a storage device 18 of the vehicle 100. In the illustrated case, the environment map describes a three-dimensional structure of the vehicle's surroundings.

[0043] The camera devices 16 are each configured as video cameras and continuously record the vehicle's surroundings. A video or image data stream from each camera device 16 is transmitted to an environment map generation device 20, as is the information acquired by the sensor devices 14. In addition to generating the environment map, the environment map generation device 20 can also provide other functions explained above and below, and for this purpose, for example, can include corresponding program instructions and / or links to vehicle units.

[0044] In Figure 1 The vehicle is shown in a state where it is driving from area A to area B. The two areas are approximately 15 m apart. During this journey, the surrounding area map is continuously updated and image information is continuously captured.

[0045] In particular, in the present case, it is assumed that area A is selected as a starting point for generating the environment map, for example because the vehicle is switched on or activated there (for example by ignition or other electrical activation) and / or because a driver's parking request is detected or user input is signaled in this area, whereupon an environment map generation is activated according to the present variant.

[0046] In area A, a vehicle position is set to zero, and the surrounding map generated for the parking process is also created for the first time or newly. Subsequently, based on oedemtricity information, for example, a movement of the vehicle 100 is detected, and a distance and preferably also an orientation relative to area A, which serves as a kind of zero point, is determined. Thus, location information of the vehicle 100 is also available for each point in time.

[0047] Accordingly, captured image information and / or environmental information from the sensor devices, and thus the areas of the environmental map created based thereon, can also be provided with location information. All of this information can be stored in the storage device 18, which can also be part of the environmental map generation device 20.

[0048] As a result, the environment map generation device 20 can thus access image information and areas of an environment map that can be associated with one another based on the location information or, in other words, between which correspondences can be determined based on the location information. Figuratively speaking, it is thus possible to determine which image information reliably relates to which areas of the environment map or depicts them.

[0049] Since the camera devices 16 continuously generate image information, image information relating to different points in time is also available for each camera device 16. If the vehicle 100 is moving, this image information, as expected, depicts different areas of the environment and thus of the environment map. As explained below, however, cases can also occur in which the image information depicts at least partially matching areas of the environment and the environment map, so that a selection can be made between these for texturing the environment map.

[0050] As generally described above, textured mapping can be understood as the projection of image information onto the environment map. In this case, the image information can be used only partially (i.e., only sections of a single image), and image information from all camera devices 14 can generally also be combined with one another. All image information preferably relates to a consistent acquisition time.

[0051] Preferably, the environment map is completely textured, although only a schematically indicated section 30 of the textured environment map can be displayed on a display device in the vehicle interior. This section 30 moves analogously to the vehicle movement within the environment map and preferably always contains the vehicle 100 in the middle or center. At least this displayed section 30 is preferably textured in the manner described herein. In a manner known per se, the section 30 can be displayed as a so-called bird's-eye view or a top view of the vehicle, in which the vehicle is displayed as a symbol. Examples of comparable representations can be found in areas 29a and 29d of Figures 4 & 5 of the aforementioned DE 10 2015 002 438 A1.

[0052] In Figure 1For each of the camera devices 16, a high-resolution detection area is marked with a solid line. The camera devices 16 are each wide-angle camera devices, the angle of view of which at least partially enables detection of the vehicle's underlying surface (ie, directed at the vehicle's underlying surface). In addition, each camera device 16 has a low-resolution detection area, which is at least partially suitable for the Figure 1 left camera device 16 is indicated by dashed lines. It can be seen that not all areas within the section 30 can be captured in high resolution and thus (at least during a first drive from area A to area C) high-resolution image information is not available for every area of ​​the surrounding map.

[0053] In order to nevertheless increase the informative value of the environment map or of an environment representation generated and displayed on this basis, the options explained below are provided, which can be implemented individually or in any combination.

[0054] When driving from A to B, area B is also captured by the camera device 16 on the front of the vehicle 100 at the starting point A, but only at a very low resolution. Thus, initially, only low-resolution image information is available for area B. Texturing of the environment map in area B can therefore only be performed based on correspondingly low-resolution image information.

[0055] However, if the vehicle 100 is located near area B, and in particular such that it lies within the high-resolution detection range of the camera devices 16, high-resolution image information is available for area B. This can be used instead of the low-resolution image information for texturing the environment map. In particular, the environment map or its texturing can be overwritten locally (i.e., overwritten in area B) as soon as high-resolution image information is available for this area B.

[0056] That this is the case can be determined, for example, by comparing the resolutions of all the available image information which can be assigned to area B (as can be seen, for example, based on their location information). According to the invention, however, it can be determined that area B is positioned at a distance from vehicle 100 that is below a distance threshold. The distance threshold is selected such that it corresponds to an outer boundary of the high-resolution detection range of the front camera devices 16. If this is the case, currently acquired image information at different resolutions can be used to texture the environment map, preferably without further investigations or comparisons, then it can be assumed that this information is acquired with a high resolution.

[0057] On the other hand, it may be the case that, as the vehicle moves away from these, area A initially enters the high-resolution detection range of the rear camera devices 16. As the vehicle continues driving toward area B, it moves out of this range and can thus only be captured with a low resolution. Accordingly, it may also be provided to consider distance observations or the above-mentioned distance threshold only in the direction of travel, although this may also be in the reverse direction.

[0058] Additionally, it can be provided that if high-resolution image information (and in particular image information with a resolution above a predetermined threshold) is available for an area of ​​the environment map, this information is used for texturing and is not overwritten by image information acquired at a different time. Thus, once high-resolution image information is available for an area, the detection of potentially even higher-resolution image information can be prevented.

[0059] An exception may exist if a change in the vehicle's surroundings is detected based on both the acquired image information and the (surroundings) information acquired by the sensor devices 14. In both cases, object movement can be detected, for example, by changing the information acquired in each case. If this is the case, current image information for the corresponding area can be selected for texturing the surroundings map, regardless of preferred high resolutions, in order to be able to show the driver a representation of the surroundings that is as realistic as possible. The detection and representation of moving objects is also safety-relevant and increases driving safety if the driver orients himself on the representation of the surroundings, for example during parking maneuvers.

[0060] As described, it is also possible to take only one of the criteria into account, i.e. whether higher resolution image information is available for the same area that is currently being used for texturing the environment map, or whether more recent image information is available than the image information currently being used for texturing.

[0061] Based on Figure 1 Another example of selecting image information for texturing the environment map is described below. Figure 1 Furthermore, an area C is shown. This corresponds to an area to which the driver can move the vehicle 100 from the position shown, for example during a reversing parking operation into a not separately shown area in Figure 1 horizontal parking space.

[0062] Also shown is a static object, such as a pillar 32. This is initially located in a low-resolution detection area of Figure 1 left camera devices 16 and also positioned it in such a way that no high-resolution image information has yet been captured for this area during the journey starting from area A. The environment map is thus initially textured in the area containing pillar 32 based on low-resolution image information.

[0063] If the vehicle 100 now moves backward according to the dashed arrow into area C, the pillar 32 would enter the high-resolution detection range of the left camera devices 16. The environment map generation device 20 would detect this either by comparing the overall image information available for this area or by the distance threshold value described above being exceeded relative to the area containing the pillar 32. Thus, the current texturing of the environment map in the area containing the pillar 32 can be locally overwritten with high-resolution image information. This supports the driver during the parking process if they wish to orient themselves based on the displayed representation of the environment. List of reference symbols

[0064] 1Arrangement 14Camera device 16Curb 18Storage device 20Environment map generation device 30Section 32Pillars 100Vehicle 102Side mirrors A, B, CAreas

Claims

1. Method for generating a surroundings map of a vehicle (100) textured with image information, comprising: capturing at least first and second image information at different times from a vehicle environment with at least one vehicle-mounted camera device (16); generating a surroundings map of the vehicle (100); determining at least one region of the surroundings map for which both the first and second image information are available; and selecting either the first or the second image information for texturing this region of the surroundings map, characterized in that the more current image information is selected when the vehicle (100) falls below a predetermined distance threshold to the region.

2. Method according to claim 1, wherein the more current image information is also used if a change in the vehicle environment and / or a predetermined object in the vehicle environment is detected within the region.

3. Method according to either claim 1 or claim 2, wherein the distance threshold is viewed in the direction of travel.

4. Method according to any of the preceding claims, wherein the surroundings map is generated during a journey of the vehicle (100).

5. Method according to any of the preceding claims, wherein the surroundings map is larger than a portion of the surroundings map that can be displayed on a display device.

6. Arrangement (1) for generating a surroundings map of a vehicle (100) textured with image information, comprising: a camera device (16) for capturing at least first and second image information at different times from a vehicle environment; and a surroundings map generating device (20) which is designed to generate a surroundings map of the vehicle (100) and to select either the first or the second image information for at least one region of the surroundings map for texturing the surroundings map, characterized in that the more current image information is selected when the vehicle (100) falls below a predetermined distance threshold to the region.

7. Vehicle (100), comprising an arrangement (1) according to claim 6.