Method, processing unit and vehicle for displaying a vehicle environment on a display device

Abstract

The invention relates to a method for displaying an environment of a vehicle (1) on a display device (7), comprising at least the following steps: - capturing the environment using at least two cameras, wherein each camera has a different field of view, wherein the fields of view of adjacent cameras overlap; - creating a panorama image (RB) from at least two individual images of different cameras, wherein the individual images are projected into a reference plane to create the panorama image (RB); - deriving depth information of at least one object (O) in the environment from at least two different individual images of the same camera by triangulation; - generating at least one overlay structure (20) in dependence on the derived depth information, wherein each overlay structure (20) is uniquely assigned to the imaged object (O); - visualizing the created panorama image (RB) containing at least one object (O) and at least one generated overlay structure (20) on the display device (7) such that the at least one overlay structure (20) is displayed on the respectively assigned object (O) and / or adjacent to the respectively assigned object.

Description

Technical Field

[0001] The present invention relates to a method for displaying the environment of a vehicle, particularly a commercial vehicle, on a display device, a processing unit for performing the method, and a vehicle. Background Technology

[0002] According to existing technology, panoramic images are created from individual images from multiple cameras and displayed to occupants on a display device. For this purpose, it is known to project individual images onto a reference plane, such as a horizontal plane beneath the vehicle, rotate accordingly, and thereby create a combined panoramic image. A drawback of this method is that objects protruding above the reference plane are displayed tilted backward in the panoramic image, thus distorting their size proportions. Consequently, the observer cannot intuitively interpret these objects geometrically. Orientation based on the panoramic image thus becomes difficult.

[0003] This impact can be minimized by using additional sensors to determine the height of the detected object, and compensating for any distortion in size proportions using the additional height information. The drawback here is that not all vehicles have such additional sensors, and / or these additional sensors are expensive. Furthermore, computational costs increase.

[0004] DE 10 2017 108 254 B4 illustrates, for example, the creation of an image synthesized from a single image and its display on a monitor. Here, each single image is captured by multiple cameras. Depth information about an object, particularly its location, can be obtained through triangulation of two or more images. Furthermore, the object can be tracked over time.

[0005] According to DE 10 2015 105 248 A1, in a two-part vehicle, a first camera is arranged on the first part of the vehicle, and a second camera is arranged on the second part of the vehicle. The first image from the first camera and the second image from the second camera are projected onto a bottom plane or reference plane through a homography matrix. Then, the first and second images are rotated to generate a combined image of the environment.

[0006] DE 100 35 223A1 describes the creation of a holistic or panoramic image from multiple individual images, where the vehicle itself is projected into the panoramic image as an artificial graphic object or overlay structure.

[0007] EP 3 293 700 B1 describes how a portion of the environment can be reconstructed from multiple individual images from a camera using a structure-of-motion motion method, thereby obtaining depth information for individual objects. Here, quality metrics are obtained to achieve an optimized reconstruction of the environment. This can also be implemented in panoramas using a combination of multiple cameras or individual images.

[0008] DE 10 2018 100 909 A1 describes how the structure-of-motion method is used to reconstruct the current environment and classify or categorize objects in a neural network. Summary of the Invention

[0009] Therefore, the objective of this invention is to describe a method for displaying a vehicle environment on a display device, which can be performed with relatively low hardware and computational costs and allows for easy orientation of the observer within the environment. This objective also describes a processing unit and a vehicle.

[0010] Therefore, according to the present invention, a method for displaying a vehicle environment on a display device and a processing unit for performing the method are provided, the method comprising at least the following steps:

[0011] First, the environment surrounding the vehicle is captured using at least two cameras, particularly in the adjacent area, with each camera having a different field of view. The fields of view of adjacent cameras overlap at least locally, particularly along the edges. Next, a panoramic image is created from the resulting individual images, each captured by a different camera at approximately the same time and projected onto a reference plane, for example, via a homography matrix.

[0012] Therefore, instead of creating a three-dimensional panoramic image from a single image, a panoramic image is created in which the environment is displayed as a two-dimensional projection by appropriately combining the individual images. Thus, if the panoramic image is displayed on a display device, a bird's-eye view of the environment can be achieved with the appropriate arrangement and orientation of the cameras.

[0013] In another step of the method according to the invention, depth information of at least one object in the acquired environment is obtained, wherein the depth information is obtained by triangulation from at least two different individual images from the same camera, preferably by a so-called structure-of-motion (SOR) method, wherein at least one object is preferably imaged from at least two different viewpoints in at least two different individual images. Thus, a stereo camera system is not used for obtaining the depth information. More precisely, a stereo reconstruction of the environment or object is obtained solely through image processing of a single image from the camera, from which the depth information can be derived.

[0014] At least one overlay structure is generated based on depth information previously obtained via the SfM method, wherein each overlay structure is uniquely assigned to the imaged object. In another step, the created panoramic image, comprising at least one object and at least one generated overlay structure, is displayed on a display device such that the at least one overlay structure is displayed on and / or adjacent to the corresponding assigned object.

[0015] Therefore, in an advantageous manner, an overlay structure can be displayed on a display device without additional sensors, solely based on image processing methods, particularly motion reconstruction. It can selectively display the location of objects on or adjacent to that location on the display device using previously known depth information. Even if the corresponding objects cannot be visually and geometrically interpreted within the panoramic image on the display device, the overlay structure can help achieve reliable orientation based on the panoramic image because it enhances or emphasizes important information about the objects on the display device. This can at least compensate for the distortion caused by protruding objects projected onto a reference plane in the panoramic image.

[0016] Additionally, this problem can be overcome by displaying an overlay structure, where objects in the overlapping area of ​​individual images from two adjacent cameras often "disappear" or are not fully perceived because these objects are tilted backward in their respective camera's field of view and typically do not even appear or are at least not fully perceived in the panoramic image. However, in the overlay structure, such objects are included according to the method described above and thus also appear in the panoramic image.

[0017] Therefore, it is preferably specified that, on the display device, bars and / or polygons and / or text are displayed as overlay structures on and / or adjacent to the corresponding associated objects. Thus, simple structures can also be displayed as overlays, sufficient to emphasize objects for orientation. For example, only bars can be specified, displayed on the display device on and / or adjacent to the outer edge of the corresponding associated object, wherein the bars are preferably perpendicular to the object normal of the corresponding associated object, which can be obtained from depth information. Here, the outer boundary of each object is understood as the outer edge, wherein the bars or their respective overlay structures are overlaid at least on the outer edge or boundary closest to the vehicle. Thus, the boundary of the object toward the vehicle can be identified by the bars in a panoramic image. This allows the observer to clearly see, for example, during parking or dispatching, where the vehicle can be dispatched or positioned to a specific point in space without touching the object.

[0018] As a supplement or alternative to the preferred specification of stripes, polygons, as overlay structures, are imaged on a display device such that the polygons at least partially, preferably completely, span the respective associated objects. This not only shows the boundaries of the respective objects toward the vehicle to the observer on the display device, but also identifies the extensions of the objects, thus emphasizing the objects more clearly on the display device. If the object contours and / or object shapes of the respective objects are known from environment reconstruction using the SfM method, the polygons can also be matched to the object contours or object shapes. If the object contours or object shapes are unknown, then, for example, rectangles are assumed as polygons, which cover the object points of the respective objects imaged in the panoramic image.

[0019] Preferably, at least one overlay structure is displayed on the display device in a predetermined color or in a color dependent on depth information known about the corresponding associated object. This also allows information about the spatial characteristics of the object to be provided to the observer through color coding. Preferably, the color of at least one overlay structure depends on the object spacing between the vehicle and the corresponding associated object, wherein the object spacing is obtained from depth information known about the corresponding associated object. This allows overlay structures on the display device that are farther from the vehicle to be given a color indicating lower danger, such as green. Overlay structures on the display device that are closer to the vehicle can be given a color indicating higher danger, such as red. Any color gradation dependent on the object spacing is feasible here.

[0020] Preferably, the color and / or type of the overlay structure of each object may also be specified to depend on the motion index assigned to the object, wherein the motion index indicates whether the respective object is capable of movement, such as a person or vehicle, or is permanently stationary, such as a building or streetlight, wherein the motion index is obtained from depth information known about the corresponding assigned object. This allows for supplementary emphasis on the display device of what additional dangers might arise based on the potential movement of the displayed objects when objects cannot be directly and clearly identified due to distortion in the panoramic image.

[0021] Preferably, the method may specify that the object outline and / or object shape of the corresponding associated object are obtained from depth information, and motion indicators for the object are derived from the object outline and / or object shape by means of a deep learning algorithm through comparison with known object outlines and / or known object shapes. This eliminates the need for computationally complex object tracking, and instead uses comparisons with known objects stored in the vehicle or accessible by the vehicle through a corresponding data connection.

[0022] Alternatively or supplementarily (for credibility checks), it may be specified that object points imaged in a single image are tracked over a time period (zeitlich) to derive motion indicators for the relevant object. For example, the behavior of individual image points over a time period can be determined by the formation of difference images, and the motion of the object can be inferred from there.

[0023] Preferably, at least one overlay structure is displayed opaquely or at least partially transparently on the display device, such that at least one overlay structure completely or at least partially covers the panoramic image with respect to transparency on and / or adjacent to the corresponding associated object. Thus, each object can be emphasized by the overlay structure, while the observer can simultaneously identify objects located behind it, allowing for automatic confidence checks of potential hazards from the objects if necessary. Advantageously, the transparency of the overlay structure can also be determined, similar to color coding, based on depth information about the objects. For example, objects with a larger distance from the vehicle can be displayed with higher transparency than those with a smaller distance, thereby more clearly emphasizing more relevant objects.

[0024] Preferably, the display device has display pixels, wherein panoramic image points of the panoramic image are displayed on the display pixels of the display device, wherein objects included in the panoramic image are displayed on object pixels, wherein object pixels are a subset of display pixels, and wherein overlay structures assigned to their respective objects are displayed or overlaid on their respective object pixels and / or displayed or overlaid adjacent to their respective object pixels on the display device. Preferably, the overlay of the overlay structures is achieved by overlaying an overlay image having at least one overlay structure onto a panoramic image having at least one object on the display device, such that the overlay structures assigned to their respective objects are displayed on their respective object pixels and / or displayed adjacent to their respective object pixels on the display device.

[0025] Therefore, the following preprocessing is first performed based on the depth information: creating a superimposed image in which the superimposed mechanism is preferably displayed only at the following locations in the panoramic image of the object, where the superimposed image is imaged or imaged adjacent to these locations. The two images can be simultaneously displayed on a display device by addition, multiplication, or any other operation to achieve the superposition according to the invention.

[0026] Alternatively or supplementarily, it may be specified that the panoramic image itself contains at least one overlay structure, wherein the panoramic image is adjusted at panoramic image points on which the object is imaged and / or adjacent to panoramic image points on which the object is imaged, such that the overlay structure assigned to each respective object is displayed on the display device on the respective object pixel and / or adjacent to the respective object pixel. In this case, the display device transmits only the image for display, and the overlay structure is "provided" in advance accordingly by "manipulating" the image points so that the overlay structure is displayed in the image.

[0027] Preferably, at least two individual images in which depth information is obtained by triangulation are acquired by the same camera from at least two different viewpoints, wherein the depth information is obtained by triangulation depending on the length of the base between at least two viewpoints. This ensures that depth information is reliably obtained within the scope of the SfM method. Preferably, the camera may be introduced to different viewpoints either through the movement of the vehicle itself (i.e., changes in vehicle dynamics) or by actively adjusting the camera without changes in vehicle dynamics.

[0028] Preferably, the environment is imaged within a panoramic view, where the panoramic view is larger than the field of view of a single camera, the panoramic view has a 360° angle, and the panoramic image is composed of at least two single images acquired almost simultaneously by different cameras. This allows for seamless imaging of the entire environment surrounding the vehicle at the current point in time, along with a superposition structure overlaying the entire panoramic field of view.

[0029] Preferably, the contour lines assigned to the vehicle can also be displayed on the display device as an additional overlay structure, wherein the contour lines are displayed at fixed contour intervals relative to the outer edge of the vehicle. Therefore, the vehicle itself can also be considered an object assigned to the overlay mechanism, allowing the observer to orient themselves according to these mechanisms. Here, the positioning of the contour lines relies on spatial information obtained from a single image. For example, contour lines can be displayed around the vehicle at 1-meter intervals. Depending on the contour intervals, the overlay structure can also be color-coded using color gradations, displaying contour lines closer to the vehicle in red and those farther away in green, while using corresponding color gradients for contour lines in between. The transparency of the contour lines can also vary depending on the contour intervals, allowing the observer to easily identify the distance relative to the vehicle.

[0030] Therefore, the vehicle according to the invention, in which the method according to the invention is performed, has at least two cameras, each camera having a different field of view, wherein the fields of view of adjacent cameras overlap at least partially, particularly at the edge side. The vehicle is also provided with a display device and a processing unit according to the invention, wherein the display device is configured to display a created panoramic image comprising at least one object and at least one generated overlay structure (as part of a panoramic image or a separate overlay image), such that at least one overlay structure is displayed on and / or adjacent to the corresponding associated object.

[0031] Preferably, each individual camera has a field of view of 120° or greater, particularly 170° or greater, wherein the camera is designed, for example, as a fisheye camera, and is positioned on at least two sides of the vehicle selected from the group consisting of the front, rear, or at least one longitudinal side. This arrangement allows for near-seamless acquisition of the environment at the corresponding field of view, enabling its display in a bird's-eye view. For this purpose, the individual cameras are preferably aimed at the adjacent area of ​​the environment, i.e., the ground, to achieve a bird's-eye view display. Attached Figure Description

[0032] The present invention is further illustrated below with reference to embodiments. Wherein:

[0033] Figure 1 A schematic diagram of a vehicle for performing the method according to the invention is shown;

[0034] Figure 2 A detailed view of a single captured image is shown;

[0035] Figure 2a A detailed view of the object is shown, imaged in two single images from a single camera.

[0036] Figure 2b It shows according to Figure 1 Detailed images of the display devices inside the vehicle; and

[0037] Figure 3 A detailed view of the environment displayed on the display device is shown. Detailed Implementation Plan

[0038] Figure 1 The illustration schematically depicts a vehicle 1, particularly a commercial vehicle, which, according to the illustrated embodiment, has a front-facing camera 3a on the front side 2a (e.g., within the roof liner) and a rear-view camera 3b on the rear side 2b. Furthermore, a side-facing camera 3c is arranged on the longitudinal side 2c of the vehicle 1, for example, on a rearview mirror. Other cameras 3 (not shown) may also be installed within the vehicle 1 to detect the environment U, particularly the adjacent area N (the environment up to 10m away from the vehicle 1).

[0039] Each camera 3 has a field of view 4, wherein the front field of view 4a of the front camera 3a is oriented forward, the rear field of view 4b of the rear space camera 3b is oriented rearward, and the lateral field of view 4c of the side camera 3c is oriented laterally relative to the vehicle 1. In order to detect relevant parts of the environment U, especially the adjacent area N, the camera 3 is oriented towards the ground on which the vehicle 1 moves.

[0040] The number and positioning of the cameras 3 are preferably chosen such that the fields of view 4 of adjacent cameras 3 overlap within the neighboring region N, so that all fields of view 4 together can cover the neighboring region N without gaps and thus completely. For this purpose, the cameras 3 can be implemented, for example, as fisheye cameras, each covering the field of view 4 with a visible angle W equal to or greater than 170°.

[0041] Each camera 3 transmits an image signal SB, which characterizes the environment U imaged on the sensor of its respective camera 3 within the field of view 4. The image signals SB are sent to a processing unit 6, which is configured to generate a single image EBk (with consecutive subscripts k) based on the image signal SB from each camera 3. Here, the k-th single image EBk is based on... Figure 2 A single image point EBkPi (with consecutive subscripts i from 0 to Ni) is formed on which the environment U is imaged. Here, according to Figure 2a An object point PPn (with consecutive subscripts n) belonging to an object O in environment U is assigned to a specific single image point EBkPi.

[0042] In processing unit 6, a single image EBk is projected onto a reference plane RE, for example, onto a horizontal plane below vehicle 1 (see Parallel to Through). Figure 2a In the plane (the unfolded plane of xO and yO), through perspective transformation, such as relying on the homography matrix, and with the aid of the panoramic algorithm A1, a panoramic image RB with a number of Np panoramic image pixels RBPp (with consecutive subscripts p from 0 to Np) is created from a single image EBk of different cameras 3. In the panoramic image RB, the environment U around the vehicle 1 is imaged seamlessly in all directions, at least within the neighboring region N (see...). Figure 3 This corresponds to a 360° field of view W. Therefore, a panoramic range 4R is obtained for the panoramic image RB, which is greater than the single field of view 4 of camera 3. This panoramic image RB of the adjacent region N can be output to the occupant, such as the driver of vehicle 1, on the display device 7, so that the driver can, for example, or during parking or dispatching, or according to their orientation. This allows a bird's-eye view of the environment U to be shown to the observer.

[0043] Here, according to Figure 2b The display device 7 has Nm display pixels APm (with consecutive subscripts m from 0 to Nm), where each panoramic image pixel RBPp is displayed on a specific display pixel APm of the display device 7, thus obtaining a panoramic image RB that can be seen on the display device 7 by an observer. A dynamic subset of the display pixels APm is the object pixels OAPq (with consecutive subscripts q), on which objects O from the environment U are displayed (only on objects O from the environment U). Figure 2b(Illustrated in the image). The object pixel OAPq is assigned a panoramic image point RBPp, and a specific object O or a specific object point PPn from the environment U is imaged on it.

[0044] In this scenario, distortion occurs at the edges of the panoramic image RB based on the applied panoramic algorithm A1. To address this issue, according to the present invention, it is proposed to overlay the created panoramic image RB with other information derived from the depth information TI for the imaging object O.

[0045] Here, depth information TI is obtained from multiple individual images EBk from a single camera 3 using the so-called Structure of Motion (SfM) method. The extraction of depth information TI is performed individually for each camera 3, depending on the camera configuration. In the SfM method, the associated 3D object O in the environment U, having its object point PPn, is acquired by the corresponding camera 3 from at least two different viewpoints SP1, SP2, such as... Figure 2a As shown. Next, depth information TI can be obtained about the corresponding 3D object O using triangulation method T:

[0046] Therefore, image coordinates xB and yB are determined for at least one first single image point EB1P1 in a first single image EB1 of the front-facing camera 3a and at least one first single image point EB2P1 in a second single image EB2 of the front-facing camera 3a. The two single images EB2 are acquired by the front-facing camera 3a from different viewpoints SP1 and SP2, meaning that the vehicle 1 or the front-facing camera 3a moves a base length L between the single images EB1 and EB2. The two first single image points EB1P1 and EB2P1 are selected in their respective single images EB1 and EB2 as follows: they are assigned to the same object point PPn on the corresponding imaged three-dimensional object O.

[0047] In this way, one or more pairs of individual image points EB1Pi, EB2Pi can be calculated for one or more object points PPn in environment U. To simplify the process, a certain number of individual image points EB1Pi, EB2Pi in their respective individual images EB1, EB2 can be combined into feature points MP1, MP2 (see [link to documentation]). Figure 2 In this process, the individual image points EB1Pi and EB2Pi to be combined are selected as follows: their respective feature points MP1 and MP2 are assigned to a definitively locatable feature M on the 3D object O. Feature M can be, for example, a corner ME or an outer edge MK on the 3D object O (see [link to documentation]). Figure 2a ).

[0048] Following an approximation approach, the absolute, actual object coordinates (xO, yO, zO) of the 3D object O, object point PPj, or feature M are calculated or estimated using triangulation T from the image coordinates xB and yB obtained for their respective object O from individual image points EB1Pi, EB2Pi or feature points MP1, MP2. To perform triangulation T, the corresponding known base length L between the viewpoints SP1 and SP2 of the front-facing camera 3a is used.

[0049] Next, if triangulation T has been performed on a sufficient number of object points PPi or features M of object O, the position and orientation of vehicle 1 relative to its respective 3D object O can be determined based on the obtained object coordinates xO, yO, zO, using geometric observation. If the precise object coordinates xO, yO, zO of multiple object points PPj or features M of object O are known, the processing unit 6 can estimate at least the object shape FO and / or object contour CO based on this. The object shape FO and / or object contour CO can then be fed to the deep learning algorithm A2 for further processing.

[0050] As described, the object O and its coordinates xO, yO, zO can also be detected by other cameras 3 inside vehicle 1, and its location and orientation in space can be determined accordingly.

[0051] To obtain depth information TI more accurately, it can be further specified that more than two individual images EB1, EB2 are acquired using the corresponding camera 3, and evaluated and / or bundle adjustment is performed by triangulation method T as described above.

[0052] As already described, for the SfM method, object O is observed from at least two different viewpoints SP1 and SP2 of the corresponding camera 3, such as... Figure 2a This is illustrated schematically. For this purpose, the corresponding camera 3 is controlled to move to different viewpoints SP1 and SP2. Here, by combining the odometer data OD, it can be determined what base length L is obtained between viewpoints SP1 and SP2 based on this movement. Different methods can be applied for this:

[0053] If the entire vehicle 1 is in motion, then the motion of the corresponding camera 3 is thus obtained. This is understood as the vehicle 1 being in motion as a whole, actively (e.g., through the drive system) or passively (e.g., due to a downhill slope). If at least two individual images EB1, EB2 are captured by the corresponding camera 3 within a time deviation during the motion, the base length L can be determined with the help of the odometer data OD. From the odometer data, the vehicle motion can be deduced, and thus the camera motion can be deduced. The two viewpoints SP1, SP2 assigned to the individual images EB1, EB2 are obtained through the odometer.

[0054] The wheel speed signals S13 from the active and / or passive wheel speed sensors 13 on the wheels of vehicle 1 can, for example, be used as odometer data OD. Based on these signals, it can be determined, depending on the time deviation, how far vehicle 1 or the corresponding camera 3 has moved between viewpoints SP1 and SP2, with the base length L corresponding to this distance. To make the odometer determination of the base length L more accurate as vehicle 1 moves, other odometer data OD available within vehicle 1 can be used. For example, the steering angle LW and / or yaw rate G, determined accordingly by sensing or analysis, can be used to also take into account the rotational motion of vehicle 1.

[0055] However, it is not mandatory to use only vehicle odometers; that is, vehicle motion can be assessed in conjunction with motion sensors on vehicle 1. Visual odometers can also be used as an alternative. In the case of visual odometers, the camera position can be continuously determined from the image signal SB of the corresponding camera 3 or from information in the detected individual images EB1, EB2, provided that the object coordinates xO, yO, zO of the determined object point PPn are known at least initially. Therefore, the odometer data OD can also contain a correlation with the camera position thus obtained, because the vehicle motion between the two viewpoints SP1, SP2 can be deduced from this, or the base length L can be directly deduced.

[0056] However, in principle, it is also possible to specify active adjustment of camera 3 without changing the motion state of the entire vehicle 1. Therefore, any movement of the corresponding camera 3 is possible to introduce different viewpoints SP1, SP2 in a controlled and measurable manner.

[0057] Next, relying on the depth information TI obtained for a given object O through the SfM method, the panoramic image RB can be generated as follows: Figure 3 As shown, it is superimposed with the superposition structure 20. Superposition can be implemented as follows: the display device 7 transmits the panoramic image RB via the panoramic image signal SRB and the superimposed image OB with the corresponding superposition structure 20 via the superposition signal SO. The display device 7 displays the two images RB and OB on the corresponding display pixels APm, for example, through pixel addition, pixel multiplication, or any other pixel operation. Alternatively, the panoramic image RB can also be directly changed or adjusted at the corresponding panoramic image point RBPp in the processing unit 6, so that the display device 7 transmits the panoramic image RB containing the superposition structure 20 via the panoramic image signal SRB for display.

[0058] Here, the additional overlay structure 20 is uniquely assigned to a specific object O in the environment U. This allows the observer to view additional information about the corresponding object O, making orientation within the environment U based on the display more convenient. The overlay structure 20 can be, for example, a bar 20a and / or a polygon 20b and / or text 20c, and additionally, they can be encoded based on the corresponding depth information TI.

[0059] The overlay is implemented such that the overlay structure 20 appears on or adjacent to the object pixel OAPq assigned to object O on the display device 7. The corresponding object pixel OAPq can be dynamically identified by the processing unit 6 through the panoramic algorithm A1. Based on this, an overlay image OB can be created or the panoramic image points RBPp of the panoramic image RB can be directly changed or adjusted, so that the corresponding overlay structure 20 appears on or adjacent to the corresponding object O on the display device 7.

[0060] The bar 20a can, for example, be displayed on the display pixel APm of the display device 7. This bar is located on or adjacent to the outer edge MK of the corresponding object O, which is closest to the vehicle 1. The orientation of the bar 20a can be chosen such that the bar 20a is perpendicular to the object normal ON, as shown below. Figure 3 As shown, if object O does not have, for example, a straight outer edge MK, then stripe 20a indicates the outer boundary of object O in all cases. The object normal ON can be estimated from the depth information TI for that object, i.e., from the location and orientation, which is derived by the SfM method.

[0061] To emphasize the positioning of the strip 20a assigned to object O on the display device 7, the object pixel OAPq of object O, which is also assigned to strip 20a, can be colored with a fixed color F. This makes object O itself appear clearer, thus making any distortions that might occur when displaying object O less noticeable. Within the range of object pixel OAPq, as another overlay structure 20, a polygon 20b having object shape OF or object outline OC is overlaid onto the panoramic image RB with a defined color F. If the object shape OF or object outline OC cannot be explicitly determined in the SfM method, it can also be assumed that object O is a rectangle extending "behind" strip 20a when viewed from the vehicle 1. In this case, the additional overlay structure 20 is a polygon 20b with four corners.

[0062] For example, black can be chosen as color F. However, color F can also depend on the object spacing OA with respect to the corresponding object O. The bar 20a itself can be encoded in terms of color depending on the object spacing OA with respect to the corresponding object O. Here, the object spacing OA between vehicle 1 and object O is also obtained from the depth information TI about the object obtained through the SfM method, that is, from the localization and orientation.

[0063] If the object spacing OA is less than 1m as determined in the SfM method, the color F of the corresponding overlay structure 20 (i.e., stripe 20a and / or polygon 20b) is displayed as a warning color, particularly red. If the object spacing OA of object O is in the range of 1m to 5m, yellow can be used as the color F for the overlay structure 20 assigned to that object O. When the object spacing OA is greater than 5m, green can be specified as the color F. In this way, it can be more clearly shown to the observer which risk comes from the corresponding object O. Because the depth information TI is obtained from a single image EBk, the distortions originating from the panorama algorithm A1 have no effect on the overlay structure 20, and therefore these distortions can be displayed from the vehicle 1 at the correct location on the display device 7.

[0064] Furthermore, the corresponding overlay structure 20 can be displayed opaquely or at least partially transparently on the display device 7, so that at least one overlay structure 20 completely or at least partially covers the panoramic image RB on and / or adjacent to the corresponding associated object O in terms of transparency.

[0065] Additionally, the color F of the overlay structure 20 can be selected based on the motion index B. Therefore, the object contour OC and / or object shape OF for the corresponding identified object O can be obtained according to the SfM method as described. However, the dynamics of object O cannot be directly inferred from the SfM method. However, if the object contour OC and / or object shape OF are fed to the deep learning algorithm A2 in the processing unit 6, at least one classification of object O can be performed, thereby inferring the possible dynamics of object O.

[0066] Here, the object outline OC and / or object shape OF of the corresponding object O can be compared with known objects. Known objects can be stored in a database, for example, stored in a memory fixed relative to the vehicle or accessible via a mobile data connection of vehicle 1. Based on the entries of known objects in the database, it can be determined whether the detected object O is a person, building, vehicle, etc. For this purpose, each detected object O can be assigned a motion index B stored in the database, indicating whether and how the object O moves in the environment U in a normal manner. From this, it can be inferred whether attention should be increased for the object O (e.g., in the case of a person). Accordingly, overlay structures 20 (e.g., bars 20a and / or polygons 20b) or other overlay structures 20 can be encoded corresponding to the motion index B, for example, in terms of color. Additionally, text 20c, as another overlay structure 20, can be displayed, for example, in the form of "!" (exclamation mark).

[0067] However, the motion index B of object O can also be estimated by tracking individual image points EBkPi of the object point PPn assigned to the environment U in time. This can be achieved, for example, by analyzing the differences in pixels of consecutive individual images EBk. The motion of the corresponding object O can then be inferred.

[0068] In addition, contour lines 20d (see also) can be superimposed as superposition structure 20. Figure 3 ), which represent fixed equal spacing AI relative to the outer side 1a of the vehicle.

[0069] List of reference numerals (part of the instruction manual)

[0070] 1 vehicle

[0071] 1a Vehicle exterior

[0072] 2a Anterior side

[0073] 2b Rear side

[0074] 2c Longitudinal side

[0075] 3 cameras

[0076] 3a Front-facing camera

[0077] 3b Rear Space Camera

[0078] 3C side-facing camera

[0079] 4 Field of View

[0080] 4a Forward field of view

[0081] 4b Rear spatial field of view

[0082] 4c Side View

[0083] 4R panoramic range

[0084] 6 processing units

[0085] 7 Display devices

[0086] 13 Wheel speed sensor

[0087] 20. Superimposed Structure

[0088] 20a bar

[0089] 20b polygon

[0090] 20c text

[0091] 20d contour lines

[0092] A1 Panoramic Algorithm

[0093] A2 Deep Learning Algorithm

[0094] AI equal spacing

[0095] APm is the m-th display pixel.

[0096] B. Exercise Indicators

[0097] EBk The kth single image from camera 3

[0098] EBkPi is the i-th single image point of the k-th single image EBk.

[0099] F color

[0100] G yaw rate

[0101] L Base length

[0102] LW steering angle

[0103] M features

[0104] ME corner

[0105] MK outer edge

[0106] Feature points in the first single image E1 of MP1

[0107] Feature points in MP2, the second single image E2

[0108] N neighboring areas

[0109] Ni: Number of image points EBkPi per image

[0110] Nm displays the number of pixels APm.

[0111] The number of pixels RBPp in a panoramic image (Np)

[0112] O object

[0113] OAPq, the qth object pixel

[0114] OA object spacing

[0115] OB overlay image

[0116] OC object outline

[0117] OD odometer data

[0118] OF object shape

[0119] ON object normal

[0120] PPn is the nth object point of object O.

[0121] RB panoramic image

[0122] RBPp, the p-th panoramic image point

[0123] RE Reference Plane

[0124] S13 Wheel Speed ​​Signal

[0125] SB image signal

[0126] SO superimposed signal

[0127] Viewpoints of SP1 and SP2 cameras 3

[0128] SRB panoramic image signal

[0129] TI Deep Information

[0130] T-triangulation method

[0131] U Environment

[0132] xB, yB image coordinates

[0133] xO, yO, zO object coordinates

[0134] subscripts i, k, m, n, p, q

Claims

1. A method for displaying the environment (U) of a vehicle (1) on a display device (7), the method comprising at least the following steps: - The environment (U) is captured using at least two cameras (3), wherein, Each camera (3) has a different field of view (4), wherein the fields of view (4) of adjacent cameras (3) overlap at least partially; - A panoramic image (RB) is created as a two-dimensional projection from at least two individual images (EBk), wherein each individual image (EBk) is acquired by a different camera (3) and the individual images (EBk) are projected onto a reference plane (RE) to create the panoramic image (RB); - Using the SfM method, i.e., structure for motion recovery, depth information (TI) of at least one object (O) in the acquired environment (U) is obtained, wherein the depth information (TI) is obtained by triangulation (T) from at least two different individual images (EBk) of the same camera (3), wherein the at least one object (O) is imaged in the at least two different individual images (EBk); and - At least one overlay structure (20) is generated based on the known depth information (TI), wherein each overlay structure (20) is uniquely assigned to the imaged object (O); and - A created panoramic image (RB) comprising the at least one object (O) and at least one generated overlay structure (20) is displayed on the display device (7), such that the at least one overlay structure (20) is displayed on and / or adjacent to the corresponding associated object (O).

2. The method according to claim 1, characterized in that, The vehicle (1) is a commercial vehicle.

3. The method according to claim 1, characterized in that, On the display device (7), the bar (20a) is displayed as a superimposed structure (20) on the corresponding associated object (O) and / or adjacent to the corresponding associated object.

4. The method according to claim 3, characterized in that, The bar (20a) is displayed on the display device (7) on the outer edge (MK) of the corresponding associated object (O) and / or adjacent to the outer edge of the corresponding associated object.

5. The method according to claim 4, characterized in that, The strip (20a) is perpendicular to the object normal (ON) of the corresponding associated object (O), wherein the object normal (ON) is obtained from depth information (TI).

6. The method according to claim 1, characterized in that, On the display device (7), polygons (20b) and / or text (20c) are displayed as overlay structures (20) on the corresponding associated objects (O) and / or adjacent to the corresponding associated objects.

7. The method according to claim 6, characterized in that, On the display device (7), the polygon (20b) is imaged as a superimposed structure (20) such that the polygon (20b) at least partially spans the corresponding associated object (O).

8. The method according to claim 6, characterized in that, On the display device (7), the polygon (20b) is imaged as a superimposed structure (20) such that the polygon (20b) completely spans the corresponding associated object (O).

9. The method according to any one of claims 1 to 8, characterized in that, The at least one overlay structure (20) is displayed on the display device (7) in a predetermined color (F) or in a color (F) that depends on the depth information (TI) known about the corresponding associated object (O).

10. The method according to claim 9, characterized in that, The color (F) of the at least one overlay structure (20) depends on the object spacing (OA) between the vehicle (1) and the corresponding associated object (O), wherein the object spacing (OA) is obtained from the depth information (TI) known about the corresponding associated object (O).

11. The method according to claim 9, characterized in that, The color (F) and / or type of the overlay structure (20) for each object (O) depends on the motion index (B) assigned to the object (O), wherein the motion index (B) indicates whether the object (O) is capable of movement or remains stationary, wherein the motion index (B) is obtained from the depth information (TI) known about the corresponding assigned object (O).

12. The method according to claim 11, characterized in that, The object outline (OC) and / or object shape (OF) of the corresponding associated object (O) are obtained from the depth information (TI), and the motion index (B) for the relevant object (O) is derived from the object outline (OC) and / or the object shape (OF) by means of a deep learning algorithm (A2) through comparison with the known object outline (OC) and / or the known object shape (OF).

13. The method according to claim 11, characterized in that, The object points (PPn) on the object (O) are tracked over time and imaged in a single image (EBk) in order to derive motion indices (B) for the relevant object (O).

14. The method according to any one of claims 1 to 8, characterized in that, The at least one overlay structure (20) is displayed opaquely or at least partially transparently on the display device (7), such that the at least one overlay structure (20) completely or at least partially covers the panoramic image (RB) on and / or adjacent to the corresponding associated object (O).

15. The method according to any one of claims 1 to 8, characterized in that, The display device (7) has display pixels (APm), wherein panoramic image points (RBPp) of the panoramic image (RB) are displayed on the display pixels (APm) of the display device (7), wherein objects (O) contained in the panoramic image (RB) are displayed on object pixels (OAPq), wherein the object pixels (OAPq) are a subset of the display pixels (APm). The superposition structure (20) assigned to each object (O) is displayed on the display device (7) on the respective object pixel (OAPq) and / or adjacent to the respective object pixel.

16. The method according to claim 15, characterized in that, An overlay image (OB) having at least one overlay structure (20) is overlaid on the display device (7) onto a panoramic image (RB) having at least one object (O), such that the overlay structure (20) assigned to each object (O) is displayed on the display device (7) on the respective object pixel (OAPq) and / or adjacent to the respective object pixel.

17. The method according to claim 15, characterized in that, The panoramic image (RB) includes at least one overlay structure (20), wherein the panoramic image (RB) is adjusted at and / or adjacent to the panoramic image point (RBPp) on which the object (O) is imaged, such that the overlay structure (20) assigned to the respective object (O) is displayed on the display device (7) on the respective object pixel (OAPq) and / or adjacent to the respective object pixel.

18. The method according to any one of claims 1 to 8, characterized in that, The at least two individual images (EBk) are acquired from at least two different viewpoints (SP1, SP2) by the same camera (3), from which depth information (TI) is obtained by triangulation (T), wherein the depth information (TI) is obtained by triangulation (T) depending on the length (L) of the base between the at least two viewpoints (SP1, SP2).

19. The method according to any one of claims 1 to 8, characterized in that, The environment (U) within the panoramic range (4R) is imaged in the panoramic image (RB), wherein the panoramic range (4R) is greater than the field of view (4) of a single camera (3), and wherein the viewing angle (W) of the panoramic range (4R) is 360°. The panoramic image (RB) consists of at least two single images (EBk) acquired almost simultaneously by different cameras (3).

20. The method according to any one of claims 1 to 8, characterized in that, The contour lines (20d) assigned to the vehicle (1) are displayed on the display device (7) as a superposition structure (20), wherein the contour lines (20d) are displayed at a fixed contour interval (AI) relative to the outer side (1a) of the vehicle (1).

21. A processing unit (6) for performing the method according to any one of claims 1 to 20.

22. A vehicle (1) having at least two cameras (3), wherein, Each camera (3) has a different field of view (4), wherein the fields of view (4) of adjacent cameras (3) overlap at least partially, and the vehicle has a display device (7) and a processing unit (6) according to claim 21. The display device (7) is configured to display a created panoramic image (RB) containing at least one object (O) and at least one generated overlay structure (20), such that the at least one overlay structure (20) is displayed on and / or adjacent to the corresponding associated object (O).

23. The vehicle (1) according to claim 22, characterized in that, Each individual camera (3) has a field of view (4) with a viewing angle (W) greater than or equal to 120°.

24. The vehicle (1) according to claim 22, characterized in that, Each individual camera (3) has a field of view (4) with a viewing angle (W) greater than or equal to 170°.

Images