Method and camera system for providing a vehicle's surroundings display

Abstract

The invention relates to a method and a camera system for providing an environment display of a vehicle.

Description

[0001] The invention relates to the field of vehicle environment monitoring systems. In particular, the invention relates to a method and a camera system for providing a representation of a vehicle's environment.

[0002] Vehicle surround-view systems are generally known. In particular, surround-view systems that provide a three-dimensional, perspective 360° view around the vehicle are known, so-called 3D surround-view systems. In such systems, the image information from multiple cameras is combined to create a panoramic view and mapped onto a three-dimensional surface as a projection surface to obtain a spatial perspective representation. The three-dimensional surface can, for example, be bowl-shaped. The representation on such a bowl-shaped surface is referred to as a 3D bowl display.

[0003] The spatial perspective representation of the vehicle's surroundings can be displayed from different perspectives in known systems. The position and orientation of a virtual camera—that is, the position and viewing direction from which a virtual observer sees the scene—can be changed, allowing different areas of the vehicle's environment to be viewed. In particular, the position and orientation of the virtual camera can be chosen, for example, so that both an area of ​​the surroundings and the vehicle within that area are visible.

[0004] It is also known that in adaptive 3D bowl rendering, static objects that protrude upwards from the surface on which the vehicle is standing and thereby penetrate the wall area of ​​the projection surface are distorted. This can lead to disturbing distortions in the environmental representation, such as kinks in objects protruding from the standing surface, also known as the Manhattan effect, which is perceived as disturbing by users of the assistance system and leads to a lack of user acceptance.

[0005] Based on this, the object of the invention is to provide a method for providing an environment representation of a vehicle that enables a realistic and at the same time as comprehensive a representation of the vehicle's environment in a technically simple and highly reliable manner.

[0006] The problem is solved by a method having the features of independent claim 1. Preferred embodiments are the subject of the dependent claims. A camera system is the subject of dependent claim 13.

[0007] According to a first aspect, a method for providing an environment representation of a vehicle is disclosed. The environment representation is generated based on image information provided by a camera system with multiple cameras distributed around the vehicle. The method comprises the following steps, the order of which may differ, at least partially, from the sequence described below: First, image information is captured using multiple cameras of the camera system. The image information depicts different areas of the vehicle's surroundings and contains information about at least one object in the vehicle's vicinity.

[0008] Subsequently, objects in the vehicle's vicinity and their spatial position are determined. This object detection is performed using the vehicle's sensors. These sensors can include the cameras of the camera system as well as other sensors, such as at least one radar sensor, at least one LiDAR sensor, and / or ultrasonic sensors.

[0009] Based on the identified objects, an object map is created, containing the objects according to their location. The object map can be, for example, a two-dimensional or three-dimensional grid map.

[0010] Furthermore, a circumferentially closed first projection surface with a basic shape is provided. This first projection surface forms a starting point for determining a second projection surface, which may have different dimensions than the first projection surface, but retains the basic shape of the first projection surface.

[0011] Based on the object map and the first projection surface, the second projection surface is determined. This is done by: - the surface area of ​​the second projection surface is maximized, while the basic shape of the first projection surface is preserved; - the vehicle is completely within the space enclosed by the second projection surface; and - no objects contained in the object map lie within the second projection surface or partially extend into it.

[0012] It should be noted that the second projection surface does not have to be centered on the position of the vehicle; the vehicle can also be positioned off-center relative to the second projection surface.

[0013] The image information provided by multiple cameras is then combined. This combining process is also known to experts as stitching. A 360-degree view of the vehicle's surroundings is created from the combined image information. The combined image information is then projected onto a second projection surface to obtain the perspective view of the environment.

[0014] The technical advantage of the proposed method is that by adjusting the projection surface, a representation of the vehicle's environment can be determined in a time- and computing-capacity-efficient manner, and a realistic and at the same time as comprehensive a representation of the vehicle's surroundings is also made possible.

[0015] According to one embodiment, the basic shape of the first projection surface is selected from several basic shape variants. These variants can include, for example, a circular, elliptical, or rectangular shape (possibly with rounded corners). The selection of a suitable basic shape in each case can depend on the surrounding situation and / or the driving maneuver. For example, when reversing into a parking space, a basic shape can be chosen that best represents the parking space into which the vehicle is to be moved (e.g., an elliptical or rectangular shape), whereas for a driving maneuver where the direction of travel has not yet been determined, a basic shape is chosen that allows the vehicle's surroundings to be clearly represented in all spatial directions (e.g., a circular shape).

[0016] According to one embodiment, determining the second projection surface involves adjusting the location, orientation, and / or size of the first projection surface. These adjustments allow a second projection surface to be determined that is adapted to the specific environmental conditions and driving maneuver.

[0017] According to one embodiment, adjusting the local position of the first projection surface includes moving the first projection surface in a plane parallel to the surface on which the vehicle is located.

[0018] According to one embodiment, adjusting the orientation of the first projection surface includes rotating the first projection surface about a vertical axis of rotation.

[0019] According to one embodiment, adjusting the size of the first projection surface includes at least a partial, parallel displacement of edge sections of the first projection surface and / or a change in the radius of curvature of a curved edge section of the first projection surface.

[0020] According to one embodiment, the local position, orientation, and / or size of the first projection surface is adjusted such that the vehicle's circumferential contour maintains a predetermined minimum distance from the first projection surface. This ensures that the image information provided by the camera system can still be projected with sufficient accuracy even onto projection surface areas located very close to the vehicle.

[0021] According to one embodiment, the surface area of ​​the second projection surface is maximized such that a maximum distance between diametrically opposed edge sections of the second projection surface is not exceeded. This ensures that an upper limit exists for maximizing the surface area of ​​the second projection surface, which is also reasonable with regard to the detection range of the sensors used for environmental perception.

[0022] According to one embodiment, after determining the second projection surface, the position, orientation, and / or size of the second projection surface remain at least temporarily unchanged when the vehicle moves, and the vehicle's position within the second projection surface changes. In other words, the second projection surface is not changed as the vehicle moves but remains stationary for at least a certain period (for example, as long as the vehicle is within the projection surface). This minimizes the computational effort required to calculate the second projection surface.

[0023] According to one embodiment, the 360° view is created by projecting images captured by the vehicle's cameras as it moves onto the second projection surface. In other words, as the vehicle moves, new images are iteratively projected onto the second projection surface, which is at least temporarily stationary, thus updating the view of the surroundings.

[0024] According to one embodiment, the second projection surface is adjusted in position, orientation, and / or size if the vehicle, due to its movement relative to the second projection surface, falls below a minimum distance to the edge of the second projection surface. In this case, the search algorithm can be run again to determine a second projection surface suitable for the changed environmental conditions.

[0025] According to one embodiment, the first and second projection surfaces have a three-dimensional, shell-like basic shape. This allows for a realistic representation of the environment.

[0026] According to another aspect, a camera system for a vehicle is disclosed, comprising several cameras distributed around the vehicle and a control unit for processing the image information provided by the cameras to generate an environmental representation of the vehicle's surroundings. The camera system is configured to perform the following steps: - Capturing image information using multiple cameras of the camera system; - Identifying objects in the vicinity of the vehicle and their spatial location; - Creating an object map that includes the identified objects according to their location; - Providing a circumferentially closed first projection surface with a basic shape; - Based on the object map and the first projection surface, determine a second projection surface such that: - the surface area of ​​the second projection surface is maximized while maintaining the basic shape; - the vehicle is completely within the space enclosed by the second projection surface; and - no objects contained in the object map lie within the second projection surface or partially extend into it; - Creating a 360° view of the vehicle's surroundings by combining the image information provided by multiple cameras, such that the combined image information is projected onto the second projection surface to provide an environment view.

[0027] The terms “approximately”, “essentially” or “about” mean, within the meaning of the invention, deviations from the respective exact value by + / - 10%, preferably by + / - 5% and / or deviations in the form of changes that are insignificant for the function.

[0028] Further developments, advantages, and possible applications of the invention will also become apparent from the following description of exemplary embodiments and from the figures. All features described and / or illustrated are, individually or in any combination, fundamentally the subject matter of the invention, irrespective of their compilation in the claims or their cross-reference. The content of the claims is also incorporated into the description.

[0029] The invention will be explained in more detail below with reference to exemplary embodiments shown in the figures. The figures show: Fig. 1. For example, a schematic top-view drawing of a vehicle comprising a camera system; Fig. 2. An exemplary and schematic perspective representation of a bowl-shaped projection surface with an ego-vehicle placed in the center of it; Fig. 3. An exemplary and schematic bird's-eye view representation of an environment and an illustration of the search algorithm for determining a second projection surface adapted to the environment; and Fig. 4. An example block diagram illustrates the process steps of a procedure for providing an environment representation of a vehicle.

[0030] Fig. Figure 1 shows an exemplary and schematic representation of a vehicle 1 equipped with a system for the three-dimensional display of the vehicle 1's surroundings on a display unit, hereinafter also referred to as a 3D surround view system. The vehicle 1 is hereinafter also referred to as the "ego vehicle" to conceptually distinguish it from other vehicles, also referred to as "third-party vehicles." The display unit can be located, in particular, inside the vehicle and serves to assist the driver during driving maneuvers, for example, when reversing or parking.Alternatively, the display unit can also be formed by a mobile device, for example a smartphone or tablet computer belonging to a user, which is provided independently of the vehicle 1 and is coupled to the vehicle 1 via a wireless communication connection in order to enable an exchange of information between the vehicle 1 and the mobile device.

[0031] The vehicle 1 has a camera system 2 with several cameras 2.1 distributed around the vehicle 1. For example, the vehicle 1 can have a front camera, a rear camera, and at least one side camera on each side. The cameras 2.1 are preferably arranged and configured such that the images provided by the respective cameras 2.1 combine to create a 360° view of the area surrounding the vehicle 1.

[0032] The vehicle 1 has a control unit 5 which is configured to combine the image information provided by the cameras 2.1 into common image information and to project this onto a bowl-shaped projection surface in order to provide the driver with a perspective view of the environment of the vehicle 1.

[0033] Fig. Figure 2 shows an example of a bowl-shaped projection surface P and the ego-vehicle 1 at the center of this projection surface P. Such a projection surface P is referred to, for example, as a 3D bowl. Using the control unit 5 provided on the vehicle, the image information provided by the cameras 2.1 can be stitched together and the resulting stitched image information projected onto the projection surface P. This allows the environmental information around the vehicle 1 to be displayed spatially in perspective, with the vehicle 1 itself, or rather its contour, being recognizable in this environmental representation. This type of perspective 360° view, unlike a top-down view (bird's-eye view), enables a three-dimensional, perspective view of the area surrounding the vehicle 1 from a specific reference point.The viewpoint is defined as the camera position. This camera position can be varied; that is, the virtual position from which vehicle 1 and its surroundings are viewed can be changed. In other words, changing the position allows for virtual movement within the 3D bowl. This enables a more realistic representation of the area surrounding vehicle 1.

[0034] In addition, for example, the camera orientation can be changed from a fixed camera position, i.e., the angle at which the vehicle 1 or the surroundings of the vehicle 1 are viewed from the camera position.

[0035] The bowl-shaped projection surface P has a flat or essentially flat base and a wall area projecting upwards from this base (3.1). In other words, the base corresponds to the bowl base and the wall area to the bowl wall of the bowl-shaped projection surface P. The bowl-shaped projection surface P can have different basic shapes. In the Fig. In the embodiment shown in Figure 2, the vehicle 1 has a circular base shape, i.e., the wall area of ​​the projection surface P has a circular shape in a plane parallel to the mounting surface on which the vehicle 1 is located. Alternatively, the projection surface P can have a different base shape, for example, an elliptical base shape or a rectangular base shape, which in particular has rounded corner areas.

[0036] If an object is located in the vicinity of the ego vehicle 1, it may be distorted due to its representation on the shell-like projection surface 3. This is particularly true for objects that lie partially within the projection surface P. For example, the pole of a street lamp that penetrates the projection surface P may be displayed unnaturally distorted, especially to the point where it appears crooked or bent (also known as the Manhattan effect). This misrepresentation leads to an unrealistic depiction of the environment, which is perceived as detrimental by users and therefore results in reduced acceptance of the driver assistance system.

[0037] To prevent such a misrepresentation of objects that does not correspond to reality, the vehicle 1 has a control unit 5 that adjusts the shell-like projection surface P.

[0038] Fig. Figure 3 shows an example of an object map OK in a top-down view. In addition to the Ego vehicle 1, the object map includes objects O1, O2, and O3. Objects O1 and O2 are, for example, third-party vehicles parked perpendicular to the direction of travel in parking spaces. In the illustrated embodiment, object O3 is an elongated object, such as a wall or similar, extending perpendicular to the longitudinal axes of objects O1 and O2. It is understood that this is merely an example of an environmental situation for vehicle 1, and object map OK can represent any other environmental situation. For the purposes of this invention, "object map" also includes any other occupancy map, such as an occupancy grid map. This object map can, in particular, be a three-dimensional map to also take into account the height-dependent contours of objects.

[0039] The object map OK is created, for example, based on the detection sensors of vehicle 1. These detection sensors can include the vehicle's cameras 2.1, but other sensors of vehicle 1 can also be used to create the object map, such as ultrasonic sensors, radar sensors, LiDAR sensors, etc.

[0040] Subsequently, a first projection surface 3 is specified, which is in particular a bowl-shaped projection surface. The first projection surface 3 has a predefined basic shape, for example a round, elliptical, or rectangular basic shape, which in particular has rounded corners. As described above, the basic shape refers to the shape of the wall area of ​​the first projection surface 3 in a plane that is parallel to the mounting surface on which the vehicle 1 is located. In the Fig. In the embodiment shown in Figure 3, the basic shape is a rectangle with rounded corners.

[0041] The basic shape can either be rigidly predetermined or chosen depending on the existing environmental situation, in particular the shape and arrangement of the objects in the object map OK, and / or depending on the driving maneuver that vehicle 1 performs.

[0042] Subsequently, based on the object map OK and the first projection surface 3 with its basic shape, a search procedure is carried out to find a second projection surface 4 starting from the first projection surface 3. Preferably, the search procedure is carried out based on a two-dimensional object map OK and a two-dimensional shape of the second projection surface 4, which results in a top view (bird's-eye view). The following search criteria are applied: Initially, the search is restricted such that the second projection surface 4 has the same basic shape as the first projection surface 3; that is, if the first projection surface 3 has a rectangular basic shape (with rounded corners), the second projection surface 4 must also have a rectangular basic shape (with rounded corners). However, the search then identifies a second projection surface 4 whose surface area is maximized. "Maximized surface area" here refers specifically to the area of ​​the wall region of the shell-shaped second projection surface 4 being maximized. This maximization may, however, be subject to limitations or upper limits, which are restricted, for example, by the size of the object map and / or the detection range of the vehicle 1's sensors.

[0043] Furthermore, a second projection surface 4 is determined, in which the ego-vehicle 1 is located entirely within the second projection surface 4. It should be noted, however, that the ego-vehicle 1 does not have to be centered within the second projection surface 4, but can also be positioned off-center within the second projection surface 4.

[0044] Furthermore, the search is restricted in such a way that the second projection surface 4 is free of surrounding objects, i.e. that no object O1, O2, O3 of the object map OK is partially or completely located within the second projection surface 4.

[0045] In Fig. Figure 3 illustrates this search procedure. Two second preliminary projection surfaces 4' and 4" are shown as preliminary search results, which do not meet the aforementioned search criteria. Therefore, these are marked with "X". The second preliminary projection surface 4' does not meet the criterion that the Ego vehicle 1 lies completely within the second preliminary projection surface 4'. The second preliminary projection surface 4" does not meet the criterion of object freedom.

[0046] The second projection surface 4, however, fulfills all search criteria (size maximization, ego vehicle within the second projection surface 4, object-free second projection surface 4) and is therefore used as the final search result, which is indicated by the label "V".

[0047] For example, a RANSAC algorithm (RANSAC: random sample consensus) can be used as a search algorithm. It goes without saying that other suitable search algorithms can also be used to find a second projection surface that meets the specified search criteria.

[0048] After determining a final second projection surface 4, this second projection surface 4 is used to provide the environment representation.

[0049] The determination of the second projection surface 4 based on the first projection surface 3 is carried out in such a way that the basic shape of the first projection surface 3 is retained, but its location, orientation and / or size are adjusted. This means, in particular, that the second projection surface 4 is derived from the first projection surface 3 by shifting, rotating, or enlarging or reducing the edge of the first projection surface 3 as it appears in the top view.

[0050] The change in the size of the first projection surface 3 while maintaining the given basic shape is carried out in such a way that the circular, elliptical or rectangular basic shape is basically retained, but either edge areas are shifted parallel, as is the case, for example, with a rectangular basic shape, or the radius of curvature is changed (in the case of a circular or elliptical basic shape).

[0051] To ensure sufficiently good projection of the image information provided by the camera system onto the second projection surface 4, it may be advantageous for the vehicle 1 not to be positioned directly at the edge of the second projection surface 4, but rather for a predetermined minimum distance to be maintained between the edge of the second projection surface 4 and the vehicle 1. This minimum distance can be used as a further search criterion in the search procedure, so that only those search results that meet this minimum distance are classified as valid. The minimum distance can, for example, be in the range of 0.5 m to 1.5 m, and in particular 1 m.

[0052] Furthermore, an upper limit can be specified as an additional search criterion for maximizing the surface area of ​​the second projection surface 4. This search criterion can, for example, specify that the distance between diametrically opposed boundary sections does not exceed a predefined upper threshold. This upper threshold can, for example, be in the range between 15 m and 25 m, and in particular, 20 m.

[0053] Preferably, the search procedure also takes into account which driving maneuver the vehicle 1 is to perform in the future. For example, in a forward parking maneuver, a second projection surface 4 can preferably be sought which, viewed in the direction of travel of the vehicle 1, has a larger clearance in front than in the rear.

[0054] Once the search process has identified a final second projection surface 4, this surface is used to provide the environmental representation, even if the position of vehicle 1 in space changes. The second projection surface 4 remains fixed in position, so that the position of vehicle 1 within the second projection surface 4 changes.

[0055] If the determined second projection surface 4 no longer meets the specified criteria, for example due to the movement of the vehicle 1 relative to the second projection surface 4 or a change in the position of one or more objects O1, O2, O3 in the vicinity of the vehicle 1 (for example, vehicle 1 leaves the second projection surface 4 or an object O1, O2, O3 lies within the projection surface due to a movement of the object), the search procedure can be carried out again to determine a new second projection surface 4 that meets the specified search criteria.

[0056] Fig. Figure 4 illustrates, using a schematic block diagram, the steps of a procedure for providing an environment representation of a vehicle.

[0057] First, image information is captured using several cameras of the vehicle's camera system (S10).

[0058] Subsequently, objects in the vicinity of the vehicle and their spatial location are determined (S11).

[0059] Subsequently, an object map is created, containing the identified objects according to their location (S12).

[0060] In addition, a circumferentially closed first projection surface with a basic shape is provided (S13).

[0061] Based on the object map and the first projection surface, a second projection surface is determined (S14) that meets the following criteria: - the surface area of ​​the second projection surface is maximized while maintaining the basic shape of the first projection surface; - the vehicle is located entirely within the space enclosed by the second projection surface; and - No objects contained in the object map lie within the second projection surface or do not partially extend into this second projection surface;

[0062] Finally, a 360° view of the vehicle's surroundings is created by combining image information provided by multiple cameras. This is done by projecting the combined image information onto the second projection surface to provide an environmental view (S15).

[0063] The invention has been described above using exemplary embodiments. It is understood that numerous modifications and adaptations are possible without thereby departing from the scope of protection defined by the patent claims. Reference symbol list 1 vehicle 2 camera system 2.1 Camera 3 first projection surface 4 second projection surface 4', 4" second provisional projection surface 5 Control unit P Projection surface O1, O2, O3 Object OK object map

Claims

[1] Method for providing an environment representation of a vehicle (1) based on image information provided by a camera system (2) of the vehicle (1) which has multiple cameras (2.1) distributed around the vehicle (1), the method comprising the following steps: - Acquisition of image information using multiple cameras (2.1) of the camera system (2) (S10); - Determining objects (O1, O2, O3) in the vicinity of the vehicle (1) and their spatial location (S11); - Creating an object map (OK) containing the identified objects (O1, O2, O3) according to their location (S12); - Providing a circumferentially closed first projection surface (3) with a basic shape (S13); - Based on the object map (OK) and the first projection surface (3), determine a second projection surface (4) (S14) such that: - the surface area of ​​the second projection surface (4) is maximized while maintaining the basic shape; - the vehicle (1) is completely within the space enclosed by the second projection surface (4); and - no objects (O1, O2, O3) contained in the object map (OK) lie within the second projection surface (4) or partially extend into it; - Creating a 360° view of the vehicle's surroundings (1) by combining the image information provided by several cameras (2.1) such that the combined image information is projected onto the second projection surface (4) to provide an environment view (S15). [2] Method according to claim 1, characterized by , that the basic shape of the first projection surface (3) is selected from several basic shape variants. [3] Method according to claim 1 or 2, characterized by, that determining the second projection surface (4) involves adjusting the local location, orientation and / or size of the first projection surface (3). [4] Method according to claim 3, characterized by , that adjusting the local position of the first projection surface (3) includes moving the first projection surface (3) in a plane parallel to the surface on which the vehicle (1) is located. [5] Method according to claim 3 or 4, characterized by , that adjusting the orientation of the first projection surface (3) involves rotating the first projection surface (3) about a vertical axis of rotation. [6] Method according to any one of claims 3 to 5, characterized by , that adjusting the size of the first projection surface (3) involves at least a partial, parallel displacement of boundary sections of the first projection surface (3) and / or a change in the radius of curvature of a curved boundary section of the first projection surface (3). [7] Method according to any one of claims 3 to 6, characterized by , that the adjustment of the local position, orientation and / or size of the first projection surface (3) is carried out in such a way that the circumferential contour of the vehicle (1) maintains a specified minimum distance from the first projection surface (3). [8] Method according to any one of the preceding claims, characterized by , that the maximization of the surface area of ​​the second projection surface (4) is carried out in such a way that a maximum distance of diametrically opposite edge sections of the second projection surface (4) is not exceeded. [9] Method according to any one of the preceding claims, characterized by , that after determining the second projection surface (4) when moving the vehicle (1) the position, orientation and / or size of the second projection surface (4) is maintained at least temporarily and the position of the vehicle (1) within the second projection surface (4) is changed. [10] Method according to claim 9, characterized by , that the 360° view is created by projecting images taken by the cameras (2.1) of the vehicle (1) while it is moving onto the second projection surface (4). [11] Method according to claim 9 or 10, characterized by , that the second projection surface (4) is adjusted in position, orientation and / or size if the vehicle (1) falls below a minimum distance to the edge of the second projection surface (4) due to its movement relative to the second projection surface (4). [12] Method according to any one of the preceding claims, characterized by , that the first and second projection surfaces (3, 4) have a three-dimensional, bowl-like basic shape. [13] Camera system for a vehicle (1) comprising several cameras (2.1) distributed around the vehicle (1) and a control unit (5) for processing the image information provided by the cameras (2.1) in order to generate an environmental representation of the environment of the vehicle (1), wherein the camera system (2) is configured to perform the following steps: - Acquisition of image information using multiple cameras (2.1) of the camera system (2); - Determining objects (O1, O2, O3) in the vicinity of the vehicle (1) and their spatial location; - Creating an object map (OK) containing the identified objects (O1, O2, O3) according to their location; - Providing a circumferentially closed first projection surface (3) with a basic shape; - Based on the object map (OK) and the first projection surface (3), determine a second projection surface (4) such that: - the surface area of ​​the second projection surface (4) is maximized while maintaining the basic shape; - the vehicle (1) is completely within the space enclosed by the second projection surface (4); and - no objects (O1, O2, O3) contained in the object map (OK) lie within the second projection surface (4) or partially extend into it; - Creating a 360° view of the vehicle's surroundings (1) by combining the image information provided by several cameras (2.1) such that the combined image information is projected onto the second projection surface (4) to provide an environment view.

Images