Method for generating a view using a camera system, and camera system

Abstract

The present invention relates to a method for generating a view for a camera system, in particular a surround-view camera system for a vehicle (1), comprising a control device (2) and at least one camera (3a-3d), the view being generated by means of the following method steps: - detecting at least one object from the environment data from the at least one camera (3a-3d), - generating a bounding box for the object, - projecting the object onto a ground plane, - creating a bounding shape, which comprises the bounding box and the projected object, - creating a mesh structure or grid structure for the bounding shape, and - arranging the mesh structure or grid structure within the bounding box, the bounding shape being adapted, in particular by image scaling and / or image distortion, to the size of the bounding box.

Description

[0001] Method for generating a view with a camera system and camera system

[0002] The present invention relates to a method for generating a view with a camera system for a vehicle and a camera system, in particular a surround view camera system for detecting the surroundings of a vehicle, which can generate a view using the method according to the invention.

[0003] Technological background

[0004] Modern vehicles are increasingly being equipped with driver assistance systems that support the driver in performing driving maneuvers. In addition to radar sensors, lidar sensors, ultrasonic sensors, and / or camera sensors, these driver assistance systems also include surround-view camera systems, which allow the vehicle's surroundings to be displayed to the driver. Such surround-view camera systems generally comprise a control device and several cameras that provide real images of the vehicle's surroundings, which are combined, in particular, by a data processing unit of the surround-view camera system to form an image of the vehicle's surroundings. The image of the vehicle's surroundings is then shown to the driver on a display unit (such as the navigation system display). In this way, the driver can be assisted during a vehicle maneuver, for example when reversing the vehicle or during a parking maneuver.Furthermore, surround-view cameras are typically "fisheye cameras," meaning they produce a fisheye image. These undistorted fisheye images are then used to present the driver with various views of the surroundings, such as the front view, rear view, curb view, and the like. Modern surround-view camera systems can then display the resulting views to the driver, e.g., on a display, cockpit, or navigation system.

[0005] Furthermore, the images can be combined to create a 360° panoramic view, allowing the driver to select the appropriate viewpoint by moving within a scene captured by a virtual camera. Various functions and views are available, such as "Bowl" or "Top View" ("bird's eye view" or "overhead view"), in which images or textures from the surround-view cameras are combined or seamlessly stitched together to create an overall view (or overall texture). The images or textures from the surround-view cameras generally have overlapping regions, particularly in the bowl view, where the textures from the cameras are projected to visualize a virtual 3D bowl representing the entire area around the car. Furthermore, texture information from the camera system can be projected onto a mesh (projection surface) or a static 2D plane, for example, toto create a top-down view. However, views created in this way can result in the captured objects being visually distorted or distorted. This occurs due to the reprojection of the object texture onto the ground plane. This effect is visually distracting for the user, so it is of particular interest to avoid such distortions.

[0006] Printed state of the art

[0007] DE 10 2014 208 664 A1 discloses a camera surround view system for a vehicle having at least one vehicle camera that provides camera images that are processed by a data processing unit to generate an environmental image that is displayed on a display unit, wherein the data processing unit replicates textures captured by the vehicle cameras on an adaptive re-projection surface similar to the vehicle environment, which is calculated on the basis of sensor data provided by vehicle sensors, thereby minimizing or eliminating distortions or distorted artifacts.

[0008] Furthermore, EP 2 973420 B1 discloses a method in which, for displaying a graphic in a three-dimensional virtual environment of a vehicle, a standard three-dimensional projection surface is generated from camera data or images from multiple cameras, which is centered around a virtual representation of the vehicle in a virtual environment. This projection surface is generated based on first polygon model data corresponding to the shape of an object in the environment. The three-dimensional projection surface is then deformed with respect to the first polygon model data for an object at a location in the virtual environment, which corresponds to a relative distance and a direction of the object, which was detected based on environmental sensor data.The images can then be projected onto the deformed three-dimensional projection surface and displayed by means of a display device in an in-vehicle information system, wherein the displayed graphic corresponds to the deformed three-dimensional projection surface with the plurality of projected images. The object of the present invention is to achieve this.

[0009] The present invention is therefore based on the object of providing a generic (surround view) camera system by which the representation of distorted objects is prevented in order to represent objects or obstacles in the vehicle environment as clearly visible and distortion-free as possible.

[0010] Solution to the task

[0011] The above object is achieved by the entire teaching of claim 1 and the subordinate claim. Advantageous embodiments of the invention are claimed in the subclaims.

[0012] In the method according to the invention for generating a view for a camera system, in particular a surround view camera system for a vehicle, the camera system comprises a control device and at least one camera - preferably several cameras, wherein the view is generated using the following method steps:

[0013] - detecting at least one object from the environmental data of the at least one camera,

[0014] - Creating a bounding box of the object,

[0015] - Projecting the object onto a ground plane,

[0016] - Create a boundary shape that includes the bounding box and the projected object,

[0017] - Creating a network structure or grid structure for the boundary shape, and

[0018] - Arranging the network structure or grid structure within the bounding box, wherein the boundary shape can be adapted to the size of the bounding box, in particular by image scaling and / or image distortion. For the purposes of the invention, image scaling is understood to mean a change in the size of an image or of the boundary shape or of the network structure of the boundary shape. For example, the image resolution can be changed during scaling, so that a new image with a higher or lower number of pixels is generated. For example, “texture mapping” or “pattern imaging” can also be carried out during image scaling, whereby surfaces of models, in particular three-dimensional surface models, are designed with two-dimensional images (textures) and possibly also surface properties. The textures make the images appear more detailed and realistic.With regard to the invention, the bounding box can be adapted to the size of the bounding box as a kind of model with the image of the projected object. Image distortion or "image warping" (image deformation or image distortion) is understood, within the meaning of the invention, to be an image-based technique in which, for example, the depth values ​​associated with an image are transformed using a so-called transformation or warping equation ("morphing" or "image morphing") in such a way that the image can be deformed / distorted in the desired manner and / or viewed from a different viewpoint (in real time).

[0019] The present invention aims to improve the diagonally distorted appearance of objects in the vehicle's surroundings. For example, in a bird's-eye view, these objects are projected onto the ground. The projected shape of these objects is positioned in the image such that their projection appears straight and not diagonally distorted. This allows the objects in the vehicle's surroundings to be displayed clearly and without distortion. This, in many cases, significantly improves the visual appearance of these objects and the spatial representation.

[0020] Conveniently, the view may comprise a 2D view, in particular a top view, a 3D view, in particular a bowl, or the like.

[0021] Preferably, the bounding box is two-dimensional and axis-oriented. In particular, it can be a two-dimensional geometric shape (e.g., a circle, polygon, such as a rectangle, quadrilateral, triangle, hexagon, or the like). This shape can be conveniently selected depending on the outline or contour of the respective object or the associated detection points.

[0022] According to a preferred embodiment of the invention, the mesh structure or grid structure comprises a triangular mesh or triangular grid. However, other forms are also conceivable, such as differently designed polygon meshes or polygon grids.

[0023] Preferably, for creating the boundary shape in step IV, a shape is selected which is created by connecting corners of the boundary box with another geometric shape, in particular a shape comprising a polygonal line (e.g. triangular, quadrilateral, or rectangular shape), which is arranged at the opposite end of the projected points, i.e., for example, a rectangle or quadrilateral (alternatively, another polygon or circular shape - particularly depending on the contour of the object) is arranged, which spans the outer points. This can then, for example, also have a similar contour to the boundary box or part of the boundary box. The distortion-free or less distorted representation is thereby significantly improved.According to a preferred embodiment, the boundary shape can include all projected points of the object, whereby any outliers that can be detected, for example, via limit values, cannot be taken into account.

[0024] For convenience, the mesh structure can be arranged within the bounding box such that the corners and edges of the final boundary are aligned along the border of the original bounding box. This gives the object the same perimeter and boundary as the bounding box created in Step II, resulting in a particularly realistic and clear representation. Furthermore, distortions are significantly reduced.

[0025] Preferably, extrinsic and / or intrinsic camera parameters as well as the object data (preferably three-dimensional data determined based on the environmental data of the cameras or other sensors) are used to create the bounding box and / or the boundary shape and / or the network structure or grid structure. For the purposes of the invention, intrinsic parameters are understood to be camera parameters that are internally and permanently linked to a specific camera or digitization device. In contrast, extrinsic parameters are camera parameters that are external to the camera and can change in relation to the world view (location / position / orientation of the camera in the world coordinate system). In relation to a camera model, this means that extrinsic parameters define the location and orientation of the camera in relation to the world view. In contrast, intrinsic parameters enable an assignment between camera coordinates and pixel coordinates in the image orField of view (relationship between camera and image coordinate system), e.g., the focal length f and the optical center in the image plane. The camera model is, in a sense, a mapping from world coordinates to image coordinates, which is achieved using a 3D-to-2D transformation. The intrinsic parameters do not depend on the position and orientation of the camera in the world and describe the image and the internal geometry of the camera.

[0026] By placing the mesh or grid structure within the bounding box, empty areas can be conveniently filled by propagating pixels from the surrounding area into this area and / or by using a historical soil structure for filling and / or by using texture information from various cameras. This significantly improves the view.

[0027] Furthermore, the present invention comprises a camera system, in particular a surround view camera system for a vehicle, which comprises a control device and one or more cameras arranged in / on the vehicle, wherein the control device generates the view based on the method according to the invention and the cameras or the camera data or camera images.

[0028] Description of the invention based on exemplary embodiments

[0029] The invention is described in more detail below using practical examples. They show:

[0030] Fig. 1 is a simplified schematic representation of an embodiment of a

[0031] Vehicle with an inventive (surround view) camera system;

[0032] Fig. 2 is a simplified schematic representation of an inventive

[0033] Procedural process, as well as

[0034] Fig. 3 shows a simplified representation of the method according to the invention using various steps (AF) schematically shown, in which a vehicle detects an object using detection points and a view of this object is generated using the method according to the invention.

[0035] Reference numeral 1 in Fig. 1 denotes a vehicle with a control device 2 (ECU, Electronic Control Unit or ADCU, Assisted and Automated Driving Control Unit), which can access various actuators (e.g., steering, engine, brakes) of the vehicle 1 in order to be able to carry out control operations of the vehicle 1. Furthermore, the vehicle 1 has several surround-view cameras or cameras 3a-3d, a camera sensor 4 (or front camera), and a lidar sensor 5 for detecting the surroundings, which are controlled via the control device 2. However, the present invention also expressly encompasses embodiments in which no common control device 2 is provided, but rather individual control devices or control units are provided for sensor control (e.g., a separate control unit or a separate control device for controlling the cameras 3a-3d, for the corresponding data processing, and for implementing the method according to the invention).In addition, additional sensors, such as radar or ultrasonic sensors, can be provided. The sensor data can then be used for environment and object detection. As a result, various assistance functions, such as parking assistants, emergency braking assistants (EBA, Electronic Brake Assist), adaptive cruise control (ACC, Adaptive Cruise Control), lane keeping control or lane keeping assistants (LKA, Lane Keep Assist), or the like, can be implemented. In practical terms, the assistance functions can also be implemented via control device 2 or a separate control device.

[0036] The cameras 3a-3d are part of a surround-view camera system, which is preferably controlled by the control device 2 (alternatively, a separate control system can be provided, for example), which offers a complete 360-degree view around the entire vehicle 1 by combining the fields of view of the individual surround-view cameras, e.g. 120 degrees, into an overall view or overall image. Due to the simple monitoring of the blind spot, this camera system has numerous advantages in many everyday situations. The surround-view camera system can show the driver different viewing angles of the vehicle 1, e.g. via a display unit (not shown in Fig. 1). As a rule, four surround-view cameras 3a-3d are used, which are arranged, for example, in the front and rear areas and on the side mirrors. In addition, three, six, eight, ten or more surround-view cameras can also be provided. These camera views are particularly helpful.Viewing angles when checking blind spots, changing lanes or parking.

[0037] The method according to the invention is shown schematically in Fig. 2 and comprises the method steps described below.

[0038] Step I: Detecting an object (or multiple objects) from three-dimensional environmental data (Fig. 3A). This step essentially depends on the camera or sensor and environmental data. For example, this data can be in the form of point clouds (as shown in Fig. 3A by the black dots or detection points), with so-called point clusters being detected if they, for example, lie above a certain threshold above the ground plane.

[0039] Step II: Creating a two-dimensional and axis-oriented bounding box of the object (Fig. 3B) from the vehicle's top view. For example, the XY axis, using the minimum and maximum values ​​on each axis for a given set of object points.

[0040] Step III: Projecting the object onto the ground plane (Fig. 3C), with, for example, a camera positioned to the left of the points and the points not on the ground, so that the points spread beyond the boundaries of the bounding box when projected onto the ground (illustrated by the triangular points in Fig. 3C). Step IV: Creating or calculating a bounding shape that encompasses both the bounding box and the projected object (Fig. 3D). Conveniently, a simple shape can be chosen that encompasses both the bounding box itself and the projected objects. This simple shape can be, for example, the junction of the bounding box, another rectangle at the other end of the projected points, and the junction of their vertices. However, the resulting shape should preferably encompass all projected points within the generated area.

[0041] Step V: Creating a mesh or lattice structure (triangular mesh or triangular grid) for the boundary shape (Fig. 3E), preferably creating a triangular mesh structure to represent the boundary shape by connecting the angles of the shape with each polygon-triangle approach.

[0042] Step VI: The arrangement of the mesh or grid structure within the bounding box (Fig. 3F), whereby the mesh structure or triangular mesh from step V is adjusted such that the corners and edges are arranged along the boundary of the original bounding box. The resulting shape essentially has the shape of the bounding box (from step II). The boundary shape is adapted to the size of the bounding box by image scaling and / or image distortion. This “texture mapping step,” for example, which transforms or reshapes the mesh or grid structure from step V into a mesh or grid structure (as shown in Fig. 3F) that essentially corresponds to the boundary of the bounding box, thus serves to adapt the object representation accordingly. The object then appears much more natural on the subsequent display and gives the user a better sense of orientation. This allows, for example,Parking operations are made particularly easy.

[0043] When transitioning from step V to step VI, or when removing the mesh or grid structure created for the boundary shape (step V), the area of ​​the original grid structure that is not located within the bounding box (see step VI) remains free or unfilled. There is, so to speak, no visual information from current camera and time data to represent this area. However, various methods can advantageously be used to fill this ground area or image area, such as propagating pixels from the environment into this area, using a historical ground structure to fill these areas, or using texture information from different cameras. In summary, the invention significantly improves the visualization quality because objects do not appear distorted and with a static reprojection surface.Another advantage can be achieved by extending the top view with parking space markers. Without stretching, obstacles or other vehicles can be drawn into the empty parking space, making the parking space markers appear to lie on the obstacle. After removing the stretch, the spot where the parking space is displayed actually appears to be empty.

[0044] LIST OF REFERENCE SYMBOLS 1 Vehicle

[0045] 2 Control device

[0046] 3a Camera

[0047] 3b Camera

[0048] 3c camera 3d camera

[0049] 4 camera sensor

[0050] 5 Lidar sensor

Claims

PATENT CLAIMS 1. Method for generating a view for a camera system, in particular a surround view camera system for a vehicle (1), comprising a control device (2) and at least one camera (3a-3d), wherein the view is generated by means of the following method steps: - detecting at least one object from the environmental data of the at least one camera (3a-3d); - Creating a bounding box of the object; - Projecting the object onto a ground plane; - Create a bounding shape that includes the bounding box and the projected object; - Creating a mesh structure for the boundary shape; and - Arranging the mesh structure within the bounding box, wherein the boundary shape is adapted to the size of the bounding box, in particular by image scaling and / or image distortion.

2. Method according to claim 1, characterized in that the view comprises a 2D view, in particular a top view, a 3D view, in particular a bowl, or the like.

3. Method according to claim 1 or 2, characterized in that the boundary box is designed two-dimensionally and axis-oriented 4. Method according to one of the preceding claims, characterized in that the network structure comprises a polygon network or polygon grid, in particular a triangular network or a triangular grid.

5. Method according to one of the preceding claims, characterized in that for creating the boundary shape, a shape is selected which is created by connecting corners of the boundary box with another geometric shape, in particular a shape comprising a polygonal line, which is arranged at the opposite end of the projected points. Method according to claim 5, characterized in that the Boundary shape includes all projected points of the object. Method according to one of the preceding claims, characterized in that the arrangement of the mesh structure within the bounding box is carried out in such a way that the corners and edges are arranged along the border of the original bounding box. Method according to one of the preceding claims, characterized in that extrinsic and / or intrinsic camera parameters as well as the object data are used to create the bounding box and / or the boundary shape and / or the mesh structure. Method according to one of the preceding claims, characterized in that free areas resulting from the arrangement of the mesh or grid structure within the bounding box are filled by - Pixels from the surrounding area are propagated into this area and / or - a historical soil structure is used for backfilling and / or - Texture information from various cameras is used. A camera system, in particular a surround-view camera system for a vehicle (1), comprising a control device (2), a plurality of cameras (3a-3d) arranged in / on the vehicle (1), and the control device (2) generates a view based on the cameras (3a-3d), characterized in that the view is generated using a method according to one of the preceding claims.

Images