Method and camera system for generating a view using a camera system

Abstract

The present invention relates to a method for generating a view of a camera system for a vehicle (1), in particular a surround view camera system, comprising a control device (2) and at least one camera (3a to 3d), the following method steps: - detecting at least one object based on the ambient data of at least one camera (3a to 3d); - creating a bounding box of the object; - projecting the object onto the ground plane; - creating a boundary shape including the bounding box and the projected object; - creating a mesh structure or a grid structure of the boundary shape; and - placing the mesh structure or the grid structure within the bounding box, the method for generating a view using a step of adjusting the boundary shape to the size of the bounding box, in particular by image scaling and / or image resizing.

Description

Technical Field

[0001] The present invention relates to a method for generating a view using a vehicle camera system and a vehicle camera system for detecting a surrounding environment capable of generating a view using the method according to the present invention, in particular, a surround view camera system.

Background Art

[0002] Recent vehicles are increasingly equipped with a driving assistance system that assists a driver when performing a driving operation. These driving assistance systems include, in addition to a radar sensor, a lidar sensor, an ultrasonic sensor and / or a camera sensor, in particular, a surround view camera system that enables the vehicle's driver to view the surrounding environment of the vehicle. Such a surround view camera system generally includes a control device and a plurality of cameras, and these plurality of cameras provide real images of the surrounding environment of the vehicle, and these real images of the surrounding environment of the vehicle are combined with the surrounding environment images of the surrounding environment of the vehicle by the data processing device of the surround view camera system. Then, the image of the surrounding environment of the vehicle is displayed on a display device (for example, the display of a navigation system) for the driver. In this way, the driver can be assisted during a driving operation, for example, when the vehicle is reversing or when performing a parking operation. Also, a surround view camera is generally a "fisheye camera", that is, a camera equipped with a fisheye lens that provides a fisheye image. Here, a fisheye image without distortion is used to represent various views of the surrounding environment for the driver, such as a front view, a rear view, a curb view, etc. And recent surround view camera systems can display the view thus generated for the driver, for example, on a display, a cockpit or a navigation system.

[0003] In addition, since the image can also be combined with a 360° panoramic view, the driver can select an appropriate viewpoint by moving within the scene of the virtual camera. In this case, there are various functions or views, such as "bowl" or "top view" ("bird's-eye view" or "top view"), which combine the images or textures of the surround view cameras into an overall view (or overall texture), that is, seamlessly continuous with each other (stitching). In this case, the images or textures of the surround view cameras generally include overlapping areas, that is, overlapping regions. In particular, in the bowl view, the texture of the camera is projected, and a virtual three-dimensional bowl representing the entire area around the vehicle is visualized. Also, the texture information of the camera system can be projected onto a mesh (projection area), that is, a static two-dimensional surface, for example, to create a top view. However, in the case of the view created in this way, the detected object may be visually distorted or disrupted. This is caused by the re-projection of the object texture onto the ground. Since this effect is visually disturbing to the user, it is particularly interesting to avoid such distortion.

[0004] Prior art documents Patent Document 1 discloses a vehicle camera surround view system having at least one in-vehicle camera that provides a camera image for creating a peripheral environment image displayed on a display unit by a data processing unit, and the data processing unit re-projects the texture captured by the in-vehicle camera onto an adaptable re-projection area similar to the vehicle peripheral environment calculated based on sensor data provided by an in-vehicle sensor, thereby minimizing or removing distortion or distorted artifacts.

[0005] Also, from Patent Document 2, in order to reproduce graphics in a three-dimensional virtual surrounding environment of a vehicle, a method of generating a standard three-dimensional projection area arranged around a virtual representation of the vehicle in the virtual surrounding environment based on camera data or images of a plurality of cameras is known. In this case, generation is performed based on first polygon model data corresponding to the shape of an object in the surrounding environment. Then, the three-dimensional projection area is deformed with respect to the first polygon model data of an object located at a position in the virtual environment corresponding to the relative distance and direction of the object detected based on the surrounding environment sensor data. Thereafter, an image can be projected onto the deformed three-dimensional projection area and can be displayed using a display device of an in-vehicle information system, and the displayed graphics correspond to the deformed three-dimensional projection area having a plurality of projection images.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] Accordingly, an object of the present invention is to provide a (surround view) camera system of a higher concept that prevents the representation of a distorted object in order to represent an object or an obstacle in a vehicle surrounding environment as clearly and undistorted as possible.

Means for Solving the Problems

[0008] This problem is solved by the teachings of all of Claim 1 and the parallel independent claims. Preferred forms of the present invention are claimed in the dependent claims.

[0009] In a method for generating views of a vehicle camera system, in particular a surround view camera system, comprising a control device and at least one camera, preferably a plurality of cameras, the following method steps: - Detecting at least one object based on the ambient environment data of at least one camera; - Creating a bounding box of the object; - Projecting the object onto the ground plane; - Creating a boundary shape including the bounding box and the projected object; - Creating a mesh structure or a grid structure of the boundary shape; and - A step of arranging a mesh structure or a grid structure within a bounding box, wherein a view is generated using a step of adjusting a boundary shape, in particular, by image scaling and / or image resizing, to the size of the bounding box. In the present invention, image scaling is understood as changing the size of an image or a boundary shape or a mesh structure of a boundary shape. For example, since the image resolution can be changed during scaling, a new image having a larger or smaller number of image points (pixels) can be created. For example, "texture mapping" or "pattern mapping" can also be performed during image scaling, and the surface of a model, in particular, a three-dimensional surface model, is configured using a two-dimensional image (texture), and in the corresponding case, surface characteristics are also configured. In this case, due to the texture, the image looks more detailed and realistic. Also, regarding the present invention, a bounding box as a kind of model having an image of a projected object can be adjusted to the size of the bounding box. Image resizing or "image warping" (image deformation or image resizing) is understood in the present invention as an image-based technique, for example, so that a depth value related to an image can be used with a so-called deformation equation or warping equation to deform / resize the image as desired and / or so that it can be observed from another viewpoint (in real time) ( "morphing" or "image morphing").

[0010] The present invention aims to improve the distorted appearance of objects in the vehicle surrounding environment. For example, in an aerial view, that is, a top view, these objects are projected onto the ground, and the projected shapes of these objects are arranged in the image so that their projections look straight and are not distorted obliquely. Therefore, the objects in the vehicle surrounding environment are clearly and undistorted in appearance, and thereby, in many cases, the visual appearance and spatial representation of these objects are significantly improved.

[0011] Preferably, the view can include a two-dimensional view, in particular, a top view, a three-dimensional view, in particular, a bowl, etc.

[0012] Preferably, the bounding box is configured in two dimensions and axially. In particular, in this case, it may be a two-dimensional geometry (e.g., a circle, a polygon, e.g., a rectangle, a square, a triangle, a hexagon, etc.). This shape can actually be selected according to the outer shape or contour of each object, or the detection points associated therewith.

[0013] According to a preferred embodiment of the present invention, the mesh structure or grid structure includes a triangular mesh or triangular grid. However, other shapes, such as polygon meshes or polygon grids configured differently, are also conceivable.

[0014] Preferably, in order to create the boundary shape in step IV, the shape created by connecting the corners of the bounding box and other geometries arranged at the opposite ends of the projected points, in particular, shapes including a polygon chain (e.g., triangular, square or rectangular), is selected, that is, for example, a rectangle or square that spreads to the outer points (or, in particular, other polygons or circular shapes according to the contour of the object) is arranged. In this case, the boundary shape can also have a contour similar to that of the bounding box or a part of the bounding box, for example. Thereby, the representation without distortion or with less distortion is particularly improved.

[0015] According to a preferred embodiment, the boundary shape can include all the projected points of the object, and for example, detectable outliers exceeding a threshold value can be excluded from consideration.

[0016] Suitably, arranging the mesh structure within the bounding box can be done such that the corners and edges of the final boundary are arranged along the boundary of the original bounding box. In practice, thereby, the object is particularly realistically and clearly represented because it obtains the same range or the same boundary as the bounding box created in step II. Also, distortion is particularly avoided.

[0017] Preferably, external and / or internal camera parameters and object data (preferably, three-dimensional data calculated based on the surrounding environment data of a camera or other sensor technology) are used to create a bounding box and / or a boundary shape and / or a mesh structure or a grid structure. In the present invention, the internal parameters are understood as camera parameters that are fixedly coupled inside a certain camera device or digitizing device. On the other hand, the external parameters are camera parameters that are outside the camera and can change with respect to the world view (the location / position / orientation of the camera in the world coordinate system). Regarding the camera model, this means that the external parameters define the position and orientation of the camera with respect to the world view. On the other hand, the internal parameters enable the assignment (correlation between the camera coordinate system and the image coordinate system) between the camera coordinate system and the image coordinate system within an image or a field of view. For example, the focal length f and the optical center of the image plane. The camera model is, so to speak, a mapping from world coordinates to image coordinates, which is performed using a three-dimensional to two-dimensional conversion. Here, the internal parameters do not depend on the position and orientation of the camera in the world and describe the mapping and the internal geometry of the camera.

[0018] Preferably, the empty space generated by placing a mesh structure or a grid structure within the bounding box can be filled by propagating the pixels of the surrounding environment into this space, and / or using the past ground structure for filling, and / or using the texture information from various cameras, thereby particularly improving the view.

[0019] Further, the present invention relates to a camera system for a vehicle, particularly a surround view camera system, comprising a control device and one or more cameras arranged inside / on the vehicle, wherein the control device generates a view using the cameras, and the control device includes a camera system that generates a view using the method according to the present invention and the camera or camera data or camera image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Hereinafter, the present invention will be described in more detail based on preferred embodiments.

Figure 1

Figure 2

Figure 3

Mode for Carrying Out the Invention

[0021] Reference numeral 1 in FIG. 1 indicates a vehicle equipped with a control device 2 (ECU (Electronic Control Unit) or ADCU (Assisted and Automated Driving Control Unit)) that can access various actuators (e.g., steering, motor, brake) of the vehicle 1 so as to execute the control process of the vehicle 1. Further, the vehicle 1 includes a plurality of surround view cameras or cameras 3a to 3d, a camera sensor 4 (or front camera), and a lidar sensor 5 that are controlled via the control device 2 to detect the surrounding environment. However, the present invention explicitly includes embodiments in which instead of a common control device 2 being provided, individual control devices or control units for sensor control (e.g., separate control units or separate control devices for controlling the cameras 3a to 3d, for corresponding data processing, and for executing the method according to the present invention) are provided. Also, additional sensors, such as radar sensors or ultrasonic sensors, can be provided. In this case, the sensor data can be used for surrounding environment detection and object detection. Accordingly, various assistance functions, such as parking assistance, emergency brake assistance (EBA (Electronic Brake Assist)), distance following control (ACC (Adaptive Cruise Control)), lane keeping control, or lane keeping assistance (LKA (Lane Keep Assist)), etc., can be implemented. In practice, the assistance functions can also be executed via the control device 2 or a separate control device.

[0022] Here, cameras 3a to 3d are preferably part of a surround view camera system controlled by the control device 2 (alternatively, for example, a separate control device can be provided), and this surround view camera system combines the fields of view of individual surround view cameras, for example, 120 degrees, into an overall view or an overall image to provide a complete 360-degree view around the entire vehicle 1. Since it is easy to monitor blind spots, this camera system has many advantages in many daily situations. With the surround view camera system, various viewpoints of the vehicle 1 can be shown to the driver, for example, via a display device (not shown in FIG. 1). Usually, in that case, for example, 4 surround view cameras 3a to 3d arranged in the front area, rear area, and side mirrors are used. Also, 3, 6, 8, 10 or more surround view cameras can be provided. The views or viewpoints of these cameras are particularly useful when checking for blind spots, changing lanes, or parking.

[0023] The method according to the present invention is schematically shown in FIG. 2 and comprises the following method steps.

[0024] Step I: Detecting one object (or a plurality of objects) based on three-dimensional surrounding environment data (FIG. 3A). This step basically depends on camera data or sensor data and surrounding environment data. For example, these data may be available in the form of a point cloud shape (shown using black dots, i.e., detected points in FIG. 3A), and for example, when located above a certain threshold ("threshold") from the ground surface, so-called point clusters are detected.

[0025] Step II: Creating a bounding box ("bounding box") of the object, particularly two-dimensional and axial, based on the viewpoint from above the vehicle (top view) (FIG. 3B). For example, in the case of the X - Y axes, for a certain set of object points, the minimum and maximum values for each axis are used.

[0026] Step III: A step of projecting an object onto the ground surface (FIG. 3C). For example, since the camera is arranged on the left side of the point and the point is not located on the ground, when the projection onto the ground is performed, the point spreads beyond the boundary of the bounding box (shown using triangular points in FIG. 3C).

[0027] Step IV: A step of creating or calculating a boundary shape that includes both the bounding box and the projected object (FIG. 3D). In this case, actually, a simple shape that includes both the bounding box itself and the projected object can be selected. This simple shape may be, for example, the connection of the bounding box, other rectangles at the other ends of the projected points, and the connection of their corners. However, it is preferable that the resulting shape includes all the projected points within the created region.

[0028] Step V: A step of creating a mesh structure or a grid structure (triangular mesh or triangular grid) of the boundary shape (FIG. 3E). Preferably, in order to represent the boundary shape, a triangular mesh structure is created by connecting the corners of the shape using the polygon triangulation method for each.

[0029] Step VI: Placing a mesh structure or a grid structure within the bounding box (Figure 3F), where the mesh structure from Step V, i.e., the triangular mesh, is adjusted such that the corners and edges are placed along the boundaries of the original bounding box. The resulting shape substantially has the shape of the bounding box (from Step II). At this time, the boundary shape is adjusted to the size of the bounding box by image scaling and image resizing. So to speak, the mesh structure or grid structure from Step V is transformed or deformed into a mesh structure or grid structure that substantially matches the boundaries of the bounding box (as shown in Figure 3F). For example, this "texture mapping step" thus serves to adjust the object representation accordingly. Thereby, the object looks very natural in subsequent displays and conveys a better sense of direction to the user. Thereby, for example, the parking process is particularly simplified.

[0030] When transitioning from Step V to Step VI, i.e., when deleting the mesh structure or grid structure created for the boundary shape (Step V), the regions of the original grid structure that are not located within the bounding box (see Step VI) remain empty and are not filled. So to speak, there is no visual information based on actual camera data and time data to represent this region. However, advantageously, various techniques can be applied to fill this ground region, i.e., the image region. For example, to fill this region, pixels of the surrounding environment can be propagated into this region, past ground structures can be used, or texture information from various cameras can be used.

[0031] In summary, according to the present invention, the visualization quality is significantly improved by the object not appearing distorted and appearing with a static reprojection region. A further advantage is that the top view is extended to include parking lot markings. Without using stretching, the obstacle or other vehicle may be included in the empty parking space so that the parking lot markings appear to straddle the obstacle. After removing the stretching, the locations where the parking spaces are displayed actually appear empty.

Explanation of Signs

[0032] 1 Vehicle 2 Control device 3a Camera 3b Camera 3c Camera 3d Camera 4 Camera sensor 5 Lidar sensor

Claims

1. In a method for generating a view of a camera system for a vehicle (1), in particular a surround view camera system, comprising a control device (2) and at least one camera (3a - 3d), the following method steps: - Detecting at least one object based on the ambient environment data of the at least one camera (3a - 3d); - Creating a bounding box of the object; - Projecting the object onto the ground plane; - Creating a boundary shape including the bounding box and the projected object; - Creating a mesh structure of the boundary shape; and - Placing the mesh structure within the bounding box, wherein the view is generated using the step of adjusting the boundary shape to the size of the bounding box, in particular by image scaling and / or image resizing.

2. The method according to claim 1, wherein the view is a 2D view, in particular a top view, or a 3D view, in particular including a bowl, etc.

3. The method according to claim 1 or 2, wherein the bounding box is configured in two dimensions and axially.

4. The method according to any one of claims 1 to 3, wherein the mesh structure includes a polygon mesh or a polygon grid, in particular a triangle mesh or a triangle grid.

5. The method according to any one of claims 1 to 4, wherein for creating the boundary shape, a shape is selected which is created by connecting the corners of the bounding box and other geometric shapes, in particular a polygon chain, arranged at the opposite ends of the projected points.

6. The method according to claim 5, wherein the boundary shape includes all the projected points of the object.

7. The method according to any one of claims 1 to 6, wherein placing the mesh structure within the bounding box is performed such that the corners and edges are arranged along the boundaries of the original bounding box.

8. The method according to any one of claims 1 to 7, characterized in that external and / or internal camera parameters and object data are used to create the bounding box and / or the boundary shape and / or the mesh structure.

9. The empty space generated by placing the mesh structure or the grid structure within the bounding box is - Propagating pixels of the surrounding environment into this area, and / or - Using the past ground structure for filling, and / or - Using texture information from various cameras, to fill, the method according to any one of claims 1 to 8.

10. A camera system for a vehicle (1), in particular a surround view camera system, comprising a control device (2) and a plurality of cameras (3a to 3d) arranged inside / on the vehicle (1), wherein the control device (2) is a camera system that generates a view using the cameras (3a to 3d), a camera system, characterized in that the view is generated using the method according to any one of claims 1 to 9.

Images