Method for generating a view using a camera system and camera system

Abstract

The present invention relates to a method for generating a view of a camera system for a vehicle (1), in particular a surround view camera system, comprising a control device (2) and at least one camera (3a to 3d), the following method steps: - detecting at least one object based on the ambient data of at least one camera (3a to 3d); - creating a bounding box of the object; - projecting the object onto the ground plane; - creating a boundary shape including the bounding box and the projected object; - creating a mesh structure or a grid structure of the boundary shape; and - placing the mesh structure or the grid structure within the bounding box, the method for generating a view using a step of adjusting the boundary shape to the size of the bounding box, in particular by image scaling and / or image resizing.

Description

Technical Field

[0001] The present invention relates to a method for generating a view using a vehicle camera system and a vehicle camera system for detecting a surrounding environment capable of generating a view using the method according to the present invention, particularly to a surround view camera system.

Background Art

[0002] Recent vehicles are increasingly equipped with a driving assistance system that assists a driver when performing a driving operation. These driving assistance systems include, in addition to radar sensors, lidar sensors, ultrasonic sensors, and / or camera sensors, particularly a surround view camera system that enables the vehicle's surrounding environment to be displayed to the vehicle driver. Such a surround view camera system generally includes a control device and a plurality of cameras, and these plurality of cameras provide real images of the vehicle's surrounding environment, and these real images of the vehicle's surrounding environment are combined with a surrounding environment image of the vehicle's surrounding environment by a data processing device of the surround view camera system. Then, an image of the vehicle's surrounding environment is displayed on a display device (e.g., a display of a navigation system) for the driver. In this way, the driver can be assisted during a driving operation, for example, when the vehicle is reversing or performing a parking operation. Also, a surround view camera is generally a "fisheye camera", that is, a camera equipped with a fisheye lens that provides a fisheye image. Here, a non-distorted fisheye image is used to represent various views of the surrounding environment for the driver, such as a front view, a rear view, a curb view, etc. And recent surround view camera systems can display the view thus generated to the driver, for example, on a display, a cockpit, or a navigation system.

[0003] Furthermore, images can be combined with a 360° panoramic view, allowing the driver to select the appropriate viewpoint by moving within the virtual camera scene. In this case, various functions or views are available, such as a "bowl" or "top view" ("bird's-eye view" or "top view"), combining the surround view camera image or texture with the overall view (or overall texture), i.e., seamlessly continuing them together (stitching). In this case, the surround view camera image or texture generally includes overlapping areas. In particular, in a bowl view, the camera texture is projected, visualizing a virtual 3D bowl representing the entire area around the vehicle. Also, the texture information of the camera system can be projected onto a mesh (projection area), i.e., a static 2D surface, creating a top view, for example. However, in views created in this way, detected objects may be visually distorted or grotesque. This occurs because the object texture is reprojected onto the ground. Since this effect is visually disruptive to the user, avoiding such distortion is of particular interest.

[0004] Prior art documents Patent Document 1 discloses a vehicle camera surround view system having at least one in-vehicle camera that provides camera images to create a surrounding environment image displayed on a display unit, wherein the data processing unit reprojects textures captured by the in-vehicle camera onto an adaptive reprojection region similar to the surrounding environment of the vehicle calculated based on sensor data provided by an in-vehicle sensor, thereby minimizing or eliminating distortion or distorted artifacts.

[0005] Furthermore, Patent Document 2 provides a method for generating a standard three-dimensional projection area centered on a virtual representation of a vehicle in a virtual surrounding environment, based on camera data or images from multiple cameras, in order to reproduce graphics in the vehicle's three-dimensional virtual surrounding environment. In this case, generation is performed based on first polygon model data corresponding to the shapes of objects in the surrounding environment. The three-dimensional projection area is then deformed with respect to the first polygon model data of objects located in the virtual environment corresponding to the relative distance and direction of objects detected based on the surrounding environment sensor data. Subsequently, an image can be projected onto the deformed three-dimensional projection area and displayed using a display device of an in-vehicle information system, and the displayed graphics correspond to the deformed three-dimensional projection area having multiple projected images. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] German Patent Application Publication No. 102014208664 [Patent Document 2] European Patent No. 2973420 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Therefore, the object of the present invention is to provide a higher-level (surround view) camera system that prevents the representation of distorted objects in order to represent objects or obstacles in the vehicle's surrounding environment as clearly and without distortion as possible. [Means for solving the problem]

[0008] This problem is solved by the teachings of claim 1 and the parallel independent claims as a whole. Preferred embodiments of the present invention are claimed in the dependent claims.

[0009] A method for generating a view in a vehicle camera system, particularly a surround-view camera system, comprising a control device and at least one camera, preferably multiple cameras, wherein the method steps are as follows: - A step of detecting at least one object based on ambient environmental data from at least one camera; - Steps to create an object's bounding box; - A step of projecting an object onto the ground surface; - Steps to create a boundary shape that includes the bounding box and the projected object; - Steps to create a mesh or grid structure of the boundary shape; and - A view is generated using the step of placing a mesh structure or grid structure within a bounding box, wherein the boundary shape can be adjusted to the size of the bounding box, in particular by image scaling and / or image resizing. In this invention, image scaling is understood as resizing an image or boundary shape or a mesh structure of a boundary shape. For example, the image resolution can be changed during scaling, so that a new image with more or fewer image points (pixels) can be created. For example, "texture mapping" or "pattern mapping" can also be performed during image scaling, so that the surface of a model, in particular a 3D surface model, is constructed using a 2D image (texture), and, if applicable, surface properties are also constructed. In this case, the texture makes the image appear more detailed and more realistic. Also in relation to this invention, a bounding box as a kind of model having an image of a projected object can be adjusted to the size of the bounding box. Image resizing or “image warping” (image deformation or image resizing) is understood in the present invention as an image-based technique that transforms, for example, depth values ​​associated with an image using so-called deformation equations or warping equations so that the image can be deformed / resized as desired and / or observed from other viewpoints (in real time) (“morphing” or “image morphing”).

[0010] The present invention aims to improve the diagonally distorted appearance of objects in the vehicle's surrounding environment. For example, in a bird's-eye view, or top view, these objects are projected onto the ground, and the projected shapes of these objects are positioned in the image so that their projections appear straight and not diagonally distorted. As a result, objects in the vehicle's surrounding environment are represented clearly and without distortion, which in many cases significantly improves the visual appearance and spatial representation of these objects.

[0011] Preferably, the views may include two-dimensional views, particularly top views, and three-dimensional views, particularly bowl views, etc.

[0012] Preferably, the bounding box is configured in two dimensions and axially. In particular, in this case, it may be a two-dimensional geometric shape (e.g., a circle, polygon, e.g., rectangle, quadrilateral, triangle, hexagon, etc.). This shape can actually be selected depending on the outline or contour of each object, or the detection points associated with it.

[0013] According to a preferred embodiment of the present invention, the mesh structure or grid structure includes a triangular mesh or triangular grid. However, other shapes, such as a differently configured polygon mesh or polygon grid, are also possible.

[0014] Preferably, in step IV, the shape to be created is selected by connecting other geometric shapes, particularly shapes including polygonal chains (e.g., triangular, quadrilateral, or rectangular), that are positioned at the corners of the bounding box and the opposite ends of the projected points, i.e., rectangles or quadrilaterals (or, in particular, other polygons or circular shapes, depending on the contour of the object) that extend to the outer points. In this case, the boundary shape may also have a contour similar to, for example, the bounding box or a part of the bounding box. This particularly improves the representation to be free of or with minimal distortion.

[0015] According to a preferred embodiment, the boundary shape can include all projected points of the object and can avoid taking into account, for example, any detectable outliers exceeding a threshold.

[0016] Preferably, the mesh structure is placed within the bounding box so that the corners and edges of the final boundary are aligned with the boundaries of the original bounding box. In practice, this results in an object that has the same extent or the same boundary as the bounding box created in step II, making it particularly realistic and sharp. Distortion is also particularly avoided.

[0017] Preferably, external and / or internal camera parameters and object data (preferably 3D data calculated based on ambient environmental data of the camera or other sensor technology) are used to create bounding boxes and / or boundary shapes and / or mesh or grid structures. In this invention, internal parameters are understood as camera parameters that are fixed and coupled internally to a particular camera or digitization device. External parameters, on the other hand, are camera parameters that are outside the camera and can change with respect to the world view (the camera's location / position / orientation in the world coordinate system). With respect to the camera model, this means that external parameters define the camera's position and orientation with respect to the world view. Internal parameters, on the other hand, enable the assignment (correlation between the camera coordinate system and the image coordinate system) between the camera coordinate system and the image coordinate system within the image or field of view, such as the focal length f and the optical center of the image plane. The camera model is, so to speak, a mapping from world coordinates to image coordinates, which is done using a 3D to 2D transformation. Here, internal parameters describe the mapping and the internal geometry of the camera, independent of the camera's position and orientation in the world.

[0018] Preferably, empty areas resulting from placing a mesh or grid structure within a bounding box can be filled by propagating pixels from the surrounding environment into these areas, and / or by using past ground structures to fill them, and / or by using texture information from various cameras. This significantly improves the view.

[0019] Furthermore, the present invention also includes a camera system for vehicles, particularly a surround-view camera system, comprising a control device and one or more cameras located inside / on a vehicle, wherein the control device generates a view using the cameras, and the control device generates a view using the method according to the present invention and the cameras or camera data or camera images. [Brief explanation of the drawing]

[0020] Hereinafter, the present invention will be described in more detail based on preferred embodiments. [Figure 1] FIG. 1 shows a simplified schematic view of the configuration of a vehicle equipped with a (surround view) camera system according to the present invention. [Figure 2] FIG. 2 shows a simplified schematic view of a method process according to the present invention. [Figure 3] FIG. 3 shows a simplified view of a method according to the present invention based on various steps (A to F), in which a vehicle detects an object based on detection points, and a view of this object is generated using the method according to the present invention.

Mode for Carrying Out the Invention

[0021] Reference numeral 1 in Figure 1 indicates a vehicle equipped with a control device 2 (ECU (Electronic Control Unit) or ADCU (Assisted and Automated Driving Control Unit)) that can access various actuators of the vehicle 1 (e.g., steering, motors, brakes) so that it can perform control processes on the vehicle 1. The vehicle 1 also includes a plurality of surround-view cameras or cameras 3a-3d, a camera sensor 4 (or front camera), and a lidar sensor 5, all controlled via the control device 2 to detect the surrounding environment. However, the present invention also explicitly includes embodiments in which, instead of a common control device 2, separate control devices or control units are provided for sensor control (e.g., separate control units or separate control devices for controlling cameras 3a-3d, for corresponding data processing, and for performing methods according to the present invention). Further sensors, such as radar sensors or ultrasonic sensors, may also be provided. In this case, the sensor data can be used for surrounding environment detection and object detection. Therefore, various support functions can be implemented, such as parking assist, emergency braking assist (EBA (Electronic Brake Assist)), distance following control (ACC (Adaptive Cruise Control)), lane keeping control or lane keeping assist (LKA (Lane Keep Assist)), etc. In practice, these support functions can also be performed via control unit 2 or a separate control unit.

[0022] Here, cameras 3a to 3d are preferably part of a surround-view camera system controlled by a control device 2 (or, for example, a separate control device may be provided), which provides a complete 360-degree view of the entire vehicle 1 by combining the fields of view of individual surround-view cameras, for example 120 degrees, into an overall view or overall picture. Due to the ease of monitoring blind spots, this camera system has numerous advantages in many everyday situations. The surround-view camera system can show the driver various viewpoints of the vehicle 1, for example, via a display device (not shown in Figure 1). Typically, four surround-view cameras 3a to 3d are used, for example, located in the front and rear areas and on the side mirrors. Alternatively, three, six, eight, ten or more surround-view cameras may be provided. The views or viewpoints of these cameras are particularly useful when checking blind spots, changing lanes, or parking.

[0023] The method according to the present invention is schematically shown in Figure 2 and comprises the following method steps.

[0024] Step I: A step to detect one (or more) objects based on 3D surrounding environment data (Figure 3A). This step basically relies on camera data or sensor data and surrounding environment data. For example, this data may be available in the form of a point cloud (shown in Figure 3A using black dots, i.e., detection points), and so-called point clusters are detected when, for example, they are located above a certain threshold ("threshold") from the ground surface.

[0025] Step II: Based on a top-down view of the vehicle, create bounding boxes for the object, particularly in two dimensions and along the axes (Figure 3B). For example, for the XY axes, the minimum and maximum values ​​for each axis are used for a given set of object points.

[0026] Step III: This step involves projecting the object onto the ground (Figure 3C). For example, if the camera is positioned to the left of the point and the point is not located on the ground, when the projection onto the ground occurs, the point will extend beyond the bounding box boundary (shown using a triangular point in Figure 3C).

[0027] Step IV: Create or calculate a boundary shape that includes both the bounding box and the projected object (Figure 3D). In this case, a simple shape that includes both the bounding box itself and the projected object can be selected. This simple shape may be, for example, the confluence of the bounding boxes, other rectangles at the other ends of the projected points, and the confluence of their corners. However, it is preferable that the resulting shape includes all the projected points within the created area.

[0028] Step V: A step to create a mesh structure or grid structure (triangular mesh or triangular grid) of the boundary shape (Figure 3E), wherein, preferably, to represent the boundary shape, a triangular mesh structure is created by connecting the corners of the shape using a polygonal triangulation method.

[0029] Step VI: This step involves placing a mesh or grid structure within the bounding box (Figure 3F), adjusting the mesh structure from Step V, i.e., the triangular mesh, so that its corners and edges align with the boundaries of the original bounding box. The resulting shape substantially possesses the shape of the bounding box (from Step II). The boundary shape is then adjusted to the size of the bounding box by image scaling and resizing. In essence, this "texture mapping step" transforms or deforms the mesh or grid structure from Step V into a mesh or grid structure (as shown in Figure 3F) that substantially matches the boundaries of the bounding box, thus adjusting the object representation accordingly. As a result, the object appears very natural in subsequent displays, conveying a better sense of direction to the user. This, for example, simplifies the parking process in particular.

[0030] When transitioning from Step V to Step VI, that is, when deleting the mesh or grid structure created for the boundary shape (Step V), the areas of the original grid structure that are not located within the bounding box (see Step VI) remain empty and unfilled. In other words, there is no visual information based on actual camera data and time data to represent this area. However, advantageously, various techniques can be applied to fill this ground area, or image area. For example, pixels from the surrounding environment can be propagated into this area, past ground structures can be used, or texture information from various cameras can be used to fill this area.

[0031] In summary, the present invention significantly improves visualization quality by ensuring that objects do not appear distorted and that they have a static reprojection area. A further advantage is that the top view is extended to include parking markings. Without stretching, obstacles or other vehicles may be included in the vacant parking space, making it appear as if the parking markings are positioned across the obstacles. After removing the stretching, the area where the parking space is displayed actually appears vacant. While this application relates to the invention described in the claims, it also includes the following other aspects. 1. A camera system for a vehicle (1), comprising a control device (2) and at least one camera (3a-3d), in particular a method for generating a view of a surround-view camera system, comprising the following method steps: - A step of detecting at least one object based on ambient environment data from at least one camera (3a to 3d); - Step of creating a bounding box for the object; - A step of projecting the aforementioned object onto the ground surface; - A step of creating a boundary shape that includes the bounding box and the projected object; - Steps to create a mesh structure for the boundary shape; and - A method for generating the view, comprising the steps of arranging the mesh structure within the bounding box, wherein the boundary shape is adjusted to the size of the bounding box, particularly by image scaling and / or image resizing. 2. The method according to claim 1, characterized in that the views include a two-dimensional view, in particular a top view, a three-dimensional view, in particular a bowl, etc. 3. The method according to 1 or 2, characterized in that the bounding box is configured in two dimensions and in the axial direction. 4. The method according to any one of 1 to 3 above, characterized in that the mesh structure includes a polygon mesh or a polygon grid, in particular a triangular mesh or a triangular grid. 5. The method according to any one of 1 to 4, characterized in that, in order to create the boundary shape, the shape to be created is selected by connecting other geometric shapes, particularly shapes including polygonal chains, that are located at the corners of the bounding box and the opposite ends of the projected points. 6. The method according to the above-mentioned claim, characterized in that the boundary shape includes all projected points of the object. 7. The method according to any one of claims 1 to 6, characterized in that the mesh structure is placed within the bounding box such that the corners and edges are positioned along the boundaries of the original bounding box. 8. The method according to any one of 1 to 7, characterized in that external and / or internal camera parameters and object data are used to create the bounding box and / or the boundary shape and / or the mesh structure. 9. The empty area created by arranging the mesh structure or grid structure within the bounding box is - To propagate pixels from the surrounding environment into this region, and / or, - Using past ground structures to fill in, and / or, - A method according to any one of the above 1 to 8, characterized by filling in by using texture information from various cameras. 10. In a camera system for a vehicle (1), particularly a surround-view camera system, Control device (2), The vehicle (1) is equipped with a plurality of cameras (3a to 3d) located inside / on top of it. The control device (2) is a camera system that generates a view using the cameras (3a to 3d), A camera system characterized in that the view is generated using the method described in any one of the above 1 to 9. [Explanation of Symbols]

[0032] 1 vehicle 2 Control device 3a Camera 3b camera 3c camera 3D camera 4 Camera Sensors 5 Lida Sensor

Claims

1. A camera system for a vehicle (1), comprising a control device (2) and at least one camera (3a to 3d), in particular a method for generating a view of a surround-view camera system, comprising the following method steps: - A step of detecting at least one object based on ambient environment data from at least one camera (3a to 3d); - Step of creating a bounding box for the object; - A step of projecting the object onto the ground surface; - A step of creating a boundary shape that includes the bounding box and the projected object; - A step of creating a mesh structure of the boundary shape, wherein the mesh structure includes a polygon mesh or polygon grid, or a triangular mesh or triangular grid; and - A method for generating the view, comprising the steps of arranging the mesh structure within the bounding box, wherein the boundary shape is adjusted to the size of the bounding box, particularly by image scaling and / or image resizing.

2. The method according to claim 1, characterized in that the view includes a two-dimensional view, a top view, or a three-dimensional view.

3. The method according to claim 1, characterized in that the bounding box is configured in two dimensions and in the axial direction.

4. The method according to claim 1, characterized in that, in order to create the boundary shape, the shape to be created is selected by connecting other geometric shapes, particularly shapes including polygonal chains, that are located at the corners of the bounding box and the opposite ends of the projected points.

5. The method according to claim 4, characterized in that the boundary shape includes all projected points of the object.

6. The method according to claim 1, characterized in that the mesh structure is placed within the bounding box such that the corners and edges are positioned along the boundaries of the original bounding box.

7. The method according to claim 1, characterized in that external and / or internal camera parameters and object data are used to create the bounding box and / or the boundary shape and / or the mesh structure.

8. The empty space created by placing the mesh structure within the bounding box is - To propagate pixels from the surrounding environment into this region, and / or, - Using past ground structures to fill in, and / or, - The method according to claim 1, characterized by filling by using texture information from various cameras.

9. In a camera system for a vehicle (1), particularly a surround-view camera system, Control device (2), The vehicle (1) is equipped with a plurality of cameras (3a to 3d) located inside / on top of it, The control device (2) is a camera system that generates a view using the cameras (3a to 3d), A camera system characterized in that the view is generated using the method described in any one of claims 1 to 8.

Images