Method for generating view by using image pickup system and image pickup system

Abstract

The invention relates to a method for generating a view for a camera system, in particular for a panoramic camera system for a vehicle (1), comprising a control device (2) and at least one camera device (3a-3d), the view being generated according to the following method steps: detecting at least one object from ambient data of the at least one camera device (3a-3d); generating a bounding box of the object; projecting the object onto the ground; creating a boundary shape including a bounding box and a projection object; a network structure or a grid structure is established for the boundary shape, and the network structure or the grid structure is arranged in the bounding box, and the boundary shape is adjusted, in particular by image scaling and / or image deformation, so that the boundary shape is adapted to the size of the bounding box.

Description

Technical Field

[0001] The present invention relates to a method for generating a view for a vehicle using a camera system, and a camera system, in particular a panoramic view camera system for detecting the surroundings of a vehicle, which can generate a view for the vehicle using the method according to the present invention. Background Art

[0002] Modern vehicles are increasingly equipped with driver assistance systems (HAS) that support drivers during driving maneuvers. In addition to radar sensors, lidar sensors, ultrasonic sensors, and / or camera sensors, these HAS systems particularly include surround-view camera systems that display the vehicle's surroundings to the driver. These systems typically include control units and multiple cameras that provide realistic images of the vehicle's environment. These images are then combined by the HAS system's data processing unit to form a comprehensive image of the vehicle's surroundings. A display unit (such as a navigation system display) then shows the driver this image. This provides support for driver maneuvers such as reversing or parking. Furthermore, HAS cameras are often "fisheye cameras," meaning they have a fisheye lens that provides a fisheye image. Distortionless fisheye images are used to show the driver different views of the surroundings, such as the front view, rear view, and curb view. Modern HAS systems can then display these views to the driver, for example, on a display screen, in the cockpit, or on a navigation system.

[0003] Furthermore, these images can be merged into a 360° panoramic view, allowing the driver to select the appropriate viewpoint by moving within the virtual camera setup. Various functions or views exist, such as a "bowl view" or "top view" ("bird's-eye view" or "top view"), in which images or textures from the panoramic surround-view camera are merged or seamlessly stitched into a single view (or texture). The images or textures from the panoramic surround-view camera typically have overlapping areas or regions. Particularly in the bowl view, the camera's texture is projected onto a virtual 3D bowl representing the entire area surrounding the vehicle. Additionally, the camera system's texture information can be projected onto a grid (projection plane) or a static 2D plane, for example, to generate a top view. However, the resulting view may cause visual distortion or interference with detected objects. This is due to the reprojection of object textures onto a base surface. Since this effect can be visually distracting for the user, avoiding such distortion is crucial.

[0004] DE 10 2014 208 664 A1 discloses a camera surround view system for a vehicle, the system comprising at least one vehicle camera, wherein camera images provided by the camera are processed by a data processing unit to generate an image of the surroundings displayed on a display unit, wherein the data processing unit reprojects a texture detected by the vehicle camera onto an adaptive reprojection surface similar to the vehicle surroundings, the surface being calculated based on sensor data provided by vehicle sensors, thereby minimizing or eliminating distortion or distortion artifacts.

[0005] Furthermore, EP 2 973 420 B1 discloses a method for generating a standardized three-dimensional projection surface from camera data or images from multiple cameras to reproduce graphics in a three-dimensional virtual vehicle environment. The three-dimensional projection surface is centered on a virtual representation of the vehicle in the virtual environment. This is generated based on first polygonal model data corresponding to the shape of an object in the environment. The three-dimensional projection surface is then deformed with reference to the first polygonal model data of the object at a position in the virtual environment corresponding to the relative distance and direction of the object detected using the environment sensor data. An image can then be projected onto the deformed three-dimensional projection surface and displayed on an in-vehicle information system via a display device, wherein the displayed graphics correspond to the deformed three-dimensional projection surface having the multiple projected images. Summary of the Invention

[0006] The object underlying the present invention is therefore to provide a generic (surround view) camera system with which distortions of object representations can be prevented in order to present objects or obstacles in the vehicle surroundings as clearly and undistorted as possible.

[0007] This object is achieved by the general teaching of claim 1 and the independent claims. Advantageous embodiments of the invention are protected in the dependent claims.

[0008] In a method according to the invention for generating a view for a camera system, in particular a surround view camera system for a vehicle, the camera system comprises a control device and at least one camera, preferably a plurality of camera devices, wherein the view is generated according to the following method steps:

[0009] - detecting at least one object from surroundings data of at least one camera device,

[0010] - Generate a bounding box for the object (“bounding box”),

[0011] -project the object onto the ground,

[0012] - Create a bounding shape containing the bounding box and the projected object,

[0013] - creating a network structure (Netzstruktur) or grid structure (Gitterstruktur) for the boundary shape, and

[0014] A network or grid structure is arranged within the bounding box, wherein the bounding shape can be adapted to the size of the bounding box, in particular by image scaling and / or image warping. Image scaling, within the meaning of the present invention, refers to a resizing of the image, the bounding shape, or the network of bounding shapes. For example, the image resolution can be changed during scaling, thereby generating a new image with a higher or lower number of image points (pixels). For example, "texture mapping" or "pattern mapping" can also be performed during image scaling, wherein a model surface, in particular a three-dimensional surface model, is constructed using a two-dimensional image (texture) and, if necessary, surface properties. The texture makes the image appear more detailed and realistic. Within the context of the present invention, the bounding box can be adapted to the size of the bounding box as a model of the image of the projected object. Image warping, within the meaning of the present invention, refers to an image-based technique in which image-related depth values ​​are transformed (morphed) using, for example, so-called deformation or warping equations, so that the image can be deformed / distorted in a desired manner and / or viewed from another viewpoint (in real time).

[0015] The present invention aims to improve the diagonally distorted appearance of objects in the vehicle's surroundings. For example, in a bird's-eye or top-down view, these objects are projected onto the ground. The projected shapes of these objects are positioned in the image so that their projections appear straight and diagonally undistorted. This allows the objects in the vehicle's surroundings to be clearly visible and undistorted, significantly improving the visual appearance and spatial view of these objects in many cases.

[0016] Depending on the intended use, the view may comprise a two-dimensional (2D) view, in particular a top view, a three-dimensional (3D) view, in particular a bowl view, or the like.

[0017] The bounding box is preferably designed to be two-dimensional and axis-aligned. In particular, it can be a two-dimensional geometric shape (e.g., a circle, a polygon such as a rectangle, quadrilateral, triangle, hexagon, or the like). This shape can be selected in a practical manner based on the outline or contour of the respective object or the associated detection point.

[0018] According to a preferred embodiment of the present invention, the network structure or grid structure comprises a triangular grid or a triangular grid. However, other forms such as polygonal networks or polygonal grids of different shapes are also conceivable.

[0019] In step 4, the bounding shape is preferably created by connecting the corners of the bounding box with another geometric shape, in particular a shape comprising a polygon (e.g., a triangle, quadrilateral, or rectangle), which is arranged at opposite ends of the projection point, i.e., for example, a rectangle or quadrilateral spanning the outer point (other polygons or circles are also possible, depending on the contour of the object). For example, the other geometric shape can also have a contour similar to that of the bounding box or a portion of the bounding box. This can significantly improve the distortion-free or less-distorted display.

[0020] According to a preferred design, the boundary shape may include all projection points of the object, wherein possible anomalies, for example, that can be detected by limits, are not taken into account.

[0021] Depending on the intended use, the mesh structure can be arranged within the bounding box in such a way that the corners and edges of the final boundary follow the boundaries of the initial bounding box. This allows the object to practically have the same perimeter or boundary as the bounding box created in step 2, resulting in a particularly realistic and clear view. Furthermore, distortion is largely avoided.

[0022] The bounding box and / or bounding shape and / or network or grid structure are preferably created using extrinsic camera parameters and / or intrinsic camera parameters and object data (preferably using three-dimensional data determined from surrounding data from the camera or other sensors). In the context of the present invention, intrinsic parameters are camera parameters that are fixedly associated with a specific camera or digitizer. In contrast, extrinsic parameters are camera parameters external to the camera that can change with changes in the world view (the camera's position / orientation / orientation in a world coordinate system). In terms of a camera model, this means that the extrinsic parameters define the camera's position and orientation relative to the world view. In contrast, intrinsic parameters enable a correspondence between camera coordinates and pixel coordinates in the image or field of view (the relationship between the camera and the image coordinate system), such as the focal length f and the optical center in the image plane. A camera model can be described as a mapping of world coordinates onto image coordinates, achieved through a 3D to 2D conversion. Intrinsic parameters do not depend on the position and orientation of the camera in the world, but rather describe the mapping and the internal geometry of the camera.

[0023] Depending on the intended use, unoccupied areas created by the arrangement of a grid or grid structure within the bounding box are filled by propagating pixels from the surrounding area into these areas and / or using historical ground structures and / or texture information from different cameras. This can significantly improve the view.

[0024] Furthermore, the present invention also comprises a camera system, in particular a surround view camera system for a vehicle, comprising a control device and one or more camera devices arranged in / on the vehicle, wherein the control device generates a view based on the method according to the present invention and the camera devices or camera device data or camera device images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will now be described in detail with reference to embodiments that meet the objectives. The accompanying drawings show:

[0026] Figure 1 A simplified schematic diagram of a vehicle design is shown, which is equipped with a (panoramic surround view) camera system according to the present invention;

[0027] Figure 2 A simplified schematic diagram of the method process according to the present invention is shown, and

[0028] Figure 3 A simplified schematic diagram of the method according to the invention is shown schematically according to different steps (A to F), wherein a vehicle detects an object based on detection points and generates a view of the object according to the method according to the invention. DETAILED DESCRIPTION

[0029] Figure 1Reference numeral 1 in the figures indicates a vehicle with a control device 2 (electronic control unit (ECU) or advanced driver assistance system (ADCU)). This control device has access to different actuators of the vehicle 1 (e.g., steering mechanism, motor, braking mechanism, etc.) to implement the control process of the vehicle 1. Furthermore, to detect the surrounding environment, the vehicle 1 also has multiple panoramic surround-view cameras or cameras 3a to 3d, camera sensors 4 (or front-facing cameras), and lidar sensors 5, which are controlled by the control device 2. However, the invention also explicitly includes designs that do not have a common control device 2, but instead have separate control devices or control units for controlling the sensors (e.g., separate control units or separate control devices for controlling camera devices 3a to 3d, for corresponding data processing, and for implementing the methods according to the invention). In addition, other sensors, such as radar sensors or ultrasonic sensors, may also be provided. Sensor data can be used for surrounding environment identification and object identification. Therefore, various auxiliary functions can be implemented, such as parking assistance, emergency braking assistance (electronic brake assist (EBA)), distance following control (adaptive cruise control (ACC)), lane keeping control or lane keeping assist (lane keeping assist (LKA)), or similar auxiliary functions. The implementation of auxiliary functions can also be carried out practically through control device 2 or a separate control device. Here, camera devices 3a to 3d are part of a panoramic surround-view camera system, which is preferably controlled by control device 2 (alternatively, a separate control device may be provided). This panoramic surround-view camera system provides a complete 360-degree view around the entire vehicle 1 by, for example, merging the views of a single 120-degree panoramic surround-view camera device into a single overall view or image. This camera system offers numerous advantages in many everyday situations through simple blind spot monitoring. The panoramic surround-view camera system can be accessed, for example, via a display unit (…). Figure 1 (Not shown) The vehicle 1 is displayed to the driver from different perspectives. Typically, four surround-view cameras 3a to 3d are used, positioned on, for example, the front, rear, and side mirrors. Alternatively, three, six, eight, ten, or more surround-view cameras can be used. These camera views or perspectives are particularly useful for checking blind spots, changing lanes, or parking.

[0030] The schematic diagram of the method according to the present invention is as follows Figure 2 As shown, and has the following method steps.

[0031] Step 1: Detect one (or more) objects from the 3D surrounding environment data ( Figure 3 A). This step mainly depends on the camera data, sensor data and surrounding environment data. For example, these data can be provided in the form of point clouds (such as Figure 3A is based on black dots or detection points), where so-called point groups can be identified if they are above a certain threshold, for example, above the ground. Step 2: Generate object bounding boxes from the vehicle top view ( Figure 3 B), in particular, bounding boxes aligned with two-dimensional axes, such as XY axes, where the minimum and maximum values ​​along each axis are used for a specific set of object points.

[0032] Step 3: Project the object onto the ground ( Figure 3 C), where, for example, the camera device is located to the left of the point, and the point is not on the ground, so if projected onto the ground, the point distribution extends beyond the boundaries of the bounding box (according to...). Figure 3 (as indicated by the triangle point in C).

[0033] Step 4: Create or calculate a boundary shape that includes both the bounding box and the projected object. Figure 3 D) Here, a simple shape can be selected that encompasses both the bounding box itself and the projected object in a practical manner. This simple shape can be, for example, a concatenation of the bounding box, another rectangle at the other end of the projected point, and a concatenation of their corners. However, the resulting shape preferably encompasses all projected points within the generated area.

[0034] Step 5: Create a network structure or grid structure (triangle network or triangle grid) for the boundary shape ( Figure 3 E) wherein a triangular network structure is preferably created in order to represent the boundary shape by connecting the corners of the shape using a polygon-triangle method.

[0035] Step 6: Set the network structure or grid structure within the bounding box ( Figure 3 F), wherein the network structure or triangular mesh in step 5 is adjusted in such a way that the corners and edges are arranged along the boundaries of the initial bounding box. The shape thus formed essentially has the shape of the bounding box (in step 2). Here, the bounding shape is adjusted by image scaling and / or image deformation to fit the bounding box size. This "texture mapping step" converts or deforms the network structure or grid structure in step 5 into (e.g. Figure 3 (F) essentially corresponds to the boundaries of the bounding box, and the object display is adjusted accordingly. This makes the objects appear more natural in subsequent displays and gives the user a better sense of their orientation. This can make parking, for example, much easier.

[0036] During the transition from step 5 to step 6, or when the grid structure or grid structure established for the bounding box is deleted (step 5), the area of ​​the initial grid structure that is not within the bounding box (see step 6) remains unoccupied or unfilled. It can be said that there is no visual information in the current camera data and time data to illustrate this area. However, different methods can be used to fill this ground area or image region in a beneficial manner, such as migrating pixels from the surrounding environment to this area, using historical ground structure to fill these areas, or using texture information from different cameras.

[0037] Overall, this invention significantly improves visualization quality, as objects appear undistorted and have static reprojected surfaces. Further advantages arise when the top view is extended with parking lot markings. Without stretching, obstacles or other vehicles can be incorporated into unoccupied parking lots, making the parking lot markings appear to rest on the obstacles. Without stretching, areas displaying parking lots appear unoccupied.

[0038] Reference Signs List

[0039] 1 vehicle

[0040] 2. Control device

[0041] 3a Camera device

[0042] 3b Camera device

[0043] 3C camera device

[0044] 3D camera device

[0045] 4. Camera sensor

[0046] 5. LiDAR Sensor

Claims

1. A method for generating a view for a camera system, in particular a surround view camera system for a vehicle (1), the camera system comprising a control device (2) and at least one camera device (3a to 3d), wherein: Create a view using the following steps: - detecting at least one object from surrounding data of at least one camera device (3a to 3d), - Generate bounding boxes of objects; - Project objects onto the ground; - Create a bounding shape containing a bounding box and a projected object; - establishing a network structure for said boundary shape, and The network structure is arranged within the bounding box, wherein the bounding shape is adapted, in particular by image scaling and / or image deformation, so that the bounding shape is adapted to the size of the bounding box.

2. The method according to claim 1, characterized in that: The views include two-dimensional (2D) views, in particular top views, three-dimensional (3D) views, in particular bowl views or the like.

3. The method according to claim 1 or 2, characterized in that: The bounding box is set to be two-dimensional and axis-aligned.

4. The method according to any one of the preceding claims, characterized in that The network structure comprises a polygonal network or a polygonal grid, in particular a triangular network or a triangular grid.

5. The method according to any one of the preceding claims, characterized in that To create a bounding shape, a shape is selected which is created by connecting the corners of the bounding box with another geometric shape, in particular a shape comprising a polygon, arranged at opposite ends of the projection point.

6. The method according to claim 5, characterized in that: The bounding shape includes all the projected points of the object.

7. A method according to any one of the preceding claims, characterized in that The network structure is set inside the bounding box in such a way that the corners and edges are set along the boundaries of the initial bounding box.

8. The method according to any one of the preceding claims, characterized in that A bounding box and / or a bounding shape and / or a network structure is created using extrinsic camera parameters and / or intrinsic camera parameters and the object data.

9. The method according to any one of the preceding claims, characterized in that The unoccupied areas resulting from the arrangement of the grid structure or the grid structure within the bounding box are filled in the following way: - migrate pixels from the surrounding environment into this area, and / or - use of historic ground structures for infill, and / or -Use texture information from different cameras.

10. A camera system for a vehicle (1), in particular a panoramic view camera system, the camera system comprising: A control device (2), a plurality of camera devices (3a to 3d) arranged in / on the vehicle (1), and The control device (2) generates views according to the camera devices (3a to 3d), It is characterized in that A view is generated according to the method of any of the preceding claims.

Images