Camera system and method for generating a 3D view with a camera system

Abstract

Camera system, in particular surround view camera system for a vehicle (1), comprising a control device (2), several cameras (3a-3d) arranged in / on the vehicle (1), and a control unit (7) which is arranged in / on an additional vehicle, in particular a trailer (6), several additional cameras arranged in / on the additional vehicle, in particular trailer cameras (8a-8d), whereby the control device (2) generates a view based on the cameras (3a-3d), the control unit (7) generates a view based on the additional cameras, and a combined view is generated from the views of the control device (2) and control unit (7), wherein the view includes a 3D view, and the combined view is generated taking into account the length and / or width of vehicle (1) and / or additional vehicle.

Description

[0001] The present invention relates to a camera system, in particular a surround view camera system for detecting the surroundings of a vehicle, and to a method for generating a view with a camera system for a vehicle with an additional vehicle (trailer), wherein a camera system according to the invention is provided as the camera system. Technological background

[0002] Modern vehicles are increasingly being equipped with driver assistance systems that support the driver in performing driving maneuvers. In addition to radar sensors, lidar sensors, ultrasonic sensors, and / or camera sensors, these driver assistance systems also include surround-view camera systems, which allow the vehicle's surroundings to be displayed to the driver. Such surround-view camera systems typically comprise a control device and multiple cameras that provide real images of the vehicle's surroundings, which are combined, in particular, by a data processing unit of the surround-view camera system to form an image of the vehicle's surroundings. The image of the vehicle's surroundings is then shown to the driver on a display unit (such as the navigation system display). In this way, the driver can be assisted during a vehicle maneuver, for example, when reversing the vehicle or during a parking maneuver.Furthermore, surround-view cameras are typically "fisheye cameras," meaning they use a camera with a fisheye lens to produce a fisheye image. The undistorted fisheye images are then used to present the driver with various views of the surroundings, such as the front view, rear view, curb view, and the like.

[0003] Furthermore, the images can also be combined into a 360° panoramic view, allowing the driver to select the appropriate viewpoint by moving within a virtual camera scene. Various functions or views are available, such as "Bowl" or "Top View" ("bird's eye view" or "overhead view"), in which images or textures from the surround-view cameras are combined or seamlessly strung together (stitched) to form an overall view (or overall texture). The images or textures from the surround-view cameras usually have overlapping regions or areas of overlap, particularly in the bowl view, where the textures from the cameras are projected to visualize a virtual 3D bowl representing the entire area around the car. Modern surround-view camera systems can then display the resulting views to the driver, e.g., on a display, a cockpit, or a navigation system.

[0004] Today, generic surround view systems can usually be expanded with up to four additional or trailer cameras on the outside of an additional vehicle, especially a trailer, if the vehicle is to be operated together with the trailer. For example, one trailer camera on the front, one trailer camera on the rear, and one trailer camera on each side of the trailer. Due to the increased image resolution, this can achieve a considerable size when transmitting the data to the car. In addition, the supported length of the cable for connecting the trailer cameras to the control unit in the vehicle has some limitations. One possible solution is therefore to integrate a separate control unit in the trailer that collects the camera data from the trailer cameras and forwards it to the control unit in the vehicle.However, the bandwidth between the trailer control unit and the vehicle control unit is usually not sufficient to pass, for example, four image streams. Printed state of the art

[0005] DE 10 2016 224 904 A1 discloses a three-dimensional all-round vision system for a combination consisting of a vehicle and a trailer, wherein the trailer is articulated to the vehicle by means of a straight drawbar about a vertical axis of the vehicle. The all-round vision system comprises several cameras of the vehicle, several cameras of the trailer, an image processing unit, and an optical output unit. The image processing unit is configured to determine, from an image captured by one of the vehicle's cameras, a first angle of rotation by which the drawbar is rotated relative to the vehicle about its vertical axis relative to a forward orientation of the vehicle.Based on the determined first angle of rotation of the drawbar, an image area obscured by the trailer in a panoramic image of the vehicle generated by the vehicle's cameras is replaced with a corresponding image area of ​​a panoramic image of the trailer generated by the trailer's cameras. This allows a three-dimensional panoramic image of the entire vehicle combination to be displayed on the display unit.

[0006] To generate a 3D view in a surround view system, the car's four cameras are typically used and projected onto a bowl. The cameras' blind spot, i.e., the area beneath the car, is covered, and a 3D car model is rendered into the scene. For a 3D view including a trailer, this approach can be extended to eight cameras, including the trailer's four cameras. When creating 3D views, all necessary camera images are loaded into the cache for image processing. The cache size in embedded surround view systems is generally not large enough to process all four camera images simultaneously. However, this can lead to cache errors and increased processing time.

[0007] Furthermore, from US 2017 341 583 A1 a surround view system for a vehicle for towing trailers is known, in which the surround view image is generated by combining all camera images of the vehicle and trailer in a common controller.

[0008] EP 3 318 469 A1 describes a method for creating a surround view image for a vehicle with a trailer and the superimposition / combination of both depending on the orientation of the trailer.

[0009] US 9 860 445 B2 describes a trailer assistance system in which two surround view images generated in two separate control units (ECU - Electronic Control Unit) are stitched together to form a common surround view 2D view (Top View / Bird's Eye View). Object of the present invention

[0010] The present invention is therefore based on the object of providing a generic (surround view) camera system by which the communication between the trailer and the vehicle is simplified and improved in a cost-effective manner and the probability of errors is reduced. Solution to the task

[0011] The above object is achieved by the entire teaching of claim 1 and the subordinate claim. Advantageous embodiments of the invention are claimed in the subclaims.

[0012] The camera system according to the invention, in particular a surround-view camera system for a vehicle, comprises a control device, a plurality of (surround-view) cameras arranged in / on the vehicle, a control unit arranged in / on an additional vehicle (in particular a trailer), and a plurality of additional or trailer cameras arranged in / on the additional vehicle. The control device generates a view based on the cameras on the vehicle or the streams / images / data from the surround-view cameras, while the control unit generates a view based on the additional or trailer cameras or the streams / images / data from the trailer cameras. Subsequently, a combined view is generated from the views of the control device and the control unit.The method according to the invention reduces the required bandwidth as well as the number of images required simultaneously in the cache, thereby significantly simplifying and improving communication between the trailer and the vehicle. Instead of three or four camera streams, only one image needs to be transmitted from the trailer control unit to the vehicle's control unit for each frame. With this approach, each control unit only needs to process four camera streams at a given time, which reduces memory consumption and the chance of a cache error during processing. Furthermore, the described method can be implemented in a particularly simple and cost-effective manner and can also be retrofitted to existing systems.

[0013] According to the invention, a 3D view, in particular a bowl view or a bowl, is provided as the view. Alternatively, however, a different 3D view can also be generated, whereby the control device and the control unit can also generate different views, which are later combined into a single 3D view type.

[0014] According to the invention, the combined view takes into account the length and / or width of the vehicle combination, i.e., the vehicle and trailer, or a portion of the vehicle combination. This allows the entire vehicle combination to be easily visualized in a 3D representation in the vehicle and thus displayed to the driver.

[0015] According to a special embodiment of the invention, the combined view can also be created taking into account the angle between the vehicle and the trailer. This allows the view to be generated even more precisely.

[0016] Preferably, the combined view represents an extension of the 3D view of the control device and / or the view of the control unit.

[0017] Furthermore, the view of the control unit can be transmitted from the control unit to the control device and the control device can generate the combined view.

[0018] For generating the view, the control unit can expediently use a virtual camera whose position essentially corresponds to the position of the vehicle's camera directed towards the trailer, i.e. the rear-facing surround view camera or reversing camera.

[0019] In addition, the present invention also claims a method for generating a view with a camera system for a vehicle with an additional vehicle or trailer, wherein a camera system according to the invention is provided as the camera system. Description of the invention based on exemplary embodiments

[0020] The invention is described in more detail below using practical examples. They show: Fig. 1 a simplified schematic representation of an embodiment of a vehicle with a surround view camera system according to the invention; Fig. 2 a highly simplified schematic representation of the vehicle from Fig. 1 with trailer; Fig. 3 a simplified schematic representation of a design of a bowl produced using the method according to the invention; Fig. 4 a simplified schematic representation of a further embodiment of Bowl from Fig. 3, which was extended; Fig. 5 a simplified schematic representation of a further embodiment of the bowl from Fig. 4, which has been expanded to include the rear view of the trailer rear camera; Fig. 6 a simplified schematic representation of a further embodiment of the bowl from Fig. 5, where the models for vehicle and trailer were inserted, and Fig. 7 highly simplified schematic representations (a)-(c) of different dimensions of a bowl depending on the length and orientation of the trailer.

[0021] Reference number 1 in Fig. 1 denotes a vehicle with a control device 2 (ECU, Electronic Control Unit or ADCU, Assisted and Automated Driving Control Unit), which can access various actuators (e.g., steering, engine, brakes) of the vehicle 1 in order to be able to carry out control processes of the vehicle 1. Furthermore, the vehicle 1 has several surround-view cameras or cameras 3a-3d, a camera sensor 4 (or front camera), and a lidar sensor 5 for detecting the surroundings, which are controlled via the control device 2. However, the present invention also expressly includes embodiments in which no common control device 2 is provided, but rather individual control devices or control units are provided for sensor control (e.g., a separate control unit or a separate control device for controlling the cameras 3a-3d, for the corresponding data processing, and for carrying out the method according to the invention). In addition, further sensors, such as, for example,Radar or ultrasonic sensors may be provided. The sensor data can then be used for environment and object detection. As a result, various assistance functions, such as parking assistants, emergency braking assistants (EBA, Electronic Brake Assist), adaptive cruise control (ACC, Adaptive Cruise Control), lane keeping control or a lane keeping assistant (LKA, Lane Keep Assist), or the like, can be implemented. In practical terms, the assistance functions can also be implemented via control device 2 or a separate control device.

[0022] The cameras 3a-3d are part of a surround-view camera system, which is preferably controlled by the control device 2 (alternatively, a separate control system can be provided, for example), which offers a complete 360-degree view around the entire vehicle 1 by combining the fields of view of the individual surround-view cameras, e.g., 120 degrees, into an overall view or overall image. Due to the simple monitoring of the blind spot, this camera system has numerous advantages in many everyday situations. The surround-view camera system can provide the driver with various viewing angles of the vehicle 1, e.g., via a display unit (in Fig. 1 not shown). Typically, four surround-view cameras (3a-3d) are used, located, for example, in the front and rear areas and on the side mirrors. However, three, six, eight, ten, or more surround-view cameras can also be provided. These camera views or viewing angles are particularly useful when checking blind spots, changing lanes, or parking.

[0023] In Fig. Figure 2 shows a simplified view of vehicle 1, in which an additional vehicle or trailer 6 is attached to the vehicle 1 via a trailer coupling. Trailer 6 comprises a control unit 7 and four trailer cameras 8a-8d, whose data or streams can be collected by control unit 7, which in turn can transmit this data to control device 2 (e.g., wired or wirelessly), e.g., as an image or a precursor thereof.

[0024] Furthermore, the control unit 7 for the trailer 6 can be used to view the four camera images of the trailer 6 in 3D from the rear camera position (rear camera 3d) of the vehicle 1. The intrinsic and extrinsic values ​​of the camera 3d can be transmitted to the control unit 7 when starting up.

[0025] A 3D bowl geometry, or bowl 10, is generated on control unit 7, with the position of trailer 6 being the center of bowl 10. The four cameras 8a-8d of trailer 6 are used to texture the 3D bowl. Control unit 7 can also cover the blind spots of the trailer's cameras 8a-8d using known algorithms, i.e., the area 11 under trailer 6 as well as all body parts of trailer 6, as shown in Fig. 3. The bottom area of ​​the Bowl 10 is enlarged (as shown in Fig. 4 (shown by the black arrow pointing towards trailer 6) in order to include trailer 6 behind vehicle 1. After that, both the blind spot under the vehicle or area 12 as well as the body parts visible from the vehicle must be covered. The 3D view is then displayed, with a virtual camera position corresponding to the rear camera position of the real vehicle and ideally a horizontal field of view of 180°. Although a smaller field of view would also work, a field of view of 180° imitates a surround-view rear camera particularly well. The generated view is then sent to the control device 2 and replaces the rear camera stream in the 3D view (as in Fig. 5 using the demarcation between the rear camera stream and the remaining streams). As a final step, the 3D vehicle and trailer model can be included in the 3D view (as shown in Fig. 6). The trailer model can either be provided by the trailer control unit 7 during the initialization phase, or a generic model adapted to the actual dimensions of the trailer 6 can be used. The information required for positioning the 3D trailer model: position, length, width, and angle of the trailer can be provided by the system and the driver.

[0026] Furthermore, the length and angle of the trailer can be used to calculate the direction in which the bowl should be extended. Examples are Fig. 7 (a)-(c) show different dimensions of the bowl 10, which were generated depending on the length and orientation of the trailer 6. The extent of the bowl 10 is shown in Fig. (b) and Fig. 7 (c) is shown by the black arrow.

[0027] Should communication only function unidirectionally (i.e. only in one direction: either from vehicle 1 to trailer 6 or from trailer 6 to vehicle 1), either the control unit 7 of trailer 6 can be used to provide a generic rear camera position and corresponding camera parameters, which are then made available to the control unit 2 of vehicle 1, which are then used to project the generated rear view in the 3D bowl 10. Or the control unit 7 for trailer 6 can be used instead of the rear camera position - the respective camera configuration can thus be made available to the control unit 2 of vehicle 1. List of reference symbols 1 vehicle 2 Control device 3a Camera 3b Camera 3c Camera 3D camera 4 camera sensor 5 Lidar sensor 6 trailers 7 Control unit 8a Trailer Camera 8b Trailer Camera 8c trailer camera 8d trailer camera 9 Rear view camera (3d) 10 Bowl 11 Area 12 Area

Claims

[1] Camera system, in particular surround view camera system for a vehicle (1), comprising a control device (2), several cameras (3a-3d) arranged in / on the vehicle (1), and a control unit (7) which is arranged in / on an additional vehicle, in particular a trailer (6), several additional cameras arranged in / on the additional vehicle, in particular trailer cameras (8a-8d), whereby the control device (2) generates a view based on the cameras (3a-3d), the control unit (7) generates a view based on the additional cameras, and a combined view is generated from the views of the control device (2) and control unit (7), wherein the view includes a 3D view, and the combined view is generated taking into account the length and / or width of vehicle (1) and / or additional vehicle. [2] Camera system according to claim 1, characterized bythat the view includes a bowl (10). [3] Camera system according to one of the preceding claims, characterized by that the combined view takes into account the angle between vehicle (1) and additional vehicle. [4] Camera system according to one of the preceding claims, characterized by that the combined view represents an extension of the view of the control device (2) and / or the view of the control unit (7). [5] Camera system according to one of the preceding claims, characterized by that the view of the control unit (7) is transmitted from the control unit (7) to the control device (2) and the control device (2) generates the combined view. [6] Camera system according to one of the preceding claims, characterized bythat the control unit (7) uses a virtual camera to generate the view, the position of which camera essentially corresponds to the position of the camera (3d) of the vehicle (1) directed towards the additional vehicle. [7] Method for generating a 3D view with a camera system, for a vehicle (1) with an additional vehicle, in particular a trailer (6), wherein a camera system according to one of the preceding claims is provided as the camera system.

Images