THREE-DIMENSIONAL ALL-AROUND VISION SYSTEM

Description

[0001] The invention relates to a three-dimensional all-round vision system for a combination consisting of a vehicle and a trailer, wherein the trailer is articulated to the vehicle by means of a straight drawbar at least around a vertical axis of the vehicle.

[0002] US 2012 / 0262580 A1 discloses an all-round view system that can provide a panoramic view of a vehicle using, for example, four cameras positioned at different locations on the vehicle. The cameras can generate image data corresponding to the panoramic view, and a processing device can process the image data and generate the all-round view on a simulated predetermined shape that can be viewed on a display. The simulated predetermined shape can, for example, be a shell shape.

[0003] WO 2016 / 026870 A1 describes a display system for generating a composite image of an area behind a vehicle towing a trailer. A first image from a first camera located at the rear of the vehicle is supplemented with a second image from a second camera located at the rear of the trailer. The first image and the second image are combined using an image processor, with the surrounding area obscured by the trailer in the first image being displayed using the second image.

[0004] US 2014 / 0085472 A1 and JP 2012 / 105158 A1 each describe devices which generate a bird's eye view of the vehicle's surroundings from several cameras arranged on a towing vehicle and a trailer.

[0005] If the vehicle is towing a trailer, the trailer is typically within the field of view of at least one of the vehicle's cameras and occupies an area within the vehicle's 360° panoramic image. This means that, for example, a camera located at the rear of the vehicle will largely only capture the area behind the vehicle up to the trailer, but not the potentially more important areas behind the trailer and to the sides (right and left) of the trailer. In this way, the trailer creates a blind spot in the vehicle's panoramic image.

[0006] It is the object of the present invention to provide a three-dimensional all-round vision system of the type mentioned above, in which blind spots are at least reduced.

[0007] The problem is solved by the subject matter of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims, the following description, and the figures.

[0008] According to a first aspect of the invention, a three-dimensional all-round vision system is provided for a combination comprising a vehicle and a trailer, wherein the trailer is articulated to the vehicle by means of a straight drawbar at least about a vertical axis of the vehicle.

[0009] The all-round vision system comprises several, in particular four, cameras of the vehicle, several, in particular three cameras of the trailer, an image processing unit and an optical output unit, e.g. a display in or on the dashboard of the vehicle. The vehicle is, for example, a motor vehicle such as a car, bus or lorry, or a motorcycle or a bicycle. The cameras are preferably digital cameras which can take pictures or film recordings of the area surrounding the vehicle or trailer and can store, output and transmit image data representing the pictures or film recordings. The image processing unit can access the image data from the cameras and process it according to the invention.

[0010] The vehicle's four cameras, in particular, are calibrated with respect to a coordinate system of the vehicle and arranged on the vehicle in such a way that, by combining one image each (in particular recorded simultaneously) from all four cameras, a three-dimensional panoramic image of the vehicle can be generated and displayed on the optical output unit, wherein a drawbar region of the vehicle's panoramic image shows the drawbar. The vehicle's coordinate system can, in particular, be a Cartesian coordinate system, with an x-axis defining the longitudinal direction, a y-axis defining the width direction, and a z-axis defining the vertical direction of the vehicle. In particular, the x-axis and the y-axis can span a plane that is parallel to a road surface on which the vehicle is moving. In this case, the z-axis represents a road normal.

[0011] Furthermore, the trailer's three cameras in particular are calibrated with respect to a coordinate system of the trailer and are mounted on the trailer in such a way that a three-dimensional panoramic image of the trailer can be generated by combining one image each (in particular taken simultaneously) from all three cameras, wherein the panoramic image of the trailer shows the lateral surroundings of the trailer and the rear surroundings of the trailer.

[0012] Furthermore, the image processing unit is configured to determine a first angle of rotation α from the drawbar area of the vehicle's panoramic image, by which angle the drawbar is rotated relative to the vehicle about its vertical axis relative to a forward orientation of the vehicle. The vertical axis of the vehicle is the vertical axis of the vehicle's coordinate system.

[0013] In addition, the image processing unit is configured to replace a trailer area of the vehicle panoramic image, which shows the trailer, with a corresponding image area of the trailer partial panoramic image depending on the determined first angle of rotation of the drawbar.

[0014] The trailer area of the vehicle's panoramic image forms a blind spot area, which shows the area behind the vehicle up to the trailer, but not objects obscured by the trailer in the vehicle's panoramic image, which are located behind the trailer or on either side of the trailer. However, these obscured objects can be detected by the trailer's cameras and displayed in an image area of the trailer's panoramic image. The trailer area in the vehicle's panoramic image (without the obscured objects) can be replaced by the corresponding image area of the trailer's panoramic image, whereby the image area contains the image area obscured by the trailer and any obscured objects located therein.

[0015] Based on the determined angle of rotation α around the vertical axis of the vehicle relative to the forward orientation of the vehicle, the image processing unit can select the image area of the trailer panoramic image that makes it possible to minimize or eliminate the blind spot and to display an image area obscured by the trailer and not detectable by the vehicle's cameras, or any obscured objects located therein, in the vehicle panoramic image. In other words, the panoramic view system according to the invention can display a three-dimensional panoramic image of the entire vehicle combination on the display unit, with the surrounding area to the side of the trailer and an area behind the trailer being visible, in particular, to the user of the panoramic view system or the vehicle with the panoramic view system.

[0016] For this purpose, the image processing unit can convert or transform the coordinates of objects from the image area of the trailer's panoramic image (which are available in coordinates relative to the trailer's coordinate system) into corresponding coordinates of the vehicle's coordinate system. In other words, the vehicle's panoramic image is modified based on the determined drawbar angle and consists of a first image area and a second image area. The first image area originates from the vehicle's four cameras, whereas the second image area originates from the trailer's three cameras and replaces the blind spot area of the vehicle's cameras.

[0017] According to the invention, the image processing unit is designed to convert the images taken by the vehicle’s cameras to a (simulated) movable,A three-dimensional first rear projection surface is projected onto it, and the images captured by the trailer's cameras are projected onto a movable, three-dimensional second rear projection surface. A region of the first dish can then replace a corresponding second region of the second dish in order to—as described above—display an image area obscured by the trailer and not detectable by the vehicle's four cameras, or obscured objects located therein, in the vehicle's panoramic image.

[0018] The three-dimensional rear projection surface enables, in particular, the distance between the vehicle or trailer and objects surrounding the vehicle and trailer to be displayed with high accuracy on the display unit. This ensures, in particular, that the objects on the display unit do not appear farther away from the vehicle or trailer than they actually are. In particular, the three-dimensional rear projection surface can be shaped like a bowl or dish. The bowl or dish can, for example, have a flat, e.g., rectangular, base and a parabolic side wall.

[0019] According to a further embodiment, the image processing unit is configured to determine a second angle of rotation from the drawbar region of the vehicle panoramic image, by which angle the drawbar is rotated relative to the vehicle about its width axis, and to replace the trailer region of the vehicle panoramic image, which shows the trailer, with a corresponding image region of the trailer partial panoramic image, depending on the determined first angle of rotation and the second angle of rotation of the drawbar. This embodiment enables, provided the trailer is also articulated to the vehicle by means of the straight drawbar around the width axis of the vehicle, blind spots in the vehicle panoramic image caused by vertical angular misalignment between the vehicle and trailer to be replaced by a corresponding image region of the trailer panoramic image, e.g.when the vehicle pitches when driving on uneven road surfaces or when driving over hills or mountains.

[0020] Furthermore, the image processing unit can be configured to determine a third angle of rotation from the drawbar region of the vehicle panoramic image, by which angle the drawbar is rotated relative to the vehicle about its longitudinal axis, and to replace the trailer region of the vehicle panoramic image, which shows the trailer, with a corresponding image region of the trailer partial panoramic image depending on the determined first angle of rotation, second angle of rotation, and third angle of rotation of the drawbar. This embodiment enables, provided the trailer is also articulated to the vehicle by means of the straight drawbar around the vehicle's longitudinal axis, the blind spot areas of the vehicle panoramic image caused by rolling movements of the vehicle and / or the trailer to be replaced with a corresponding image region of the trailer panoramic image.

[0021] According to a second aspect of the invention, a vehicle-trailer combination is provided, wherein the trailer is pivotally attached to the vehicle by means of a straight drawbar at least about a vertical axis of the vehicle. Furthermore, the trailer can also be pivotally attached to the vehicle about a lateral axis and a longitudinal axis of the vehicle. The vehicle-trailer combination further comprises a three-dimensional panoramic view system according to the first aspect of the invention.

[0022] In the following, embodiments of the invention are explained in more detail with reference to the schematic drawing. Fig. 1a side view of a motor vehicle with a trailer, Fig. 2a highly schematic plan view of the vehicle and the trailer according to Fig. 1 , Fig. 3 a highly schematic front view of the vehicle according to Fig. 1 , Fig. 4 a structure of an embodiment of an all-round vision system according to the invention.

[0023] Fig. 1 to 3 show a vehicle in the form of a motor vehicle 1. The motor vehicle 1 has four cameras 2 to 5, which are Fig. 2 are shown. The cameras 2 to 5 are calibrated in a Cartesian first coordinate system I of the motor vehicle 1. The coordinate system I of the motor vehicle 1 comprises a longitudinal axis x, in which a forward orientation of the vehicle 1 runs, a latitudinal axis y, which is oriented transversely to the forward orientation of the vehicle 1, and a vertical axis z of the vehicle 1, which is oriented transversely or normal to a road surface on which the vehicle 1 is moving. Rotational movements of the vehicle 1 about the longitudinal axis x are referred to as rolling movements, rotational movements of the vehicle 1 about the y-axis as pitching movements, and rotational movements of the vehicle 1 about the z-axis as yaw movements.

[0024] The motor vehicle 1 pulls a trailer 6 with three further cameras 7 to 9, which are also in Fig. 2are shown. Together, the motor vehicle 1 and the trailer 6 form a vehicle combination 10. The cameras 7 to 9 are calibrated in a Cartesian second coordinate system II of the trailer 6. The coordinate system II of the trailer 6 comprises a longitudinal axis x', a latitudinal axis y' and a vertical axis z' of the trailer 6. The trailer 6 is attached to a trailer coupling 12 of the vehicle 1 by means of a straight drawbar 11, wherein the trailer 6 is pivotably or rotatably mounted about all three axes x, y and z of the vehicle 1.

[0025] Fig. 2 shows how the four cameras 2 to 5 of the vehicle 1 and the three cameras 7 to 9 of the trailer 6 can be arranged. The vehicle 1 comprises a first camera 2, a second camera 3,a third camera 4 and a fourth camera 5. The trailer 6 comprises a fifth camera 7, a sixth camera 8, and a seventh camera 9. Furthermore, possible fields of view of the cameras 2 to 5 and 7 to 9 are shown. Furthermore, an image processing unit 13 and an optical output unit in the form of a monitor 14 are arranged within the vehicle 1, wherein the monitor 14 is located in the field of view of a driver of the vehicle 1.

[0026] In the Fig. 2In the exemplary embodiment shown, the first camera 2 is arranged in a front region of the vehicle 1 and captures a substantially front field of view 15 in the immediate vicinity of the vehicle 1. The second camera 3 and the third camera 4 are each arranged in a laterally outer region of the vehicle 1 and each capture a substantially lateral field of view 16 and 17 in the immediate vicinity of the vehicle 1. The third camera 5 is arranged in a rear region of the vehicle 1 and captures a substantially rear field of view 18 in the immediate vicinity of the vehicle 1. The four fields of view 15 to 18 of the four cameras 2 to 5 together cover the entire circumference (360°) of the vehicle 1.

[0027] The four cameras 2 to 5 of vehicle 1 can simultaneously capture images, which can be combined to form a three-dimensional panoramic image of the vehicle and displayed on monitor 14. The drawbar 11 of the trailer 6 is located in the rear field of view 18 of the fourth camera 5 of vehicle 1. The drawbar 11 is thus located within images captured by the fourth camera 5 of vehicle 1. An image area of these images from the fourth camera, which contains the drawbar 11, is referred to below as the "drawbar area." The image processing unit 13 determines from the drawbar area a first angle of rotation α by which the drawbar 11 is rotated relative to the vehicle 1 about its vertical axis z.

[0028] The image processing unit 13 further determines from the drawbar area a second angle of rotation β, by which the drawbar 11 is rotated relative to the vehicle 1 about its width axis y. This relative rotation of the drawbar 11 occurs in the Fig. 1 shown example is caused by the fact that the vehicle 1 is on a first road section 22 which is inclined relative to a second road section 23 on which the trailer 6 is located ( Fig. 1 ).

[0029] Furthermore, the image processing unit 13 determines a third angle of rotation φ by which the drawbar 11 is rotated relative to the vehicle 1 about its longitudinal axis x. Fig. 3 shows in this regard - for clarification in a highly exaggerated way - the vehicle after Fig. 1, which is located on the first road section 22 and performs a rolling movement about the longitudinal axis x, wherein a rolling angle φ in the example shown corresponds to the third rotation angle φ.

[0030] The fifth camera 7 and the sixth camera 8 are each arranged in a laterally outer region of the trailer 6 and each capture a substantially lateral and front field of view 19 and 20 respectively in a closer environment of the trailer 6. The seventh camera 9 is arranged in a rear region of the trailer 6 and captures a substantially rear field of view 21 in the closer environment of the trailer 6. Lateral boundaries of the fields of view 15 to 18 and 19 to 21 are in Fig. 2Each is indicated by dashed lines. The three cameras 7 to 9 of the trailer 6 can simultaneously capture images, which can be combined to create a three-dimensional panoramic image of the trailer, with the panoramic image showing the surroundings to the side of the trailer 6 and the surroundings to the rear of the trailer 6.

[0031] The three-dimensional panoramic image of the vehicle is created by the image processing unit 13 projecting four images simultaneously captured by the four cameras 2 to 5 of the vehicle 1 onto a movable three-dimensional rear projection surface in the shape of a dish or bowl. Similarly, the three-dimensional panoramic image of the trailer is created by the image processing unit 13 projecting four images simultaneously captured by the three cameras 7 to 9 of the trailer 6 onto a movable three-dimensional rear projection surface in the shape of a dish or bowl.

[0032] Not only the drawbar 11 of the trailer is located in images taken by the fourth camera 5, but the entire trailer 6. An image area of these images from the fourth camera 5, which contains the entire trailer 6, is referred to below as the "trailer area".

[0033] The trailer area forms a blind spot area, which shows an area behind the vehicle up to the trailer, but not objects obscured by the trailer 6 in the vehicle's all-round image, which are located behind the trailer 6 or on both sides next to the trailer 6. However, these obscured objects are detected by the three cameras 7 to 9 of the trailer 6 and displayed in an image area of the trailer's all-round image. The image processing unit 13 replaces the trailer area in the vehicle's all-round image with the corresponding image area of the trailer's all-round image, wherein the image area of the trailer's all-round image contains the image area obscured by the trailer 6 and the obscured objects located therein.

[0034] Based on the determined rotation angles α, β, and φ, the image processing unit 13 selects the corresponding image area of the trailer's panoramic image, which contains the image area obscured by the trailer and not detectable by the four cameras 2 to 5 of the vehicle 1. To this end, the image processing unit 13 transforms the coordinates of objects from the image area of the trailer's panoramic image (which are present in coordinates relative to the trailer coordinate system II) into corresponding coordinates of the vehicle coordinate system I.

[0035] Fig. 4 shows how the three-dimensional all-round vision system works and can be operated. From images 24 of the four cameras 2 to 5 of vehicle 1, as explained above in connection with Fig. 1 to 3described - the angle α and (optionally) also the angles β and β of the drawbar 11 of the trailer 6 relative to the vehicle 1 are calculated (illustrated by block 25). Furthermore, four images 24 of the four cameras 2 to 5 of the vehicle 1 can be combined with three images 26 of the three cameras 7 to 9 of the trailer 6 (illustrated by the "+" symbol), which is done by means of the image processing unit 13.

[0036] From the total of seven images (four images from the four cameras 2 to 5 of the vehicle 1 and three images from the three cameras 7 to 9 of the trailer 6) and the determined angle α (and optionally additionally the determined angle β and / or the determined angle φ), the image processing unit 13 renders the three-dimensional vehicle panoramic image as described above in connection with Fig. 1 to 3described (illustrated by block 27). The three-dimensional vehicle panoramic image is displayed on a monitor 14, which may be part of a human-machine interface 28, via which a desired virtual perspective (depending on which the rendering takes place) for the three-dimensional vehicle panoramic image can be selected.

Claims

1. Three-dimensional surround-view system for a combination (10) composed of a vehicle (1) and a trailer (6), wherein the trailer (6) is fixed to the vehicle (1) by means of a straight drawbar (11) in an articulated manner at least about a vertical axis (z) of the vehicle (1), the surround-view system comprising: - a plurality of cameras (2 to 5) of the vehicle (1), - a plurality of cameras (7 to 9) of the trailer (6), - an image processing unit (13) and - an optical output unit (14), wherein - the cameras (2 to 5) of the vehicle (1) are calibrated with respect to a coordinate system (I) of the vehicle (1), and are arranged on the vehicle (1) in such a way that a three-dimensional vehicle all-round image can be generated by stitching together one image each from all the cameras (2 to 5) and can be displayed on the optical output unit (14), wherein a drawbar region of the vehicle all-round image shows the drawbar (11), - the cameras (7 to 9) of the trailer (6) are calibrated with respect to a coordinate system (II) of the trailer (6), and are mounted on the trailer (6) in such a way that a three-dimensional trailer all-round image can be generated by stitching together one image each from all the cameras (7 to 9), wherein the trailer all-round image shows lateral surroundings of the trailer (6) and a surrounding area to the rear of the trailer (6), wherein the image processing unit (13) is configured - to determine from the drawbar region of the vehicle all-round image a first angle of rotation (α), by which the drawbar (11) is rotated relative to the vehicle (1) about the vertical axis (z) thereof relative to a forwards direction (x) of the vehicle (1), and - to replace a trailer region of the vehicle all-round image showing the trailer (6) with a corresponding image region of the trailer partial all-round image depending on the determined first angle of rotation (α) of the drawbar (11), characterized in that the image processing unit (13) is configured to project the images recorded by the cameras (2 to 5) of the vehicle (1) onto a movable, three-dimensional first back projection surface and to project the images recorded by the cameras (7 to 9) of the trailer (6) onto a movable, three-dimensional second back projection surface.

2. Three-dimensional surround-view system according to Claim 1, wherein the three-dimensional back projection surface has the shape of a dish.

3. Three-dimensional surround-view system according to either of the preceding Claims 1 and 2, wherein the image processing unit (13) is configured - to determine from the drawbar region of the vehicle all-round image a second angle of rotation (β), by which the drawbar (11) is rotated relative to the vehicle (1) about the width axis (y) thereof, and - to replace the trailer region of the vehicle all-round image showing the trailer (6) with a corresponding image region of the trailer partial all-round image depending on the determined first angle of rotation (α) and the second angle of rotation (β) of the drawbar (11).

4. Three-dimensional surround-view system according to any of the preceding claims, wherein the image processing unit (13) is configured - to determine from the drawbar region of the vehicle all-round image a third angle of rotation (φ), by which the drawbar (11) is rotated relative to the vehicle (1) about the longitudinal axis (x) thereof, and - to replace the trailer region of the vehicle all-round image showing the trailer (6) with a corresponding image region of the trailer partial all-round image depending on the determined first angle of rotation (α), second angle of rotation (β) and third angle of rotation (φ) of the drawbar (11).

5. Combination (10) composed of a vehicle (1) and a trailer (6), wherein the trailer (6) is fixed to the vehicle (1) by means of a straight drawbar (11) in an articulated manner about at least a vertical axis (z) of the vehicle (1), the combination (10) comprising a three-dimensional surround-view system according to any of the preceding claims.