Method for operating an image capturing device of a vehicle, control device, image capturing device, vehicle, and computer program

Abstract

The invention relates to a method for operating an image capturing device (2) of a vehicle (1), said image capturing device (2) comprising at least one camera (3) and at least one display device (9). The camera (3) detects image data from a sub-region of the surroundings of the vehicle, and a view (18) of the surroundings generated from the image data is displayed on the display device (9). The view (18) of the surroundings is generated from the image data using a projection of the image data, wherein a changeover between at least two different projection methods for projecting the image data is carried out on the basis of the speed of the vehicle (1), and each projection method uses a differently shaped projection surface (12, 13, 14) for the projection of the image data.

Description

[0001] Description

[0002] Method for operating an image recording device of a vehicle, control unit, image recording device, vehicle and computer program

[0003] The invention relates to a method for operating an image recording device of a vehicle, wherein the image recording device comprises at least one camera and at least one display device. The camera captures image data from a partial area of ​​the vehicle's surroundings, and a view of the surroundings generated from the image data is displayed on the display device. The view of the surroundings is generated from the image data by projecting the image data. Furthermore, the invention relates to a control unit, an image recording device, a vehicle, and a computer program.

[0004] In vehicles, image recording devices that capture part of the vehicle's surroundings can be used to provide driver assistance functions. For example, images of the vehicle's surroundings captured by one or more of the vehicle's cameras and / or representations synthesized from these images can be displayed on a display device for a driver of the vehicle. By providing such a view of the surroundings, the driver's perception of the surroundings and thus also the maneuverability of the vehicle can be improved. In particular, areas that are not directly visible from inside the vehicle can be made visible to the driver. This makes maneuvering in tight spaces and / or parking the vehicle easier, for example.

[0005] For example, to correct distortion in image data generated by wide-angle cameras and / or to display the surrounding view from a virtual viewpoint that differs from the actual viewpoint of the camera, it is known to project the image data. This involves projecting the image data from the plane of an image sensor onto a virtual projection surface. The image presented to the driver can then be derived from the projection surface based on the virtual camera position.

[0006] US 2019 / 0001887 A1 describes a method for generating a view of an area in front of a vehicle. Images captured by a camera of the vehicle are projected parallel onto a virtual, flat projection surface, with the projection surface being parallel to the plane of an image sensor of the camera.

[0007] It is known from the prior art to adapt the field of view of the surrounding cameras of a vehicle depending on the current operating state of the vehicle so that the driver of the vehicle can be given an improved perception of the surroundings.

[0008] For example, US 2015 / 0350607 A1 describes a device for displaying a vehicle's surroundings, which comprises multiple cameras. The surrounding areas captured by the cameras can be changed depending on the vehicle speed by mechanically tilting and / or rotating the cameras. Tilting or rotating the cameras is achieved by tilting or rotating an image sensor of the camera and / or a lens of the camera optics, and causes a change in the camera's detection range relative to the vehicle's surroundings.

[0009] The invention is based on the object of specifying a method which is improved compared to the method and which in particular enables an improved adaptability of the environmental display to a current ferry operation.

[0010] To achieve this object, the invention provides for a method of the type mentioned at the outset to switch between at least two different projection methods for projecting the image data depending on the speed of the vehicle, wherein the different projection methods each use a differently shaped projection surface for projecting the image data. By means of the vehicle's image recording device, camera images of the surroundings of the vehicle are recorded, in particular continuously in the form of an image data stream or a video stream. For this purpose, the image recording device comprises at least one camera which is arranged on the vehicle and, depending on its position on the vehicle, can capture a partial area of ​​the vehicle's surroundings, for example in front of, behind or next to the vehicle.

[0011] A view of the surroundings is generated from the image data captured by one or more cameras, which is presented to the driver on the display device of the image recording device. The display device can be arranged, in particular, in an interior of the vehicle so that it is visible to the driver and can be designed, for example, as a screen, a touch display, or the like.

[0012] The environmental view is generated from the image data by projecting the image data onto a projection surface. The image data, which was recorded by an image sensor of the at least one camera, is transferred to the projection surface using an assignment rule. The environmental view can then be derived from the image data transferred to the projection surface. In particular, the environmental view can be generated starting from a virtual camera position, which can either correspond to the actual camera position or can differ from the actual camera position. When using a virtual camera position that is different from the actual camera position, the environmental view can be displayed from a perspective that differs from the perspective from which the image data was recorded by the camera.

[0013] The projection surface used for the projection is a virtual construct to which the image data is transferred, for example, using a mapping rule. The resulting ambient view depends, among other things, on the geometry of the projection surface. Depending on the shape of the projection surface, ambient views with different image properties result.

[0014] According to the invention, depending on the vehicle's speed, the system switches between two different projection methods, which differ in the geometry of the projection surface used. This advantageously enables a suitable view of the surroundings to be generated and presented to the driver depending on the vehicle speed, and thus also for different driving situations. This advantageously facilitates maneuvering the vehicle, as a favorable view of the surroundings can be provided for the respective speeds or driving situations. Advantageously, mechanical actuators for changing the camera's field of view can be dispensed with.

[0015] Different projection methods that can be used include, for example, a planar projection, which uses a planar projection surface, and / or a cylindrical projection, which uses a cylindrically curved projection surface, or an elliptical projection, which uses an elliptically curved projection surface. In this context, a cylindrically curved projection surface means that the projection surface in virtual space has, in particular, two edges that are in the shape of a circular arc segment. Preferably, the projection surface comprises two further edges that are non-curved or straight. This results in a projection surface that corresponds to a segment of a cylinder surface, with the projection taking place, in particular, onto the concave side or onto the inside of the cylinder surface segment.

[0016] Accordingly, in this context, an elliptically curved projection surface is understood to mean a projection surface that, in virtual space, has two curved edges corresponding to an ellipse segment. Here, too, the projection surface can have two additional straight edges, and the projection can occur, in particular, onto the concave side of the projection surface.

[0017] The change between the projection methods or between the different geometries of the projection surface can, in particular, occur automatically, i.e., without prior user input or the like. This can be achieved, for example, by a control device configured to implement the method determining and using a projection surface for generating the surrounding view based on a measured speed value of the vehicle.

[0018] By changing the geometry of the projection surface depending on the speed, it is possible to adapt the image information displayed in the surroundings view to the current vehicle state. For example, the proportion of the vehicle's surroundings displayed in the surroundings view, or the field of view (FOV), can be adjusted. Additionally or alternatively, the selection of the geometry of the projection surface can also influence the distortion of the surroundings view, i.e., the image data can be displayed with less distortion and / or with different magnifications in sub-areas of the displayed vehicle's surroundings.

[0019] According to the invention, a projection method can be used to image a larger horizontal field of view of the camera at lower speeds than at higher speeds. This can be achieved, for example, by using a cylindrical projection or an elliptical projection at lower speeds.

[0020] This allows the largest possible portion of the horizontal field of view to be captured at low speeds, even if this field of view exhibits greater distortion compared to a view of the surroundings generated, for example, using a planar projection. At higher speeds, a smaller horizontal field of view is generally sufficient, since the vehicle is generally moving essentially in a straight line at higher speeds and is generally no longer maneuvering at higher speeds.

[0021] In a preferred embodiment of the invention, a projection method with less distortion of the image data can be used at higher speeds than at lower speeds. This can be achieved, for example, by using a planar projection at higher speeds. This advantageously allows for a less-distorted or distortion-free image display at higher speeds.

[0022] To trigger the switch between the projection methods, the invention can provide for a first projection method to be used at a speed below a threshold value, and a second projection method to be used at a speed above the threshold value or above a further threshold value. Below the threshold value for the speed, the first projection method can thus be used to generate a desired environmental representation for lower speeds. Above the threshold value, or above a further threshold value different from the threshold value, a switch can be made to a second projection method to achieve a representation preferred for higher speeds.

[0023] According to the invention, it can be provided that when the vehicle speed exceeds or falls below the limit value or at speeds between the limit value and the further limit value, a, in particular continuous, transition between the projection methods is used to generate the surroundings view. The continuous transition can be used both when the vehicle accelerates, i.e. when the vehicle speed increases, and when the vehicle decelerates or decreases in speed. The continuous transition between the projection methods can be generated, for example, by calculating and displaying intermediate images of the surroundings view between a surroundings view generated by the first projection method and a further surroundings view generated by the second projection method.This allows for a smooth transition between the projection methods. The smooth transition can be triggered by exceeding or falling below the threshold value, depending on whether the vehicle is accelerating or decelerating. Using an additional threshold value, the smooth transition can be displayed particularly in the speed range between the threshold value and the additional threshold value.

[0024] As an alternative to generating intermediate images, the geometry of the projection surface can be adjusted to create a smooth transition from the geometry of the projection surface of the first projection method to the geometry of the projection surface of the second projection method. This ensures that the surrounding view generated using the projection surface also exhibits a continuous progression over time, allowing the driver to view the surrounding view with a continuous change or morph between the two different display modes. Here, too, the display of the continuous transition can be triggered when the limit value is exceeded or undershot, or displayed in the speed range between the limit value and the further limit value.

[0025] According to the invention, it can be provided that a different limit value and / or a different further limit value is used when the vehicle is accelerating than when the vehicle is decelerating. In this way, the behavior of the projection of the image data or the image properties of the environmental representation can be further adapted to different driving situations. In a preferred embodiment of the invention, it can be provided that a cylindrical projection with a cylindrically curved projection surface or an elliptical projection with an elliptically curved projection surface is used as the or a first projection method, and a planar projection with a planar projection surface is used as the or a second projection method.

[0026] According to the invention, it can be provided that a front camera, a side camera, and / or a rearview camera of the vehicle is used as at least one camera. The surrounding view can be generated from the image data of a single one of the cameras, in particular a front camera or a rearview camera. However, it is also possible for the image data from multiple cameras to be used, for example, by combining the image data from cameras arranged adjacent to the vehicle, or from cameras with at least partially overlapping detection areas, in order to be able to cover a larger portion of the vehicle's surroundings.

[0027] The cameras are preferably designed as cameras with a wide-angle lens, for example a fisheye lens or the like, and have a horizontal field of view of at least 180°. Alternatively, it is possible to use multiple cameras with a smaller field of view at the front of the vehicle, the sides of the vehicle, and / or the rear of the vehicle, in order to capture, in particular, the entire vehicle surroundings.

[0028] For a control device according to the invention, it is provided that it is configured to carry out a method according to the invention when image data from at least one camera is provided.

[0029] An image recording device according to the invention for a vehicle is provided for comprising at least one camera, at least one display device, and a control unit according to the invention. A vehicle according to the invention is provided for comprising an image recording device according to the invention.

[0030] A computer program according to the invention is provided to include instructions that configure a control unit to execute a method according to the invention based on image data provided by a camera. The computer program can be stored on a data storage medium, in particular a non-transient one. It is also possible for the computer program to be accessible from a data storage device, for example, a server, via a communications connection, such as the Internet.

[0031] The advantages and embodiments described above in relation to the method according to the invention apply accordingly to the control unit according to the invention, the image recording device according to the invention, the vehicle according to the invention and the computer program according to the invention.

[0032] The advantages and details described for the control device according to the invention, the image recording device according to the invention, the vehicle according to the invention and the computer program according to the invention can also be transferred analogously to the respective other invention objects.

[0033] Further advantages and details of the invention will become apparent from the exemplary embodiments described below and from the drawings. These are schematic representations and show:

[0034] Fig. 1 : an embodiment of a vehicle according to the invention,

[0035] Fig. 2: a block diagram of an embodiment of a method according to the invention,

[0036] Fig. 3: several examples of differently shaped projection surfaces and their arrangement in the virtual space, Fig. 4: an environmental view when using a first

[0037] projection method and

[0038] Fig. 5: the surrounding view when using a second

[0039] projection process.

[0040] Fig. 1 shows an embodiment of a vehicle 1. The vehicle 1 can be, for example, a motor vehicle, in particular a passenger car, a truck, or another type of commercial vehicle. It is also possible for the vehicle 1 to be a non-motorized vehicle such as a trailer, or for it to be a combination consisting of a towing vehicle and a trailer. The vehicle 1 can also be a rail-bound vehicle, such as a tram or the like, or a robot.

[0041] The vehicle 1 comprises an exemplary embodiment of an image recording device 2, which comprises one or more cameras 3 and a control unit 4. In the present exemplary embodiment, the vehicle 1 comprises four surroundings cameras 3, which form a camera arrangement designed as a surround-view system. A first camera 5 is arranged as a front camera of the vehicle 1, a second camera 6 as a rear-view camera of the vehicle 1, and a third camera 7 and a fourth camera 8 are each arranged as a side camera of the vehicle 1. By means of the cameras 3, image data can be generated from a partial area of ​​the surroundings of the vehicle 1.

[0042] The cameras 3 are each designed as a wide-angle camera, for example, as cameras with a fisheye lens, and preferably have a horizontal detection range of 180° or more. The control unit 4 of the image recording device 2 can be designed, for example, as a microcontroller, a processor, or another type of computing device.

[0043] The vehicle 1 further comprises a display device 9, via which graphic information can be presented to a driver or a user of the vehicle 1. The display device 9 can, for example, be one or more screens or displays arranged in an interior of the vehicle 1.

[0044] The cameras 3 are connected to the control unit 4 via a communication connection 10. The communication connection 10 can, for example, comprise a plurality of point-to-point connections or be a bus connection such as a CAN bus or the like. Camera images or video streams are transmitted via the communication connection 10, in particular continuously as a video stream.

[0045] Image data from the cameras 3 is transmitted to the control unit 4. At least a portion of the camera images and / or an environmental view generated from the image data of at least one of the cameras 3 can then be displayed to a driver of the vehicle 1 on the display device 9.

[0046] Furthermore, the vehicle 1 comprises at least one speed sensor 11, by means of which a measured value describing a speed of the vehicle 1 can be continuously determined. The continuously determined measured values ​​can be transmitted to the control unit 4, in particular, via the communication connection 10. The speed sensor 11 can be designed, for example, as a wheel speed sensor or as any other type of speed sensor.

[0047] The control unit 4 is designed to carry out an embodiment of a method for operating the image recording device 2. Fig. 2 shows a block diagram of an embodiment of the method.

[0048] In a first step S1 of the method, image data is recorded with at least one of the cameras 3 of the vehicle 1. The camera 3 can, for example, be the camera 6 arranged as a rear-view camera. The image data is transmitted from the camera 6 to the control unit 4 via the communication connection 10. The image data comprise, for example, a continuous series of camera images which reproduce a partial area of ​​the vehicle's surroundings, more precisely the partial area behind the vehicle 1. In step S2 of the method, the control unit 4 generates a view of the surroundings from the image data. For this purpose, the image data is projected onto a virtual projection surface. The projection is therefore a mathematical operation and not a projection in the physical sense. During the projection, the image data orthe individual pixels of at least part of the image data generated by the camera are projected onto the projection surface by means of an assignment rule, as described in more detail below with reference to Figures 3 to 5.

[0049] Finally, in step S3 of the method, the generated view of the surroundings is displayed on the display device 9 of the vehicle 1. The view of the surroundings can, in particular, be updated several times per second, resulting in at least essentially a live reproduction of the surroundings of the vehicle 1 in the form of a video. For this purpose, the control unit 4 can continuously calculate the surroundings representation or an image of the surroundings representation from the camera images, or at least part of the camera images, of the cameras 3 as image data and transmit it to the display device 9.

[0050] Fig. 3 schematically shows various projection surfaces 12, 13, 14. The projection surfaces 12, 13, 14 differ in terms of their geometry. In a virtual space defined, for example, by a vehicle-fixed coordinate system 15, the projection surfaces 12, 13, 14 are arranged relative to a virtual vehicle model 16. Using the projection surfaces 12, 13, 14, the environmental representation can be generated starting from a virtual camera position 17, which may correspond to the actual position of the camera 6 in the coordinate system 15 or may differ from this position.

[0051] The projection surface 12 is a planar projection surface that has no curved edges. In other words, both the two edges extending in the y-direction corresponding to the vehicle width direction and the two edges extending in the z-direction corresponding to the vehicle height direction are straight.

[0052] The projection surface 13 is a cylindrically curved projection surface with two circular arc-shaped edges in the xy-plane. The edges extending in the z-direction are also straight, so that the projection surface 13 has the shape of a cylindrical shell segment.

[0053] The projection surface 14 is an elliptical projection surface, which has two elliptically curved edges in the xy plane, i.e., two edges with the shape of an elliptical arc segment. Here, too, the edges are straight in the z-direction, resulting in a surface curved in the x-direction and y-direction. The image data from camera 6, for example, is projected onto the concavely curved side of the respective projection surface 12, 13, 14 facing the virtual vehicle model 16.

[0054] Depending on the speed of the vehicle 1, determined, for example, by the speed sensor 11, the control unit 4 switches between at least two different projection methods for projecting the image data when creating the surrounding view. The projection methods differ in the use of differently shaped projection surfaces 12, 13, 14, as described above with reference to Fig. 3, for example.

[0055] At low speeds, a projection method is used to image a larger horizontal field of view of the camera 3 than at higher speeds. This projection method can be a cylindrical projection, using, for example, the cylindrically curved projection surface 13. Alternatively, an elliptical projection, using, for example, the elliptically curved projection surface 14, can also be used. At higher speeds, a projection method with less distortion than at low speeds is used. Preferably, a planar projection, using the planar projection surface 12, is used at higher speeds.

[0056] Fig. 4 shows an exemplary environmental view 18 that uses a cylindrical projection as the projection method for displaying the image data. For comparison, Fig. 5 shows the environmental view 18 using a planar projection. It is clear that by using the cylindrical projection, the field of view displayed in the environmental view 18 is larger than when using the planar projection.

[0057] In contrast, the environmental view 18 has less distortion when using the planar projection than when using the cylindrical projection.

[0058] Preferably, at a speed of vehicle 1 below a threshold value, the cylindrical or elliptical projection can be used as a first projection method. At a speed above the threshold value, the planar projection can be used as a second projection method. Alternatively, the second projection method can only be used at a speed above a further threshold value that differs from the threshold value.

[0059] The limit value or the additional limit value is a speed limit, which can be, for example, 7 km / h, 30 km / h, 60 km / h, 90 km / h, 120 km / h, 130 km / h, or the like. When using an additional limit value, this is particularly higher than the limit value. It is also possible that a speed of 0 km / h is used as the limit value.

[0060] Using cylindrical or elliptical projection for low speeds provides an enlarged field of view, particularly when maneuvering the vehicle in parking lots or similar locations, making maneuvering easier, especially in tight spaces. At higher speeds, a narrower viewing angle is sufficient, for example, to display the area in front of or behind the vehicle. Using planar projection reduces distortion, enabling a more realistic representation of the vehicle's surroundings and thus facilitating rapid perception of the vehicle's surroundings by the driver, especially at higher speeds.

[0061] In order to avoid sudden changes in the representation of the surrounding view 18 on the display device 9, when the vehicle speed exceeds or falls below the limit value or at speeds between the limit value and the further limit value, a particularly continuous transition between the projection methods for generating the surrounding view 18 is preferably used.

[0062] The continuous transition between the two projection methods can be generated, for example, by calculating and displaying intermediate images of the surrounding view 18 between the first projection method and the second projection method. In other words, the intermediate images can correspond to a projection using a projection surface that is an intermediate form between the projection surface of the first projection method and the projection surface of the second projection method.

[0063] As the speed of vehicle 1 increases, a transition from the cylindrical projection or an elliptical projection to the planar projection can occur, for example, upon reaching the limit value for the vehicle speed. For this purpose, after the limit value is exceeded or at speeds that lie between the limit value and the further limit value, intermediate images can be calculated by control unit 4, which represent the transition between the cylindrical projection and the planar projection, in particular continuously. In this case, for example, the effect of a successively changing curvature of the projection surface can be subsequently added to the surrounding view 18 using a calculation rule for images generated using the first or second projection method.On the display device 9, the user of the vehicle 1 then sees a changing view of the surroundings, in which the type of representation changes continuously or in which morphing takes place between the different projection methods.

[0064] As an alternative to generating intermediate images, it is possible to adjust the geometry of the projection surface 12, 13, 14 so that, for example, as the vehicle speed increases, there is a smooth transition from the geometry of the cylindrical or elliptical projection surface 13, 14 of the first projection method to the planar geometry of the projection surface 12 of the second projection method. Conversely, as the vehicle speed decreases, there can be a transition from the projection surface 12 of the second projection method to the projection surface 13, 14 of the first projection method. Each change in the geometry of the projection surface 12, 13, 14 also changes the resulting representation of the surroundings, so that the effects caused by the change in the projection method, such as the larger field of view, continuously increase.

[0065] The limit value and / or the further limit value can have the same values ​​or different values ​​for the acceleration and for the braking of the vehicle 1, or for increasing speeds and decreasing speeds of the vehicle 1.

[0066] In addition to using the image data from camera 6 arranged as a rear-view camera, it is also possible to use the image data from several of the cameras 3. In particular, the image data from two or more adjacently arranged cameras 3, or from cameras 3 with at least partially overlapping detection areas, can be combined and used to generate a representation of the surroundings.

Claims

Patent claims 1. A method for operating an image recording device (2) of a vehicle (1), wherein the image recording device (2) comprises at least one camera (3) and at least one display device (9), wherein the camera (3) captures image data from a partial area of ​​a vehicle's surroundings and an environmental view (18) generated from the image data is displayed on the display device (9), wherein the environmental view (18) is generated from the image data by means of a projection of the image data, characterized in that, depending on a speed of the vehicle (1), a change is made between at least two different projection methods for projecting the image data, wherein the different projection methods each use a differently shaped projection surface (12, 13, 14) for projecting the image data.

2. Method according to claim 1, characterized in that at lower speeds a projection method is used to image a larger horizontal field of view of the camera (3) than at higher speeds.

3. Method according to claim 1 or 2, characterized in that at higher speeds a projection method with a lower distortion of the image data is used than at lower speeds.

4. Method according to one of the preceding claims, characterized in that that a first projection method is used for a speed below a limit value and a second projection method is used for a speed above the limit value or above a further limit value.

5. Method according to claim 4, characterized in that when the vehicle speed exceeds or falls below the limit value or at speeds between the limit value and the further limit value, a, in particular continuous, transition between the projection methods is used to generate the surrounding view.

6. Method according to claim 4 or 5, characterized in that when the vehicle accelerates, a different limit value and / or a different further limit value is used than when the vehicle decelerates.

7. Method according to one of the preceding claims, characterized in that a cylindrical projection with a cylindrically curved projection surface (13) or an elliptical projection with an elliptically curved projection surface (14) is used as the or a first projection method and a planar projection with a planar projection surface (12) is used as the or a second projection method.

8. Method according to one of the preceding claims, characterized in that a front camera, a side camera and / or a rear view camera of the vehicle (1) is used as at least one camera (3).

9. Control unit which is configured to carry out a method according to one of the preceding claims upon provision of image data from at least one camera (3).

10. Image recording device for a vehicle, comprising at least one Camera (3), at least one display device (9) and a control device (4) according to claim 9.

11. A vehicle comprising an image recording device (2) according to claim 10.

12. Computer program comprising instructions which set up a control device (4) to carry out a method according to one of claims 1 to 8 on the basis of image data provided by a camera (3).

Images