A method for operating a vehicle, and a vehicle

Abstract

The invention relates to a method for operating a vehicle 10 which comprises a camera 12, a display 14 and a computing unit 16. Comprising the steps of: Determining a degree of glare of an environment

Description

FIELD OF THE INVENTION

[0001] The present invention relates to the field of vehicles, in particular automobiles, such as vans, cars, or trucks. More specifically, the present invention relates to a method for operating a vehicle, which comprises a camera, in particular a front-facing camera, a display, in particular, a display in a center console and / or a dashboard, and a computing unit, such as an ECU. Furthermore, the present invention relates to a corresponding vehicle comprising a camera, a computing device and a display. BACKGROUND INFORMATION

[0002] For a user, in particular a driver of a vehicle, blinding glare, for example from low standing sunlight, may be a safety risk while driving. Therefore, even when the drive wears sunglasses, it could be sometimes hard for him to see the environment or surroundings of the vehicle. Protecting the driver from glare should be taken into account. SUMMARY OF THE INVENTION

[0003] It is an object of the present invention to provide a method for operating a vehicle and a corresponding vehicle, in which protection against glare to a driver or other user of the vehicle may be performed very efficiently.

[0004] This object is solved by a method for operating a vehicle, and a vehicle according to the present invention. Advantageous embodiments are presented in the dependent claims, the description and the drawings.

[0005] A first aspect of the present invention relates to a method for operating a vehicle, which comprises a camera, a display and a computing unit. The camera is in particular a front-facing camera and / or flanked fisheye cameras. The camera could therefore be a camera with one lens. The camera could consist of a camera unit with more than one lens, or several cameras could be used whose fields of view overlap at least partially. In particular, the camera is able to record a video of the field of view of a driver of the vehicle. The video is a stream of images.

[0006] The display is, in particular, a screen, such as a liquid crystal display or an OLED display and is placed in the center console or in the dashboard, for example. An alternative could be that the display is built as a head-up display. The computing unit is, in particular, to be understood as a data processing device, which comprises processing circuitry. The computing unit may process data to perform computing operations. This may also include operations to perform index accesses to data structure, for example, a lookup table. In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application specific integrated circuits, ASICs, one or more field programmable gate arrays, FPGA, and / or one or more system on a chip, SOC. In particular, the computing unit could be built as an electronic control unit, ECU, which is built as a control unit for the vehicle that regulates, controls, and monitors the engine, for example.

[0007] The method according to the invention comprises the following steps:

[0008] A first step is determining a degree of glare, for example, the position of the sun, of an environment of the vehicle for a user, in particular the driver, of the vehicle.

[0009] A second step is recording a video of the vehicle's environment by the camera.

[0010] A third step is processing of the video by the computing device, by using a glare mitigation algorithm GMA, whereby recognized glare elements in the video are replaced.

[0011] A fourth step is providing the processed video on or via the display.

[0012] In other words, a method is provided, where the camera, in particular a frontfacing camera, mounted on the vehicle, in particular a car, to capture a live video. Then a computer vision algorithm is applied to the images of the wheel stream to mitigate the glare effect, in particular of the low standing sun or, for example, of glaring lights of other traffic participants. The post-processed video is then shown on a display, for example, on an in-cabin screen.

[0013] An advantage of the invention is that the user or driver of the vehicle may use the displayed video stream to navigate the vehicle while outside light conditions would be too glary to navigate the vehicle otherwise. Another advantage is that the procedure can be performed without special camera filters and therefore without special hardware.

[0014] In an embodiment of the present invention, the degree of glare in the video or in the images of the video stream is determined by the computing unit by using a glare detection algorithm GDA.

[0015] In still another embodiment of the present invention, the glare mitigation algorithm comprises computer vision and / or a machine learning method.

[0016] In still another embodiment of the present invention, the machine learning method of the glare mitigation algorithm, in particular, a deep neural network model of the machine-learning method, is trained that it predicts and / or generates images without glare, given an image with glare as input.

[0017] In yet another embodiment of the present invention, the glare detection algorithm comprises computer vision and / or a machine learning method.

[0018] In still another embodiment of the present invention, the degree of glare is determined by detecting the brightness of pixels of the images of the video stream on the basis of predefined threshold values, with spatial information in particular also being considered, so that only pixels with a brightness above the threshold value, which also follow a specific spatial distribution, are taken into account.

[0019] In a further embodiment of the present invention, the machine learning method, in particular a deep neural network model, of the glare detection algorithm is trained to predict for each pixel of the image of the video stream or the wheel, whether it is glared.

[0020] In still another embodiment of the present invention, during processing, the contrast of a low-contrast overexposure area is increased and / or a gamma correction and / or a white balance is carried out and / or, for corresponding pixels, the brightness of the pixels is reduced in proportion to their brightness.

[0021] A second aspect of the present invention relates to a vehicle with a camera, a computing device, and a display, which is designed to carry out a method according to the first aspect of the present invention.

[0022] Advantages and advantageous embodiments of the first aspect of the invention are to be regarded as advantages and advantageous embodiments of the second aspect of the invention and vice versa.

[0023] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figure and / or shown in the figure alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figure.

[0025] The drawings show in:

[0026] Fig. 1 a schematic view of a diagram for a method for operating a vehicle in a vehicle cabin. DETAILED DESCRIPTION

[0027] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0028] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0029] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0030] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0031] Fig. 1 shows a schematic view of a vehicle's 10 interior with a diagram, which shows a method for operating the vehicle 10. The vehicle 10 comprises a camera 12, a display 14 and a computing unit 16.

[0032] The method comprises the following steps:

[0033] In a first step, determining a degree of glare of an environment 18 for a user of the vehicle 10;

[0034] In a second step, recording a video of the vehicle's environment 18 by the camera 12, which in particular shows the field of view 24 of the user, in particular the driver.

[0035] Afterwards, in a third step, the video is processed by the computing device 16, by using a glare mitigation algorithm 20 whereby recognized glare elements in the video or images of the video stream, which are replaced.

[0036] In the next and fourth step, the processed video is provided on the display(s) 40 for usage by the driver.

[0037] The degree of glare in the video may be determined by the computing unit 16 by using a glare detection algorithm 22.

[0038] The glare mitigation algorithm 20 as well as the glare detection algorithm 22 may comprise a computer vision and / or machine learning method each.

[0039] The camera 12 is, in particular, a front-facing camera combined with flanked fisheye cameras, so the cameras are used to capture the image and video as the input of the algorithms individually or jointly.

[0040] The glare detection and mitigation should both be performed in the in-vehicle computing unit, for example, an ECU or a GPU or TPU chip. The output of the glare mitigated image or video is displayed on an in-vehicle car screen - the displays 40. The glare detection algorithm 22 aims to detect if glare happens and output the glared regions and / or pixels, if possible. This may be done in both classical computer vision-based methods as well as machine learning-based methods.

[0041] Classical computer vision methods operate on values of the pixels directly: For example, detecting the brightness of the pixel based on predefined thresholds. Then may further integrate spatial information so that only pixels with the above threshold brightness that also follows certain spatial distributions shall be considered.

[0042] The machine learning or deep learning-based methods are used, wherein a deep neural network model may be trained so that it predicts for each pixel, whether or not it is glared.

[0043] The glare mitigation algorithm 20 performs the glare mitigation after detection via the glare detection algorithm 22. It may also be done in both, classical computer visionbased methods as well as machine learning-based methods. The classical computer vision methods operate on values of the pixels directly, for example, reducing the brightness of the pixel’s proportion to their brightness, and hence the contrast of low contrast glare regions, and / or gamma correction, white balancing etc. For machine learning- or deep learning-based methods, a deep neural network model may be trained that predicts and / or generates images without glare, giving an image with glare as input.

[0044] The method as well as the vehicle 10 provide a vision assistance system for glaring from low-standing sun Reference signs 10 vehicle 12 camera 14 display 16 computing unit 18 environment 20 glare mitigation algorithm 22 glare detection algorithm 24 field of view

Claims

1. A method for operating a vehicle (10) which comprises a camera (12), a display (14) and a computing unit (16), comprising the steps of:Determining a degree of glare of an environment (24) for a user of the vehicle (10);Recording a video of the vehicle's environment (24) by the camera (12); Processing of the video by the computing device (16), by using a Glare Mitigation Algorithm (20) whereby recognized glare elements in the video are replaced; andProviding the processed video on the display (14) for usage by the user.

2. The method according to claim 1, characterized in thatthe degree of glare in the video is determined by the computing unit (16) by using a glare detection algorithm (22).

3. The method according to claim 1 or 2, characterized in thatthe glare mitigation algorithm (20) comprises computer vision and / or a machine learning method.

4. The method according to any one of claims 1 to 3, characterized in thatthe machine learning method of the glare mitigation algorithm (20) is trained to predict and / or generate image without glare, given an image with glare as input.

5. The method according to any one of the preceding claims, characterized in thatthe glare detection algorithm (22) comprises computer vision and / or a machine learning method.

6. The method according to any one of the preceding claims, characterized in thatthe degree of glare is determined by detecting the brightness of pixels of images of the video on the basis of predefined threshold values, with spatial information in particular also being considered, so that only pixels with a brightness above the threshold value, which also follow a specific spatial distribution, are taken into account.

7. The method according to any one of the preceding claims, characterized in thatthe machine learning method of the glare detection algorithm (22) is trained to predict for each pixel whether it is glared.

8. The method according to any one of the preceding claims, characterized in thatduring processing, the contrast of a low-contrast overexposure area is increased and / or a gamma correction and / or a white balance is carried out and / or, for corresponding pixels, the brightness of the pixels is reduced in proportion to their brightness.

9. A vehicle (10) with a camera (12), a display (14), and a computing device (16), which is designed to carry out a method according to one of the preceding claims.

Images