Method for indicating a traffic area behind a vehicle, indicator device and vehicle

Abstract

The invention relates to a method for displaying a traffic area behind a vehicle (100) to a driver of the vehicle when the driver is seated in a driver's seat of the vehicle, comprising the method steps: A) Recording at least one traffic space image in a traffic space behind a vehicle (100) by at least one traffic space camera (2), B) Recording at least one image of the head (K) of the driver of the vehicle (100) by at least one head camera (3), C) Analysis of the image of the driver's head (K) taken in procedure step B), D) Analysis of the traffic space image recorded in process step A) and performance of object recognition in the recorded image, whereby recognized objects (O) are classified, E) Displaying an image to inform the driver about the traffic area behind the vehicle on a display (4). The invention further relates to a display device (10) and a vehicle (100) with a display device.

Description

[0001] The invention relates to a method for displaying a traffic area behind a vehicle to a driver of the vehicle when the driver is seated in a driver's seat of the vehicle, comprising the method steps: A) Recording at least one traffic space image in a traffic space behind a vehicle by at least one traffic space camera, B) Recording at least one image of the driver's head using at least one head-mounted camera, C) Analysis of the image of the driver's head taken in procedure step B), D) Analysis of the traffic space image recorded in process step A) and performance of object recognition in the recorded image, whereby recognized objects are classified, E) Displaying an image to inform the driver about the traffic area behind the vehicle on a display. The invention further relates to a display device and a vehicle with a display device.

[0002] To see the area behind a vehicle, traditional side mirrors are used. To avoid the aerodynamic disadvantages of such mirrors, virtual side mirrors exist, which transmit a camera image from a camera pointed at the area behind the vehicle to a display inside the vehicle. The design of virtual side mirrors attempts to replicate the properties of classic analog mirrors through technological innovations.

[0003] DE 10 2008 019 732 A1 describes a method and a device for controlling an electronic exterior mirror. In this method, the driver's head position in the vehicle is determined, and the displayed section of the image of the rear traffic area is changed on a display element depending on the head position. In this way, the behavior of an analog mirror is replicated.

[0004] FR 3 061 978 A1 describes a device for assisting in the observation of the area behind a vehicle. The device detects the position of the driver's head and adjusts the display of an electronic exterior mirror to the driver's head position.

[0005] DE10 2012 002 149 B3 describes a method for visualizing the surroundings of a motor vehicle and a corresponding motor vehicle. In the method, the surroundings of a motor vehicle are recorded with a camera and the camera image is displayed on a screen. Other road users are detected in the display and highlighted with a colored outline.

[0006] German patent application DE 10 2006 047 777 A1 describes a virtual spotlight for marking objects of interest in image data. A method is proposed in which image data from the surroundings of a vehicle is captured using a camera and displayed on a screen. Object recognition is performed in the image data before it is displayed. Different types of objects are then graphically manipulated and displayed in different ways.

[0007] DE 10 2017 209 908 A1 and DE 10 2013 020 950 A1 disclose a method for indicating a traffic area behind a vehicle according to the preamble of claim 1.

[0008] DE 10 2018 100 194 B3 discloses a mirror replacement system and a method for displaying image and / or video data of the surroundings of a motor vehicle.

[0009] DE 10 2012 200 762 A1 discloses a method and a driver assistance system for signaling a traffic situation in the vicinity of a vehicle.

[0010] DE 10 2021 005 680 A1 relates to a method for assisting a driver of a vehicle in maneuvering the vehicle, wherein the vehicle has environmental sensors with at least one mirror camera.

[0011] The object of the invention is to propose solutions that improve the display of a traffic area behind a vehicle, whereby this display is intended to improve the safety of the driver of the vehicle when dealing with dangerous situations in road traffic.

[0012] This problem of the invention is solved by a method for displaying a traffic area behind a vehicle to a driver of the vehicle when the driver is sitting in a driver's seat of the vehicle, comprising the method steps: A) Recording at least one traffic space image in a traffic space behind a vehicle by at least one traffic space camera, B) Recording at least one image of the driver's head using at least one head-mounted camera, C) Analysis of the image of the driver's head taken in step B) and determination of the driver's gaze direction and / or the position of the driver's eyes, D) Analysis of the traffic space image recorded in process step A) and performance of object recognition in the recorded image, whereby recognized objects are classified, E) Displaying an image to inform the driver about the traffic area behind the vehicle on a display, wherein the image is changed based on the traffic area image depending on the direction of the driver's gaze determined in process step C) and / or the position of the driver's eyes as well as depending on the objects detected and classified in process step D).

[0013] The method according to the invention serves to display the traffic area behind a vehicle to the driver of the vehicle when the driver is seated in the driver's seat and can, for example, be used to display the traffic area on a virtual or electronic mirror. Method steps A) to E) are preferably carried out continuously in the specified order. However, method steps A) and B) as well as C) and D) can also be carried out simultaneously.

[0014] In a first process step A), at least one traffic space image is captured in the traffic area behind the vehicle. Preferably, a large number of traffic space images are captured, for example in the form of a video. For this purpose, a traffic space camera is used, which is mounted, for example, on the side or at the rear of the vehicle.

[0015] In a second process (step B), a head-mounted camera aimed at the driver of the vehicle captures at least one image of that head. Here, too, a large number of images are preferably captured. This head-mounted camera is positioned so that the driver's eyes are visible. It is also possible to use multiple head-mounted cameras to ensure that the driver does not move their head out of the camera's field of view.

[0016] In a third step (C), the previously captured image of the driver's head is analyzed. The driver's eyes are detected, and their gaze direction is determined. Gaze direction is understood as a direction in space, which can be represented, for example, by a vector. Alternatively or additionally, the position of the driver's eyes can be determined. The gaze direction is directed away from the position of the eyes toward the target the driver is looking at.

[0017] In a fourth process step D), the previously recorded traffic image is analyzed. Object recognition is performed to identify, for example, other road users, obstacles, or other objects relevant to driving safety. The identified objects are then classified. Classification refers to the categorization of objects into different classes. In particular, a classification is made according to the potential hazard these objects pose to driving safety.

[0018] In a fifth process step E), a display image is shown on a display. This display is positioned so that the driver can see it unobstructed while driving. Such a display can, for example, be located on or in a door panel or on or in the center console of the vehicle. The display image is based on the previously acquired traffic space image. However, according to the invention, the traffic space image is modified for display purposes. This modification is made depending on the driver's viewing direction and / or the position of the driver's eyes. For example, the traffic space image can be enlarged in the display image if the driver moves their head closer to the display. Furthermore, the traffic space image can be rotated or distorted for display if the driver moves their head or changes their viewing direction.In this way, the ergonomic behavior of an analog mirror is replicated. This eliminates the need for the driver to make changes to the display image via a menu. Such changes are made automatically based on the driver's gaze direction and eye position. The driver is therefore not distracted by display settings while driving and can intuitively derive relevant information from the display image through natural head movements.

[0019] In addition to displaying the information on the screen, the traffic image is modified based on previously identified classified objects. For example, objects classified as particularly dangerous can be highlighted or enlarged. This display of classified objects improves driver safety when dealing with hazardous situations on the road. Furthermore, by modifying the traffic image based on the driver's viewing direction and / or eye position, as well as the classified objects, the visual representation of the traffic area behind the vehicle can be further enhanced.

[0020] By knowing the driver's gaze direction and / or eye position, the system can also recognize gestures, which in turn can be used to modify the display image according to the driver's wishes. For example, the driver can influence the area of ​​the traffic image shown on the display by moving their head or eyes. Furthermore, it is possible to acknowledge previously displayed warnings about particularly dangerous objects and return to a different display view using gestures.

[0021] The method according to the invention thus increases the user-friendliness of a display, as manual adjustment of the display image is no longer necessary. Furthermore, object recognition and the corresponding modification of the display image facilitate the driver's perception of objects classified as hazardous. By adapting the display image to the driver's line of sight and head position, a more ergonomic depth perception is achieved. This allows for better estimation of distances to objects behind the vehicle. The method is also suitable for use in display devices that can be retrofitted into the vehicle after its manufacture. In this way, the method can also improve safety in existing vehicles with regard to the representation of the traffic area behind the vehicle.

[0022] In one embodiment, it is provided that, for displaying the display image in process step E), at least a section of the traffic image is shown depending on the driver's viewing direction and / or the distance of the driver's eyes from the display. In this embodiment, only a partial area of ​​the traffic image is selected and displayed. The previously determined viewing direction and / or distance of the driver's eyes from the display selects a corresponding section of the traffic image. This simulates the behavior of an analog mirror.

[0023] In a case where, for example, the distance between the driver's eyes and the display is reduced, a smaller portion of the traffic image is shown on the display, which is equivalent to magnification. This behavior is also observed in an analog mirror when the driver moves their head closer to it. Furthermore, when the display image is shown, a section of the traffic image is selected based on the driver's viewing direction. The previously determined viewing direction, for example, in the form of a vector, allows the system to calculate the driver's viewing angle relative to the display and adjust the display image accordingly. If the driver looks at the display at an acute angle, a different image is displayed than if the driver looks at it at an obtuse angle. This behavior also corresponds to an analog mirror and can be described as the parallax effect.

[0024] In a further embodiment, the section of the traffic image displayed on the screen is deformed, in particular shifted, rotated, or distorted, by an image processing algorithm depending on the driver's viewing direction. It is possible that the display image is virtually deformed based on the traffic image before being shown on the screen. The traffic camera is preferably installed in a fixed location. However, the driver's viewing direction can change, so that the viewing angle to the display varies. With an analog mirror, the perceived reflection changes depending on the viewing direction. This effect can be simulated by an image processing algorithm. A section of the traffic image can be shifted, rotated, or distorted to simulate different viewing directions of the driver.

[0025] In a further embodiment, the objects detected in process step D) are classified according to their hazard potential and highlighted in the display depending on this classification. Object detection in process step D) preferably occurs in real time. An intelligent algorithm detects objects in the traffic area. These objects can be, for example, other road users or obstacles such as trees or walls. The objects are classified according to their hazard potential. For example, very fast-moving objects can be classified as more dangerous than slow-moving objects. Furthermore, emergency vehicles, such as ambulances, can be classified in a different category than ordinary road users. This classification makes it possible to specifically alert the driver to objects with a high hazard potential.The system handles the classification process, providing the driver with targeted information most relevant to improving road safety. For example, detected objects can be highlighted by flashing. Furthermore, classified objects can be displayed in color against a black and white background. This highlighted display can be shown for a limited time and acknowledged, for instance, by the driver moving their head, allowing the display to return to a different view.

[0026] In one embodiment, the objects detected in process step D) are classified according to their hazard potential. Depending on this classification, a section of the traffic image is displayed, showing at least one classified object, and in particular, this classified object is displayed enlarged relative to other objects in the image. In this embodiment, at least one classified object is displayed in a specifically targeted section of the traffic image. For example, if an object with a high hazard potential, such as an ambulance, is detected, this object can be displayed in an enlarged view. This intuitively draws the driver's attention to the object with the high hazard potential.It is also possible to display a non-enlarged section of the traffic image and an enlarged section containing the classified object simultaneously. In this embodiment as well, the section of the traffic image selected based on the classification is only displayed for a limited time, and the display reverts to a different image once the driver has recognized the hazard and acknowledged the warning.

[0027] In a further embodiment, the analysis of the traffic space image in process step D) is performed by artificial intelligence trained for object recognition. Object recognition can be carried out by an AI-based algorithm. This algorithm has already been trained to detect objects in the traffic space. Furthermore, it is possible for this algorithm to continue learning during object recognition and, for example, to be further trained via a radio link between multiple vehicles using information and image data from other vehicles. The use of such an algorithm enables very accurate object recognition in real time.

[0028] According to the invention, the analysis in process step D) determines whether a classified object leaves the display area of ​​the traffic camera. If such an object leaves the display area, at least a portion of an additional image of this object, captured by an auxiliary camera, is displayed when the main display image is shown, particularly with the object highlighted in that portion of the additional image. In this embodiment, at least a portion of an additional image captured by an auxiliary camera is displayed as needed when the main display image is shown. This auxiliary camera is located at a different position than the main traffic camera and captures a different viewing angle than the main traffic camera.The additional camera can, for example, be aimed at the so-called blind spot, which cannot be detected either by the traffic camera or by the driver looking sideways from inside the vehicle.

[0029] The system analyzes, preferably continuously, whether at least one classified object leaves the display area or detection angle of the traffic camera due to its movement. If such a departure is detected, a portion of the supplementary image is displayed on the screen, showing the object that has left the camera's display area. The display can then show this portion of the supplementary image together with the main display image based on the traffic camera image. If the object shown in the supplementary image is classified as having a high hazard potential, it can be further highlighted or enlarged. Such highlighting or enlargement can also occur, for example, only when it is detected that the driver is steering the vehicle in such a way that it is moving towards the detected object.In this way, the driver can be better alerted to objects that are in the blind spot and therefore pose an increased danger in road traffic.

[0030] In a further embodiment, it is provided that in process step C), the image of the driver's head is analyzed for at least one gesture, and upon detection of this gesture, the display image is modified in process step E). In this embodiment, the image or a sequence of images of the driver's head is analyzed for gestures. For this purpose, the corresponding algorithm was previously trained to recognize these gestures. The gestures could, for example, be a nod of the head. If a corresponding gesture is detected, it can be used to control or influence the display image. For example, if it was previously detected that an object with an increased hazard potential is moving quickly in the traffic area, it can initially be highlighted to draw the driver's attention to this object.Once the driver has recognized this object and adjusted their driving accordingly, the warning can be acknowledged with a gesture, and the display image can be changed to show a standard view of the traffic situation. This gesture-based change to the display image is very ergonomic and ensures that the driver is not distracted from driving by manual input or adjustments to the display.

[0031] The object of the invention is further solved by a display device for showing a traffic area behind a vehicle to a driver of the vehicle when the driver is sitting in a driver's seat of the vehicle, comprising - at least one traffic camera aimed at the traffic area behind the vehicle, - at least one head-mounted camera that can be aimed at the driver's head, - at least one display which is positioned within the driver's field of vision, - at least one computing unit, which is connected via a data connection to the traffic space camera, the head camera and the display, wherein the computing unit is configured to perform a procedure according to one of the embodiments described above.

[0032] The display device according to the invention is configured to carry out a method according to the invention. The modification of the traffic image for displaying the image is performed by a processing unit, which makes the intended changes to the traffic image based on an intelligent algorithm. The processing unit is preferably arranged in or on the vehicle. It is also possible for the processing unit to communicate wirelessly with a central processing unit, or for part of the algorithm to be performed by a further processing unit located outside the processing unit. The features, effects, and advantages previously described in connection with the method also apply analogously to the display device.

[0033] The object of the invention is ultimately achieved by a vehicle, in particular a motor vehicle, which includes at least one display device according to the embodiment described above.

[0034] The vehicle according to the invention comprises at least one display device. It is also possible for the vehicle to comprise several display devices. For example, a first display device can form a first virtual exterior mirror and a second display device a second virtual exterior mirror. A single computing unit can perform the process steps C), D), and E) of the method according to the invention. A vehicle according to the invention improves the display of the traffic area behind a vehicle and enables increased safety for the driver when dealing with hazardous situations in road traffic.

[0035] Features, effects, and advantages disclosed in connection with the method are also deemed disclosed in connection with the display device and the vehicle. Conversely, features, effects, and advantages disclosed in connection with the display device and the vehicle are also deemed disclosed in connection with the method.

[0036] The invention is schematically illustrated in the drawing with reference to one embodiment and is further described with reference to the drawing. It shows: Fig. 1 in a schematic top view an embodiment of a vehicle with a display device according to the invention.

[0037] Fig.Figure 1 shows a schematic top view of an embodiment of a vehicle 100 with a display device 10 according to the invention. The vehicle 100 is schematically depicted in a top view from above. The illustrated embodiment of the display device 10 here forms a virtual exterior mirror on the driver's side. The head K of the driver of the vehicle 100 is symbolically shown from above. The driver is looking with his head K in a direction BR, which here is directed from the position of the eyes towards a display 4 located in the interior of the vehicle 100. The head K is captured by a head-mounted camera 3, which can be aligned with the head K of the driver of the vehicle 100. In the illustrated embodiment, the head-mounted camera 3 is aligned with the head K of the driver. The images captured by the head-mounted camera 3 are analyzed, and the direction BR and the position of the eyes are determined from them.A traffic camera 2 is arranged on the left exterior side of the vehicle 100 and records images of the traffic area behind and beside the vehicle 100. In the illustrated embodiment, the display device 10 according to the invention further comprises an additional camera 5, which is arranged at the rear of the vehicle 100. The viewing angle or recording angle of the additional camera 5 differs from the viewing angle or recording angle of the traffic camera 2. The viewing angles or recording angles of the traffic camera 2 and the additional camera 5 partially overlap. The display device 10 further comprises a processing unit R, which is arranged in the interior of the vehicle 100. This processing unit R is connected via a data connection to the traffic camera 2, the head camera 3, the additional camera 5, and the display 4. The processing unit R is configured to carry out a method according to the invention.In the traffic area behind the vehicle, several objects O are depicted, symbolized as circles. These objects O could, for example, be other road users moving in the traffic area behind vehicle 100.

[0038] During the execution of the procedure, the driver's gaze direction BR and eye position are determined, preferably continuously. Based on these parameters, the distance D between the driver's eyes and the display 4 is calculated. The processing unit R modifies the traffic image captured by the traffic camera 2 depending on the gaze direction BR and the driver's eye position. For example, if the driver's head K moves closer to the display 4, thereby reducing the distance D, the displayed image on the display 4 can be enlarged accordingly. This simulates the behavior of an analog mirror, and the driver is not distracted by the need for manual adjustments to the display 4. Additionally, the traffic image is modified depending on the detected objects O. The traffic image is then analyzed, and object recognition is performed.The detected objects O are classified, for example, based on their potential danger to the current driving situation. The two objects O displayed on the right and in the center of the traffic camera 2's field of view could, for instance, be detected motorcyclists who have begun to overtake. Since these motorcyclists could pose a danger to road safety if the vehicle is steered 100 degrees to the left, the objects O classified as motorcyclists are highlighted on display 4, at least temporarily. This serves to draw the driver's attention to the danger. Once the driver has recognized the danger, they can acknowledge the highlighted display of these objects O to return to a display where these objects O are no longer highlighted.Such acknowledgment can be effected, for example, by a head gesture K, which is recognized by the processing unit R based on the images from the head camera 3. Influencing or changing the display image in this way via a gesture is very ergonomic and does not distract the driver from their driving task.

[0039] The object O shown on the left side of the display area or field of view of traffic camera 2 is marked with a dashed arrow pointing to the left. Based on previously acquired images from traffic camera 2, it was detected that this object O is moving to the left, out of the camera's field of view. Once this object O leaves the field of view of traffic camera 2, it is no longer visible in the traffic image. This can occur, for example, if the object O moves into a blind spot next to or behind vehicle 100. After moving out of the field of view of traffic camera 2, the object O is shown again on the far left. This leftmost image of object O is within the field of view of the auxiliary camera 5 and is therefore still captured by it.If the described movement of object O is detected, at least a portion of the supplementary image is displayed on screen 4 to inform the driver that object O is still present and must be taken into account while driving. If it is detected that object O poses a high hazard potential outside the field of view of the traffic camera 2, the corresponding portion of the supplementary image can be highlighted on screen 4 to specifically alert the driver to the increased hazard potential. REFERENCE MARK LIST: 10 Display device 2 traffic camera 3 Head camera 4 Display 5 additional cameras 100 vehicles BR Viewing direction Distance O object K head R processing unit

Claims

[1] Method for indicating a traffic area behind a vehicle (100) to a driver of the vehicle when the driver is seated in a driver's seat of the vehicle, comprising the following steps: A) Recording at least one traffic space image in a traffic space behind a vehicle (100) by at least one traffic space camera (2), B) Recording at least one image of the head (K) of the driver of the vehicle (100) by at least one head camera (3), C) Analysis of the image of the driver's head (K) taken in process step B) and determination of the driver's gaze direction (BR) and / or the position of the driver's eyes, D) Analysis of the traffic space image recorded in process step A) and performance of object recognition in the recorded image, whereby recognized objects (O) are classified, E) Displaying an image to inform the driver about the traffic area behind the vehicle (100) on a display (4), wherein the image is changed based on the traffic area image depending on the direction of view (BR) of the driver determined in process step C) and / or the position of the driver's eyes as well as depending on the objects (O) detected and classified in process step D), characterized by , that in the analysis in process step D) it is determined whether a classified object (O) leaves the display area of ​​the traffic space camera (2), wherein, in the event of such a detected departure from the display area, at least a sub-area of ​​an additional image of this object (O) taken by an additional camera (5) is displayed when the display image is shown, in particular wherein the object (O) is highlighted on the sub-area of ​​the additional image. [2] Method according to claim 1, wherein, for displaying the display image in method step E), at least a section of the traffic space image is displayed depending on the direction of view (BR) of the driver and / or the distance (D) of the position of the driver's eyes to the display (4). [3] Method according to claim 1, wherein the section of the traffic space image displayed when the display image is shown is deformed, in particular shifted, rotated or distorted, by an image processing algorithm depending on the viewing direction (BR) of the driver. [4] Method according to any one of claims 1 to 3, wherein the objects (O) detected in method step D) are classified according to their hazard potential and are highlighted in the display image depending on this classification. [5] Method according to one of claims 1 to 4, wherein the objects (O) detected in method step D) are classified according to their hazard potential and, depending on this classification, a section of the traffic area image is displayed in the display image in which at least one classified object (O) is shown, in particular wherein this classified object (O) is shown enlarged relative to other objects (O) in the display image. [6] Method according to any one of claims 1 to 5, wherein the analysis of the traffic space image in method step D) is carried out by an artificial intelligence trained for object recognition. [7] Method according to any one of claims 1 to 6, wherein in process step C) the image of the driver's head (K) is analyzed for at least one gesture and, upon recognition of this at least one gesture, the display of the display image is changed in process step E). [8] Display device (10) for indicating a traffic area behind a vehicle (100) to a driver of the vehicle when the driver is seated in a driver's seat of the vehicle, comprising - at least one traffic camera (2) which is directed towards the traffic area behind the vehicle (100), - at least one head-mounted camera (3) which can be directed towards the head of the driver of the vehicle (100), - at least one display (4) which is arranged within the driver's field of vision of the vehicle (100), - at least one computing unit (R) which is connected via a data connection to the traffic space camera (2), the head camera (3) and the display (4), wherein the computing unit (R) is configured to perform a method according to one of claims 1 to 7. [9] Vehicle (100), in particular motor vehicle, comprising at least one display device (10) according to claim 8.

Images