Driver monitoring system for a motor vehicle

Abstract

The invention relates to a driver monitoring system (B) for a motor vehicle (1) comprising a detection device (5) for detecting a driver area (F) in which a driver (D) is located while steering the motor vehicle (1), and a display device (3) with a screen (3a) for displaying information to the driver (D). The screen (3a) is arranged behind a steering mechanism (2) of the motor vehicle (1) from the driver's (D) position. The detection device (5) comprises at least two detection units (5a), each integrated into the edge region (R) of the screen and / or attached directly to the edge region (R) of the screen (3a). The detection units (5a) are positioned relative to each other on opposite sides (S1 - S4) of the screen (3a). They are configured to jointly detect the entire driver area (F) for driver monitoring purposes.

Description

[0001] The invention relates to a driver monitoring system for a motor vehicle. Furthermore, the invention relates to a motor vehicle with a corresponding driver monitoring system. The invention also relates to a method for operating a corresponding driver monitoring system.

[0002] To comply with legal requirements, modern vehicles that enable (partially) automatic or (partially) autonomous driving are required to monitor the driver. This is achieved, for example, through the use of driver monitoring cameras. Their function is to monitor the driver's attention and fatigue while operating the vehicle. This ensures that the driver can always take over the driving task or control the vehicle. Various types of driver monitoring systems are known from current technology.

[0003] For example, DE 10 2014 215 843 A1 discloses a driver observation system for a motor vehicle, with several cameras integrated into a combination instrument for driver observation.

[0004] German patent DE 10 2017 207 206 A1 discloses a method for controlling a driver monitoring system for observing the driver of a vehicle. Two camera modules are arranged on the vehicle's instrument cluster. They can, for example, be located below the field of view of a head-up display.

[0005] German patent DE 10 2019 211 967 B3 discloses a device for displaying a display element and for detecting a driver in a motor vehicle. The device comprises a driver monitoring device arranged on a steering mechanism and a display device. Display areas of the display device and detection units of the driver monitoring device are arranged together in a display field of the device.

[0006] Furthermore, the publication DE 10 2014 215 856 A1 describes a driver observation system in a motor vehicle and the publication DE 11 2018 003 728 T5 describes an imaging device and a vehicle display device.

[0007] In current technology, a driver observation camera is located on the steering column shroud directly above the steering wheel housing. The camera's field of view extends directly over the center of the steering wheel, ensuring an unobstructed view of the driver at all times. However, a disadvantage of this design is that the camera, and any necessary infrared lights required for nighttime driver observation, are also positioned within the driver's field of vision for the driver information display. This display is typically located behind the steering wheel housing from the driver's perspective. Therefore, the camera's placement can obstruct the driver's view of the display in certain seating or steering wheel positions. This display, however, shows traffic-related and legal information that the driver should be able to see at all times.

[0008] The goal is therefore to avoid this obstruction of the driver's view of relevant information on the driver information display. One idea is to place a driver monitoring camera within the display. However, studies have shown that the camera cannot be positioned at the edge of the driver information display, but rather would need to be located precisely in the center to reliably capture the driver under all conditions. At the edge of such a display, the steering wheel rim always obstructs the view, and the driver's face may not be fully captured by the camera. This "optimal position" for the camera, however, has the disadvantage of creating a blind spot in the display where nothing can be displayed, or at least the image quality is degraded.This poses a particular challenge within a dynamic display.

[0009] The object of the present invention is to provide a means of driver observation that reliably monitors the driver while simultaneously allowing the driver an unrestricted view of a display.

[0010] The problem is solved by the subject matter of the independent patent claims. Advantageous embodiments of the invention are disclosed by the dependent claims, the description, and the figures.

[0011] According to one aspect, the invention relates to a driver monitoring system for a motor vehicle. The driver monitoring system comprises a detection device for capturing the driver's area while steering the motor vehicle. Furthermore, the driver monitoring system comprises a display device with a screen for displaying content to the driver, the screen being arranged behind the steering mechanism of the motor vehicle from the driver's perspective. The detection device comprises at least two detection units, each integrated into an edge or peripheral area of ​​the screen and / or attached directly to an edge area of ​​the screen. The detection units are positioned relative to each other on opposite sides of the screen.The detection units are designed to jointly and completely capture the driver's area, in which the driver is located while steering the vehicle.

[0012] In other words, a dual-view camera is provided as a detection device. This preferably involves using two geographically separate cameras at the edge of the display, such as video cameras (detection units), to generate a combined image of the driver or the driver's area. This allows the driver to be detected regardless of their seat or steering wheel position.

[0013] This offers the advantage that a complete image or representation of the driver can be generated from the image data of the detection units. This allows for reliable observation of the driver. Furthermore, by positioning the detection units at the edge of the vehicle, it is effectively prevented that the driver's view of the display is obstructed.

[0014] The screen can, for example, be configured as a driver information display. The displayed content can include vehicle information such as vehicle status and / or settings, which are typically shown in a driver information display. This includes, for example, vehicle speed, fuel or charge level, engine speed, navigation information, and warning messages. In particular, this information is relevant to driving the vehicle and / or to the driver.

[0015] The screen is, for example, mounted on the dashboard and, from the driver's perspective, at least partially obscured by the steering mechanism, particularly the steering wheel rim. However, the screen, or a relevant section thereof, is fully visible to the driver through a gap between the screen housing and the inside of the steering wheel rim when the driver and steering wheel are positioned correctly. "Correctly" in this context means that the driver has adjusted the seat and steering wheel settings appropriately to their body size or optimized for viewing the displayed content.

[0016] The two or more detection units, which are primarily designed as cameras, are positioned on opposite sides of the screen. This means they can, for example, be positioned opposite each other on the same axis or offset from each other relative to the horizontal and vertical center axes of the screen.

[0017] The term "edge area" refers specifically to the peripheral area of ​​the screen. It is preferably an area where the screen no longer provides any display area for the content. The edge area is therefore an area free of the displayed content. Such an edge area results from the design, for example, when the screen is contained within a frame. Alternatively, the edge area can be artificially created by, for example, deactivating the display area in the desired edge region, i.e., by not using it for display purposes.

[0018] The two detection units jointly capture or map the driver's area. This means that the detection areas of the cameras are combined into a single observation area. This observation area encompasses the driver's area completely. To achieve this, a combined observation image for driver monitoring can be generated from the image data of individual frames (camera images) from the detection units. Generating this combined observation image from individual frames is done using established methods. The image data is fused together in a known manner.

[0019] The term "complete" specifically means that all relevant driver characteristics are visible within the camera's field of view and therefore unobstructed by the steering wheel or steering wheel rim. Relevant characteristics include, in particular, the driver's head and eyes, as these areas are known to be used for fatigue and attention detection.

[0020] In one variant, the respective edge area includes a notch in which one of the respective detection units is located. Such a notch is also referred to as a "notch." The notch is graphically integrated into the edge area. It extends at least partially into the display area. Preferably, no display content is shown in these areas. The display area is therefore reduced in size to accommodate the notch of the edge area.

[0021] As an alternative to or in addition to the first variant, a second variant of the respective detection unit comprises at least one infrared camera and at least one infrared illumination element for illuminating at least the driver's area. The use of infrared components has the advantage that driver observation can be carried out reliably, especially in darkness. The respective infrared illumination element is preferably designed as an LED (light-emitting diode).

[0022] Furthermore, in the second variant, the respective infrared camera and infrared illumination element are positioned on opposite sides of the screen. This means that not only the detection units, but also the detection unit and the illumination element are spatially separated from each other at the edge of the screen.

[0023] The invention includes embodiments that offer additional advantages.

[0024] In one embodiment, the driver monitoring system comprises a data processing device configured to generate a unified observation image from the captured image data of the detection units, which completely encompasses the driver's area. The data processing device is configured to extract at least one attention feature of the driver from the unified observation image, representing the driver's state of attention to the driving task.

[0025] The aim is therefore to determine the driver's attention to road traffic, and thus, for example, to taking over the driving task, i.e., driving the vehicle, by analyzing the observation image.

[0026] The extraction or detection of attentional features is performed using known methods. For example, an object recognition algorithm, particularly a facial expression recognition algorithm, can be applied to the observed image. In this process, attentional features such as gaze direction vectors, blinks, eye opening, and the driver's head position are determined in a known manner. These features are representative of the driver's state of attention and can therefore indicate whether the driver is capable of performing the driving task. This extraction is preferably computer-implemented, particularly using artificial intelligence methods. For example, a machine learning model, especially an artificial neural network, or a computer vision algorithm can be implemented.

[0027] Depending on the determined state of attention, for example a driver assistance system can automatically intervene in the vehicle steering, i.e. the driving task, or the driver can be warned acoustically, visually or haptically.

[0028] The data processing device can include a processor unit (processor circuit) configured to perform an embodiment of the method according to the invention. For this purpose, the processor unit can include at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor unit can include program code configured to perform the embodiment of the method according to the invention when executed by the processor unit. The program code can be stored in a data memory of the processor unit. The processor unit can, for example,based on at least one circuit board and / or at least one SoC (System on Chip).

[0029] In one embodiment, the detection areas of the detection units overlap at least partially, particularly along a respective central axis of the screen. This overlap ensures that there are no "blind spots or areas" for the cameras when detecting the driver's area. The central axis can be a vertical axis of the screen, referring to a vehicle axis in the vertical direction of the vehicle. Alternatively, the central axis can be a horizontal axis, meaning an axis that runs parallel to the vehicle's transverse direction.

[0030] In one embodiment, the edge area is limited by a mounting edge for a screen cover. The necessary mounting for the cover, such as a glass substrate of the screen, is used to define or limit the edge area. This mounting edge is also referred to as a primer edge. It is an area where, during screen assembly, an adhesive (primer) is applied to bond the cover to the screen electronics. This adhesive edge can therefore also be used to position and / or attach the sensor units.

[0031] In one embodiment, the respective detection units are provided as separate modules and attached to the edge of the screen. This means that separate components containing the detection units are provided and mounted at the edge. This offers the advantage that existing screens in vehicles can also be retrofitted with a corresponding driver monitoring capability.

[0032] Preferably, the camera and the lighting element are arranged or mounted in a common positional area on one side, a few millimeters apart, i.e., side by side. Alternatively, it is conceivable, for example, to mount the lighting element and the camera at opposite ends of one side, i.e., several centimeters apart.

[0033] According to one aspect, the invention relates to a motor vehicle with a driver monitoring system according to the invention. The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or as another vehicle with a steering wheel.

[0034] According to one aspect, the invention relates to a method for operating a driver monitoring system according to the invention. The method comprises the following steps. First, the driver's area, in which a driver is located while steering the motor vehicle, is completely and jointly detected by means of at least two detection units for driver monitoring. Furthermore, a display message is shown on the screen for the driver. Due to the design of the driver monitoring system, the display message is completely visible to the driver.

[0035] For use cases or application situations that may arise during the procedure and are not explicitly described here, it may be provided that, according to the procedure, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0036] The invention also includes further developments of the inventive method and the inventive motor vehicle, which have features as have already been described in connection with the further developments of the inventive driver observation system.

[0037] For this reason, the corresponding further developments of the inventive method and the inventive motor vehicle are not described again here.

[0038] The invention also includes combinations of the features of the described embodiments. The invention therefore also includes realizations that each exhibit a combination of the features of several of the described embodiments, provided that the embodiments have not been described as mutually exclusive.

[0039] The following are exemplary embodiments of the invention described. This is illustrated by: Fig. 1 a schematic representation of a motor vehicle with a driver monitoring system; Fig. 2 a schematic representation of the driver monitoring system for a first embodiment, and Fig. 3 A schematic representation of the driver monitoring system for a second embodiment.

[0040] The exemplary embodiments described below are preferred embodiments of the invention. In these exemplary embodiments, the described components each represent individual features of the invention, which can be considered independently of one another and each further develops the invention independently. Therefore, the disclosure is intended to include combinations of features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0041] In the figures, identical reference symbols denote functionally equivalent elements.

[0042] Fig. Figure 1 shows a schematic representation of a motor vehicle 1 from a side perspective. Fig. Figure 1 shows a section of the interior I, specifically a passenger cabin in the area of ​​the driver's seat. The vehicle 1 is, for example, an automated or autonomous vehicle. This means that the vehicle 1 is designed for automatic or automated, or at least semi-automatic or semi-automated, driving or steering, specifically without the need for intervention in the vehicle's operation by a driver. For this purpose, the vehicle 1 includes a [missing information - likely a specific feature or component]. Fig. 1 Driver assistance system or automatic vehicle guidance system not shown.

[0043] With such vehicle guidance systems, it is important that the driver D can take over control of the vehicle at any time. Therefore, so-called driver monitoring systems B are used in conjunction with such systems. These systems can monitor, for example, the driver D's attention or fatigue. Furthermore, such a driver monitoring system B can be used for other tasks, such as video calls, in the vehicle 1.

[0044] In the embodiment according to Fig. 1. The driver monitoring system B is implemented by means of a display device 3. From the perspective of the driver D, when seated in the driver's seat, the display device 3 is located behind a steering mechanism 2 of the motor vehicle 1. The display device 3 includes a screen 3a, which can, for example, be configured as a driver information display. The screen 3a can display content A, such as information about the vehicle status or vehicle settings for the driver D. This includes, for example, the vehicle speed.

[0045] In the present embodiment, the display device 3 is, for example, mounted on the dashboard and, from the driver's position D, is at least partially obscured by the steering device 2, in particular by the steering wheel rim 2a. The screen 3a, or a relevant section of the screen on which the display content A is to be shown, is fully visible to the driver D through a cutout in the steering wheel rim. In particular, the screen 3a is fully visible when the driver D and / or the steering wheel are positioned correctly relative to each other. "Intended position" means that the driver D has selected a seat or steering wheel setting that is correctly adjusted to the driver D's body dimensions or their preferences for the driving task.

[0046] To form the driver monitoring system B, a detection device 5 is integrated into the display device 3. The detection device 5 serves to detect a driver area F in which the driver D is located while steering the motor vehicle 1. The driver area F is, in particular, an area relevant for driver monitoring in which the driver D can preferably be fully detected. That is, all relevant characteristics of the driver D for driver monitoring should be visible or detectable by the detection device 5. For this purpose, it is important that the driver area F is not obstructed, in particular by the steering wheel or the steering wheel rim 2a.

[0047] In this context, the driver area F encompasses, in particular, the head and eye region of the vehicle driver D. Attention features of the vehicle driver D can be extracted from this area, representing their state of attention for assuming the driving task. These features include, for example, a gaze direction vector, blinking, the degree of eye opening, and the head position of the vehicle driver D.

[0048] In order for the detection device 5 to detect the desired driver area F, it is important to avoid obstructions to its detection range or field of view. At the same time, however, as much space as possible should be available on the screen 3a for displaying the desired content. Therefore, in the present embodiment, a special arrangement or integration of the detection device 5 into the display device 3 is implemented.

[0049] This shows Fig. 2 the steering device 2 and the driver monitoring system B from a top or frontal view, corresponding, for example, to the view of the vehicle driver D when he is in the driver area F. In the embodiment according to Fig. 2 The detection device 5 comprises, for example, two detection units 5a, which are designed in particular as cameras or dual-view cameras. The detection units 5a are each integrated into or arranged in a border area R or peripheral area of ​​the screen 3a. The detection units 5a are positioned relative to each other on different sides S1 to S4 of the screen 3a. In the present embodiment, the detection units 5a are attached to the two opposite sides S2 and S4. This type of arrangement enables the detection units 5a to jointly capture the entire driver area F for driver observation.

[0050] To illustrate this, in Fig. Figure 2 shows, for example, the detection areas E1 and E2, i.e., the viewing areas of the detection units 5a and 5b. The two detection areas E1 and E2 intersect or meet at a horizontal central axis M of the screen 3a. Of course, the detection areas E1 and E2 can also partially overlap. Combined, the detection areas E1 and E2 form the driver area F, which is to be detected or displayed by the detection device 5.

[0051] The acquisition units 5a can now each generate image data or individual images for these acquisition areas E1 and E2. These individual images can be fused or combined into a single observation image, for example, using a data processing device 4 of the vehicle 1. This observation image contains or completely depicts the driver's area F. This observation image can be processed or analyzed by the data processing device 4 using known methods to extract the desired attention characteristic of the vehicle driver D. For this purpose, the data processing device 4 can, for example, preferably perform a facial expression analysis or an object recognition algorithm automatically. The data processing device 4 can contain at least one processing circuit, in particular a processor circuit with at least one microprocessor or microcontroller.Preferably, the data processing device 4 comprises a machine learning model, in particular an artificial neural network.

[0052] As an alternative to arranging the detection units 5a opposite each other on an axis, it is also conceivable to arrange the detection units 5a on different sides S1 to S4, but offset from each other, i.e., shifted along the respective sides S1 to S4. With this type of arrangement, it is only necessary to ensure that the detection areas E1, E2 of the detection units 5a cover the driver area F as desired. It is also conceivable to provide more than two detection units 5a, for example, three, four, or more cameras for driver monitoring. With more than two detection units 5a, it is also conceivable to position two detection units 5a on a common side S1 to S4 at a predetermined distance from each other along side S1 to S4.

[0053] By integrating the detection units 5a into the edge area R, the advantage arises that a portion of the screen 3a's display area is entirely available for displaying the relevant content A. In particular, this avoids, for example, a blind spot or a blurred display, which can occur, for instance, if the detection device 5 is positioned in the center of the screen 3a or at any other arbitrary position within the display area.

[0054] In this context, the edge area R refers specifically to a mounting edge for a cover glass of the screen 3a. This mounting edge is also called the primer edge and is used to attach the cover glass, for example a glass substrate, to the screen electronics using an adhesive, the so-called primer. The edge area R is therefore particularly well suited for the integration of the detection units 5a. No display takes place in the edge area R. The edge area R thus defines the display area A of the screen 3a.

[0055] In the embodiment according to Fig. 2. The edge area R is extended to integrate the detection units 5a by means of a respective notch N. Such a notch N is also known as a notch, for example in the smartphone sector. This notch is graphically integrated into the edge area R and is therefore not part of the display area A.

[0056] The detection device 5 is intended to be used for driver observation in darkness, for example at night or in a tunnel. For this purpose, the detection device 5 comprises an infrared camera as a detection unit 5a and at least one infrared illumination element 5b. Preferably, exactly one illumination element, such as an LED or another type of light source, is provided for each camera. The respective illumination element 5b is used to illuminate the driver area F. In the exemplary embodiment according to Fig. 2. The camera and the lighting element are positioned in a common position area, here the notch N, at a distance of a few millimeters from each other.

[0057] Alternatively, it is of course also conceivable to arrange the respective camera and the respective lighting element 5b on opposite sides S1 to S4 of the screen 3a. It is also conceivable to arrange the respective camera and the respective lighting element 5b at opposite ends of a side or a few centimeters apart on a common side. The arrangement of the lighting elements 5b ensures that the driver's area F, in particular, can be fully illuminated.

[0058] Fig. Figure 3 shows an alternative embodiment of the driver monitoring system B. Here, the two or more detection units 5a are not integrated into the edge area R, but are attached or fastened directly to the edge area R of the screen 3a by means of modules K or components. This can also be implemented analogously for the lighting elements 5b. As a result, the detection units 5a and, in particular, the lighting elements 5b are provided as separate modules K, which are attached to the edge area R. This offers the advantage that an existing screen can be retrofitted with the detection device 5 as a so-called carry-over part if required.

[0059] As in the embodiment according to Fig. As shown in Figure 3, the respective camera and lighting element 5b are arranged on a common side S2, S4, but positioned at opposite ends of side S2 and S4, respectively. Of course, alternative positioning options for the detection units 5a and the lighting elements 5b are also possible, as already exemplified in Figure 3. Fig. 2 were mentioned, conceivable.

[0060] Overall, the examples demonstrate how a dual-view camera for driver monitoring can be implemented.

Claims

[1] Driver monitoring system (B) for a motor vehicle (1), comprising - a detection device (5) for detecting a driver area (F) in which a vehicle driver (D) is located while driving the motor vehicle (1), and - a display device (3) with a screen (3a) for displaying a display content (A) for the vehicle driver (D), wherein the screen (3a) is arranged from a position of the vehicle driver (D) behind a steering device (2) of the motor vehicle (1), wherein the detection device (5) comprises at least two detection units (5a), each of which is integrated into an edge area (R) of the screen (3a) and / or is attached directly to an edge area (3a) of the screen (3a), wherein the detection units (5a) are positioned relative to each other on different sides (S1-S4) of the screen (3a), and are configured to jointly detect the driver area (F) completely for driver observation, characterized by , that - the edge area (R) includes a respective notch (N) in which one of the respective detection units (5a) is arranged; and / or - the detection device (5) as a respective detection unit (5a) comprises at least one infrared camera and at least one infrared illumination element (5b) for illuminating at least the driver area (F), wherein the respective infrared camera and the respective infrared illumination element (5b) are positioned on opposite sides (S1-S4) of the screen (3a). [2] Driver observation system (B) according to claim 1, wherein the driver observation system (B) comprises a data processing device (4) which is configured to generate a common observation image from captured image data of the capture units (5a) which fully contains the driver area (F), and to extract at least one attention feature of the driver (D) from the common observation image for driver observation, which represents a state of attention of the driver for the driving task. [3] Driver monitoring system (B) according to one of the preceding claims, wherein the detection areas (E1, E2) of the detection units (5a) overlap at least partially, in particular on a respective central axis (M) of the screen (3a). [4] Driver monitoring system (B) according to one of the preceding claims, wherein the edge area (R) is limited by a mounting edge for a cover glass of the screen (3a). [5] Driver monitoring system (B) according to one of the preceding claims, wherein the respective detection units (5a) are provided as separate modules (K) and are attached to the edge area (R) of the screen (3a). [6] Motor vehicle (1) with a driver observation system (B) according to any one of claims 1 to 5. [7] Method for operating a driver monitoring system (B) according to any one of claims 1 to 5, comprising the following steps: - complete, joint recording of the driver area (F) in which a vehicle driver (D) is located while steering the motor vehicle (1), for driver observation by means of at least two recording units (5a), and - Displaying a display content (A) for the vehicle driver (D) on the screen (3a).

Images