Method for operating a gaze direction detection device of a motor vehicle, gaze direction detection device and motor vehicle

By dynamically determining the camera's position relative to a reference object using camera data, the method enhances the precision and reliability of viewing direction detection in motor vehicles, addressing issues of thermal deformation and component aging, and allowing camera placement on movable components.

DE102014206629B4Active Publication Date: 2024-11-07BAYERISCHE MOTOREN WERKE AG
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
DE102014206629
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2014-04-07
Publication Date
2024-11-07
Estimated Expiration
2034-04-07

AI Technical Summary

Technical Problem

Existing viewing direction detection systems for motor vehicles face challenges in accurately determining the viewing direction of vehicle occupants due to thermal deformation and material aging of components, which are not accounted for when the camera's relative position is initially fixed, and the inability to mount the camera on moving parts.

Method used

The method involves determining the camera's relative position to a reference object, such as a lighting unit, using camera data during operation, allowing precise detection of the viewing direction, even when the camera is attached to movable components like the steering wheel, by comparing current camera images with stored reference data.

Benefits of technology

Ensures precise and reliable detection of the viewing direction, unaffected by thermal deformation or aging, enabling the camera to be mounted on movable parts and reducing computational effort in position determination.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Method for operating a gaze direction detection device (9) of a motor vehicle (1) by which a current gaze direction (10) of a vehicle occupant (4) relative to a vehicle interior (5) and / or a vehicle environment (11) is detected, wherein camera data of the vehicle interior (5) are provided to the gaze direction detection device (9) by means of a camera (12) and the current gaze direction (10) is determined by the gaze direction detection device (9) on the basis of the camera data taking into account a relative position of the camera (12) with respect to a reference object, where the relative position of the camera (12) during operation is determined by the gaze direction detection device (9) based on camera data from the camera (12), characterized by the fact that the position of the camera (12) relative to a lighting unit (14) as a reference object is determined, through which the vehicle interior (5) is illuminated to determine the viewing direction (10).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method for operating a gaze direction detection device of a motor vehicle, by which the current gaze direction of a vehicle occupant relative to a vehicle interior and / or a vehicle environment is detected, wherein camera data of the vehicle interior is provided to the gaze direction detection device by means of a camera, and the current gaze direction is determined by the gaze direction detection device based on the camera data, taking into account the relative position of the camera with respect to a reference object. The invention further relates to a gaze direction detection device configured for carrying out such a method, and to a motor vehicle with such a gaze direction detection device.

[0002] Eye-tracking devices for motor vehicles are already known from the prior art. They are used to detect the current gaze direction of a vehicle occupant—especially the driver. The occupant's gaze direction is detected relative to the vehicle interior, for example, relative to an infotainment system display, and / or relative to the vehicle's surroundings. Depending on the occupant's current gaze direction, various functionalities can then be provided within the vehicle. For example, it may be possible to detect the driver's gaze direction with respect to a display, whether a conventional LCD display or a head-up display. If the display contains multiple menu items or graphical objects, the menu item on which the driver is currently looking can be selected.This could look something like this: the driver first selects the menu item they are currently viewing, and then makes further adjustments using a rotary push-button. In this way, various parameters of the infotainment system can be configured. Furthermore, the driver's gaze direction can also be tracked in relation to objects outside the vehicle, and it can be detected, for example, which object the driver is currently looking at. Based on this detection, the head-up display can then be controlled, and the display projected onto the windshield can be adjusted accordingly.

[0003] One such head-up display is known, for example, from document DE 101 31 720 A1.

[0004] A gaze direction detection device is known, for example, from document DE 10 2004 005 816 A1. Within the driver's field of vision are various symbol surfaces, at least one of which is a fixed, unchanging symbol with a fixed function, and at least one other of which is a variable symbol with a variable function. The gaze direction detection device detects the driver's gaze direction with respect to the symbol surfaces and determines the currently viewed symbol surface. Depending on the detection result, a function associated with the viewed symbol surface is then triggered. A camera is used to detect the current gaze direction.

[0005] The publication EP 1 284 432 A1 describes a method for operating a driver assistance system, such as a navigation system. It employs a head-up display consisting of a projector and a mirror mounted on the dashboard, which projects menu options and other information from the driver assistance system onto the windshield. A camera detects the driver's gaze, allowing them to select menu items of the driver assistance system by looking at the screen. If necessary, a selection made by looking can be confirmed using a button on the steering wheel.

[0006] Gaze direction detection devices are also known from documents DE 102 42 262 A1, DE 10 2009 002 979 A1 and DE 10 2011 084 367 A1.

[0007] DE 10 2010 053 120 A1 describes a method and a device for observing a vehicle driver by means of at least one image acquisition unit arranged on a steering wheel. According to the invention, a change in the position of the image acquisition unit resulting from a steering movement is determined using acquired image data relative to a static reference line or relative to a number of static reference points specified in the vehicle interior, based on which a steering wheel angle is determined.

[0008] From DE 10 2010 013 243 A1 a method for determining information relating to the direction of a driver's gaze and / or the position of the driver's hands in relation to the steering wheel in a motor vehicle is known, wherein data recorded by an imaging device, in particular a camera, which captures the steering wheel and the head of a driver, are evaluated to determine the information by means of image processing.

[0009] For gaze direction detection, cameras sensitive to infrared light are used in the prior art. Such a camera is typically positioned at a fixed, predetermined location within the vehicle interior, providing image data about the interior. In addition to the camera, at least one illumination unit can be used, generating infrared light and thus illuminating the vehicle interior in this specific spectral range. Eye reflections of the vehicle occupant can then be detected in the images, allowing the gaze direction to be determined. For accurate gaze direction detection, it is crucial that the relative position of the camera with respect to the vehicle interior and its components—and especially with respect to the at least one illumination unit—is known.This relative position of the camera is taken into account when determining the current viewing direction. To ensure that the system always knows the camera's relative position, the prior art typically involves mounting the camera on a vehicle component that has fixed dimensions (defined by CAD data) and is located at a known, fixed position within the vehicle interior. The camera's relative position is then known through a CAD model. An alternative is the classic method of measuring the camera's relative position to the lighting unit and the interaction surfaces within the vehicle interior. However, this method involves a relatively high measurement effort. A disadvantage of the prior art is that the camera's relative position is determined and stored in the system only once.Firstly, this does not take into account either thermal deformation of the components or material aging during operation; secondly, it is not possible to install the camera on moving parts of the vehicle.

[0010] The object of the invention is to demonstrate a solution for how, in a method of the aforementioned type, the current direction of view of the vehicle occupant can be detected more precisely than in the prior art.

[0011] This problem is solved according to the invention by a method, by a gaze direction detection device, and by a motor vehicle with the features according to the respective independent claims. Advantageous embodiments of the invention are the subject of the dependent claims, the description, and the figures.

[0012] A method according to the invention serves to operate a gaze direction detection device of a motor vehicle, by which the current gaze direction of a vehicle occupant relative to the vehicle interior and / or the vehicle environment is detected. Camera data of the vehicle interior is provided by means of a camera of the gaze direction detection device, and the current gaze direction is determined based on the camera data, taking into account the relative position of the camera with respect to a reference object, such as the vehicle interior and / or a lighting unit. According to the invention, the relative position of the camera is determined during operation of the gaze direction detection device based on camera data.

[0013] The camera's position is determined relative to a lighting unit that serves as a reference object, illuminating the vehicle interior to determine the viewing direction. This enables precise and reliable detection of the current viewing direction.

[0014] The effect according to the invention is achieved by determining the relative position of the camera with respect to the reference object during operation and thus "online," using camera data or image data provided by the camera of the gaze direction detection device. This approach has the advantage that the exact relative position of the camera is always known, and the gaze direction of the vehicle occupant can therefore be detected with particular precision and accuracy. The detection of the gaze direction is not affected by thermal deformation or aging of the components. Furthermore, the method according to the invention makes it possible to mount the camera and / or a lighting unit of the gaze direction detection device on a movable component in the vehicle interior.According to the invention, for example, it is possible to arrange the camera and / or the lighting unit on the steering wheel of the motor vehicle, which has the particular advantage that the driver's head can be captured with exceptional accuracy in the camera data. It is also possible to provide a bracket by means of which the camera's orientation can be manually adjusted within the vehicle interior. Since the exact relative position and thus also the orientation of the camera relative to the reference object is determined during operation of the gaze direction detection device, precise capture of the current gaze direction can always be guaranteed, even if the camera position changes.

[0015] The lighting unit can be designed to generate light in the infrared range. Accordingly, the camera can be sensitive to infrared light and thus detect it. The camera can then provide infrared images, which are used to determine the current viewing direction of the vehicle occupant. This method has proven to be particularly reliable.

[0016] Regarding the determination of the relative position of the camera, the following embodiments may be provided: In one embodiment, the camera data is compared with stored reference data to determine the relative position. The relative position is then determined based on this comparison. This embodiment is based on the understanding that the current relative position of the camera can be determined particularly precisely during vehicle operation by comparing the currently recorded camera data or images with stored reference data of the vehicle interior. Since the reference data can contain the relative position of reference points within the vehicle interior relative to each other, and these reference points can also be detected in the current camera data, this comparison allows the current relative position of the camera to be determined reliably and without significant computational effort.

[0017] In this context, it has proven particularly advantageous if the camera provides at least one image of the vehicle interior as camera data, and if determining the relative position involves detecting characteristic features in the image and comparing them with a stored feature image of the vehicle interior as reference data. Characteristic features can thus be extracted from the current camera image, generating a current feature image of the vehicle interior, which is then compared with a stored feature image (reference data). Since the comparison only needs to be performed with respect to the characteristic features, the computation time for determining the current relative position of the camera can be reduced. Furthermore, comparing the feature images ensures a highly precise determination of the current relative position of the camera.

[0018] The characteristic features detected in the current image could be, for example, predetermined markers specifically distributed throughout the vehicle interior to aid in positioning. These markers can be installed or attached at fixed positions within the vehicle interior, allowing them to be detected in the current camera data and the camera's current relative position to be determined. The use of such markers offers the advantage of easy image detection and reliable reference points.

[0019] If at least one lighting unit of the gaze direction detection device is used as a marker, the effort can be reduced to a minimum, since existing components are used and the use of additional markers with their associated disadvantages is unnecessary.

[0020] In addition to or as an alternative to the markers mentioned, edges and / or corners, and thus generally distinctive features of the vehicle interior, can also be detected in the camera image. A gradient image or edge image (feature vector) can be generated from the current camera image, which can then be compared with a stored feature image or feature matrix based on CAD data of the vehicle interior as reference data. This also ensures a reliable determination of the camera's current relative position.

[0021] The invention further relates to a gaze direction detection device for a motor vehicle, comprising at least one camera for providing camera data of the vehicle interior, and an evaluation unit designed to determine, based on the camera data and taking into account the relative position of the camera with respect to a reference object, the current gaze direction of a vehicle occupant relative to the vehicle interior and / or relative to the vehicle's surroundings. The gaze direction detection device includes a position determination device that determines the relative position of the camera during operation of the gaze direction detection device based on camera data.

[0022] The position detection device is designed to determine the position of the camera relative to a lighting unit as a reference object, the lighting unit being designed to illuminate the vehicle interior to determine the viewing direction.

[0023] A motor vehicle according to the invention, in particular a passenger car, comprises a gaze direction detection device according to the invention.

[0024] The preferred embodiments and their advantages presented with reference to the method according to the invention apply accordingly to the gaze direction detection device according to the invention as well as to the motor vehicle according to the invention.

[0025] Further features of the invention will become apparent from the claims, the figures, and the description of the figures. All features and combinations of features mentioned above in the description, as well as those subsequently mentioned in the description of the figures and / or shown in the figures alone, are usable not only in the combinations specified, but also in other combinations or individually.

[0026] The invention will now be explained in more detail with reference to a preferred embodiment and the accompanying figures.

[0027] They show: Fig. 1 in schematic representation a motor vehicle with a gaze direction detection device according to an embodiment of the invention; and Fig. 2 a flowchart of a process according to an embodiment of the invention.

[0028] A in Fig. The vehicle 1 shown is, for example, a passenger car. The vehicle 1 includes an infotainment system 2 with a display 3. The display 3 can be, for example, an LCD display, optionally with a touch-sensitive surface, or a so-called head-up display. The infotainment system 2 can include various functions and thus be operated in different modes, such as navigation mode, telephony mode, radio mode, multimedia mode, and the like. Accordingly, the display 3 can show relevant information for the respective operating mode. Each operating mode can, for example, include its own main menu, in which the corresponding menu items or graphical symbols are displayed, which can be selected by a driver 4 of the vehicle 1.

[0029] With further reference to Fig. 1 The motor vehicle 1 includes a vehicle interior 5 in which a steering wheel 6 is located on the driver's side in a manner known per se, which is arranged between a windshield 7 on the one hand and a driver's seat 8 on the other hand and serves to steer the steerable (front) wheels of the motor vehicle 1.

[0030] The infotainment system 2 includes a gaze direction detection device 9, which is designed to detect the current gaze direction 10 of the driver 4 (and / or another vehicle occupant) relative to the vehicle interior 5, in particular relative to the display 3, and / or relative to the vehicle's surroundings 11. For this purpose, the gaze direction detection device 9 includes a camera 12, an electronic evaluation unit 13, and optionally at least one illumination unit 14. The camera 12 provides camera data or images of the vehicle interior 5 and transmits this data to the evaluation unit 13, which uses the camera data to detect the current gaze direction 10. The camera 12 is sensitive to infrared light, so the camera data provided in this embodiment represents infrared images. Accordingly, the at least one illumination unit 14 is designed to generate light in the infrared range.The determination of the gaze direction 10 can be carried out, for example, by the evaluation device 13 searching for eye reflexes in the images and determining the gaze direction 10 depending on the detected eye reflexes.

[0031] Various configurations are possible with regard to how the infotainment system 2 takes the current viewing direction 10 into account. In general, the current viewing direction 10 can be used to control the infotainment system 2. For example, depending on the viewing direction 10, the aforementioned menu items can be selected on the display 3. If the display 3 is designed as a head-up display, the display can be controlled depending on the current viewing direction 10 relative to the vehicle's surroundings 11.

[0032] When determining the current viewing direction 10, the evaluation unit 13 takes into account the relative position of the camera 12 relative to a reference object. The at least one lighting unit 14 is used as the reference object. In the exemplary embodiment, the relative position of the camera 12 is determined during operation of the infotainment system 2. For this purpose, the evaluation unit 13 includes a position detection device 15, which determines the relative position of the camera 12 based on current camera data or a current image of the vehicle interior 5. Since the relative position of the camera 12 is thus always known in the evaluation unit 13, the camera 12 and / or the at least one lighting unit 14 can also be mounted on a movable component of the motor vehicle 1, such as the steering wheel 6.In addition or alternatively, a holder can be provided for the camera 12 mount, by means of which the orientation of the camera 12 in the vehicle interior 5 can be manually adjusted.

[0033] A method for determining the current position of camera 12 is described below with reference to Fig.2. In more detail: According to a first step S1, a CAD model of the vehicle interior 5 is provided during the manufacture / development of the motor vehicle 1. Additionally or alternatively, so-called markers can be installed in the vehicle interior 5, which then support position determination during operation of the infotainment system 2. For example, at least one lighting unit 14 can be used as a marker. If the CAD model of the vehicle interior 5 is available and / or the aforementioned markers are installed at predefined positions in the vehicle interior 5, a feature extraction can be performed in a further step S2. This provides reference data according to step S3, which contains a feature image or feature matrix of the vehicle interior 5 and / or the installed markers. The reference data thus includes fixed features of the vehicle interior 5, such as edges and / or corners, and / or the permanently installed markers as a reference.

[0034] Steps S1 to S3 are therefore performed once during the development / manufacturing of the vehicle 1, and the reference data is stored in the evaluation unit 13. Further steps of the procedure are then carried out during the operation of the infotainment system 2: According to step S4, the camera 12 provides a current image of the vehicle interior 5 and transmits it to the evaluation unit 13. The position detection unit 15 then extracts predetermined characteristic features from the current image according to step S5 and provides a current feature image (feature vector) of the vehicle interior 5 according to step S6. These characteristic features can include the permanently installed markers and / or prominent features, such as edges and / or corners, detected in the current image.

[0035] According to step S7, the position determination device 15 then compares the features extracted from the current image with the stored reference data. Depending on this comparison, the relative position of the camera 12 is determined according to step S8.

[0036] Depending on the vehicle component on which camera 12 is located, steps S4 to S8 can be repeated at the required frequency. For example, if camera 12 is mounted on the steering wheel 6 or another moving component, steps S4 to S8 can be performed several times per second, allowing the relative position of camera 12 to be determined in real time. If camera 12 is mounted on a static component, the position determination frequency can be set lower.

Claims

[1] Method for operating a viewing direction detection device (9) of a motor vehicle (1), by means of which a current viewing direction (10) of a vehicle occupant (4) relative to a vehicle interior (5) and / or a vehicle environment (11) is detected, wherein camera data relating to the vehicle interior (5) are provided by means of a camera (12) of the viewing direction detection device (9) and the current viewing direction (10) is determined by the viewing direction detection device (9) on the basis of the camera data, taking into account a relative position of the camera (12) with respect to a reference object, where the relative position of the camera (12) is determined during operation by the viewing direction detection device (9) based on camera data from the camera (12), characterized by , that the position of the camera (12) is determined relative to a lighting unit (14) as a reference object, by means of which the vehicle interior (5) is illuminated in order to determine the viewing direction (10). [2] Method according to claim 1, characterized by that the position of the camera (12) relative to the vehicle interior (5) is determined as a reference object. [3] Method according to one of the preceding claims, characterized by that the lighting unit (14) generates light in the infrared range. [4] Method according to one of the preceding claims, characterized by that determining the relative position comprises comparing the camera data with stored reference data and determining the relative position depending on the comparison. [5] Method according to claim 4, characterized bythat at least one image of the vehicle interior (5) is provided as camera data by the camera (12), and determining the relative position comprises detecting characteristic features in the image and comparing them with a stored feature image as reference data. [6] Method according to claim 5, characterized by that predetermined markers are detected in the image as characteristic features, which are distributed in the vehicle interior (5) to support the position determination. [7] Method according to claim 1 or 3 and claim 6, characterized by that the at least one illumination unit (14) is detected in the image as a marker. [8] Method according to one of claims 5 to 7, characterized by that edges and / or corners of the depicted vehicle interior (5) are detected as characteristic features in the image. [9] Viewing direction detection device (9) for a motor vehicle (1), with at least one camera (12) for providing camera data to a vehicle interior (5) of the motor vehicle (1), and with an evaluation device (13) which is designed to determine a current viewing direction (10) of a vehicle occupant (4) relative to the vehicle interior (5) and / or a vehicle environment (11) based on the camera data and taking into account a relative position of the camera (12) with respect to a reference object, where the viewing direction detection device (9) has a position determination device (15) which is designed to determine the relative position of the camera (12) during operation of the viewing direction detection device (9) on the basis of camera data of the camera (12) characterized by , that the position determination device (15) is designed to determine the position of the camera (12) relative to a lighting unit (14) as a reference object, wherein the lighting unit (14) is designed to illuminate the vehicle interior (5) to determine the viewing direction (10). [10] Motor vehicle (1), in particular passenger car, with a viewing direction detection device (9) according to claim 9.

Citation Information

Patent Citations

  • head-up display system and method

    DE10131720A1

  • motor vehicle

    DE102004005816A1

  • Projection display device for vehicles, comprises sensor units formed to generate projection data from sensor values and location information, where control unit is formed to perform filtering of projection data

    DE102009002979A1

  • Methods for determining information relating to the direction of a driver's gaze and the position of the driver's hands in relation to the steering wheel in a motor vehicle and motor vehicle

    DE102010013243A1

  • Method and device for observing the driver of a vehicle

    DE102010053120A1