Assistance system and assistance procedures for a vehicle

Abstract

The present disclosure relates to an assistance system (100) for a vehicle (10), comprising: - an occupant detection module (110) that is set up to detect an occupant of the vehicle (10) and provide corresponding detection data; - an analysis module (120) configured to determine, based on the acquisition data, a first gaze direction (BR1) of the occupant's first eye and a second gaze direction (BR2) of the occupant's second eye, wherein the analysis module (120) is further configured to determine, in the event of a discrepancy between the first gaze direction (BR1) and the second gaze direction (BR2), a primary eye of the occupant based on at least one context factor; and - at least one control module (130) that is set up to control at least one vehicle function based on a viewing direction (BR1, BR2) of the primary eye.

Description

[0001] The present disclosure relates to a vehicle assistance system, a vehicle with such an assistance system, a vehicle assistance method, and a storage medium for executing the assistance method. The present disclosure relates in particular to a correction of gaze direction detection in cases of eye misalignment. State of the art

[0002] Monitoring driver attention is crucial for modern driver assistance systems. Precise and reliable detection of the driver's gaze direction is essential to ensure that the driver reacts to relevant traffic situations and recognizes potential hazards in time. Due to individual differences between people and the fact that the systems are often based on generic assumptions, difficulties can arise in accurately determining the gaze direction in certain situations. This can lead to erroneous assessments, triggering, for example, unnecessary driver attention warnings. Disclosure of the invention

[0003] The purpose of this disclosure is to specify a vehicle assistance system, a vehicle with such an assistance system, a vehicle assistance method, and a storage medium for executing the assistance method, all of which enable precise detection of a vehicle occupant's gaze direction. In particular, the purpose of this disclosure is to enable improved driver attention monitoring and thus enhance road safety.

[0004] This problem is solved by the subject matter of the independent claims. Advantageous embodiments are specified in the dependent claims.

[0005] According to an independent aspect of the present disclosure, an assistance system for a vehicle, in particular a motor vehicle, is specified. The assistance system comprises: - an occupant detection module that is set up to detect an occupant of the vehicle and provide corresponding detection data; - an analysis module configured to determine, based on the acquisition data, a first gaze direction of the occupant's first eye and a second gaze direction of the occupant's second eye, wherein the analysis module is further configured to determine, in the event of a discrepancy between the first gaze direction and the second gaze direction, a primary eye of the occupant based on at least one context factor, and - at least one control module that is set up to control at least one vehicle function based on the direction of gaze of the primary eye.

[0006] According to the invention, the gaze directions of both of the occupant's eyes are recorded separately in order to detect any eye misalignment. If such a misalignment is detected, a primary eye—that is, the eye predominantly responsible for visual perception and information processing—is determined by taking contextual factors into account. For example, the contextual factor could be a red traffic light, with the eye whose gaze is directed towards the red light being identified as the primary eye. Such precise detection of the gaze direction enables, for example, improved driver attention monitoring, which in turn can increase road safety.

[0007] The occupant detection module, the analysis module and the at least one control module may include software components / algorithms that are set up to run on at least one processor and thereby perform the functionalities of the respective module.

[0008] The occupant detection module and / or the analysis module and / or the at least one control module can be implemented in a single software and / or hardware module. Alternatively, the occupant detection module and / or the analysis module and / or the at least one control module can each be implemented in separate software and / or hardware modules.

[0009] Preferably, the occupant of the vehicle is the driver of the vehicle. However, the present disclosure is not limited to this and it is conceivable to apply the embodiments of the present disclosure to other vehicle occupants, such as a front passenger or rear passenger.

[0010] The analysis module is designed to detect discrepancies between the first and second gaze directions, and thus eye misalignment. In this context, the term "eye misalignment" (medically: strabismus) refers to a deviation of the visual axes where the gaze directions of both eyes are not correctly aligned on a common target. This results in the eyes not focusing synchronously on the same object.

[0011] The direction of gaze refers to the orientation of the eyes towards a specific object, person, or point in space. It indicates where a person is looking or what area the person is visually perceiving. The direction of gaze can be determined using various methods, such as observing eye movements, pupil position, and / or head orientation.

[0012] The term "primary eye" in the context of a refractive error refers to the eye that, despite the error, is predominantly responsible for visual perception and information processing. This eye takes on the main task of visually perceiving the environment and driving situations, while the other eye may be partially impaired in its function. The primary eye is therefore the dominant eye, which receives the most visual information and transmits it to the brain, for example, to optimize the driver's vision, particularly in the context of attention detection or gaze tracking.

[0013] Preferably, the analysis module is configured to detect the deviation between the first and second viewing directions when these directions deviate from parallel alignment by more than a threshold. This ensures that only deviations that actually impair visual perception are considered, while deviations that do not affect visual perception or only minimally do so are disregarded.

[0014] Preferably the threshold is 3° or more, or 5° or more. Typically, the threshold is 5°.

[0015] Preferably, the occupant detection module comprises, or is, interior sensors. The interior sensors can include at least one optical sensor, such as a camera, in particular an interior camera. Optionally, the occupant detection module can include at least one light source. In some embodiments, the light source can be an infrared light source, and the at least one optical sensor can be an infrared sensor, such as an infrared camera.

[0016] The analysis module can be configured to evaluate the data collected by the interior sensors, such as image data, using suitable software. By evaluating this data, the occupant's gaze direction can be determined.

[0017] Preferably, the assistance system, and in particular the analysis module, is configured to capture the direction of gaze using eye tracking. The term "eye tracking" refers specifically to the recording of eye movements, which are detected, for example, by a camera in the interior sensor system. This allows insights to be gained into where a person is looking.

[0018] Preferably, at least one contextual factor relates to a driving situation. The driving situation refers to the circumstances and conditions under which the vehicle moves in road traffic. Factors that determine the driving situation may include, for example, traffic conditions and traffic rules; however, the present disclosure is not limited to these.

[0019] Preferably, the at least one context factor relates to at least one situationally relevant object in the vehicle's environment. A situationally relevant object in the vehicle's environment is an object that, in a specific driving situation, is significant for road safety and the driver's behavior. Such objects can influence the vehicle's behavior or require special attention to avoid accidents. These objects can be either static or dynamic. Examples of situationally relevant objects include, but are not limited to, other road users, road infrastructure (e.g., traffic signs, traffic lights, etc.), and obstacles.

[0020] Preferably, the analysis module is configured to determine the primary eye of the occupant whose gaze is directed towards at least one situationally relevant object.

[0021] Preferably, the analysis module is configured to further determine the occupant's primary eye based on the occupant's head orientation and / or head rotation direction. For example, the head orientation and / or head rotation direction with respect to the at least one situationally relevant object can be detected and analyzed to determine which of the two eyes is directed at the at least one situationally relevant object.

[0022] Preferably, the vehicle includes an environment detection module that is configured to detect at least one situationally relevant object based on environmental data from an environmental sensor system of the vehicle.

[0023] Preferably, the environmental sensor system comprises at least one LiDAR system and / or at least one radar system and / or at least one camera and / or at least one ultrasonic system. The environmental sensor system can provide the environmental data (also referred to as "surrounding data") that maps the area surrounding the vehicle.

[0024] Preferably, at least one context factor relates to a user interface module in the vehicle. The user interface module can, for example, be a central information output and input device of an infotainment system, such as a head unit or a pillar-to-pillar display. Preferably, the user interface module is permanently installed in the vehicle.

[0025] Preferably, the user interface module includes, or is, a touchscreen.

[0026] Preferably, the analysis module is configured to determine the primary eye of the occupant whose gaze is directed towards the user interface module, in particular a display element and / or control element of the user interface module. When operating a vehicle function, the gaze and often also the head position are usually directed towards the display or switch to trigger the desired function. Simultaneously, the gaze direction of both eyes can be recorded separately, so that at the moment of operation, it can be precisely determined which eye is the primary eye.

[0027] Preferably, the analysis module is configured to further determine the occupant's primary eye based on the occupant's head orientation and / or head rotation direction. For example, head orientation and / or head rotation direction can be detected and analyzed in relation to the user interface module to determine which of the two eyes is directed at a specific display element and / or control element of the user interface module.

[0028] Preferably, the assistance system is configured to determine, either once (e.g., with a new driver) or repeatedly (e.g., at predetermined intervals, e.g., once a month) and / or in specific situations (e.g., at red lights), which eye is the primary eye in the event of a detected eye misalignment. In particular, it is not necessary to perform this determination continuously. Instead, the determination can be carried out selectively, stored, and subsequently used to control various vehicle functions.

[0029] Preferably, the analysis module is configured to determine a difference (e.g., a relative angle) between the first and second viewing directions. Without any eye misalignment, this difference is equal to or less than a threshold (e.g., zero).

[0030] Preferably, at least one control module is configured to control at least one vehicle function based on the difference between the first and second viewing directions. This allows for a correction of the viewing direction if, for example, only the secondary eye is captured by the camera because the head is turned to the side or a pair of glasses is obscuring the primary eye. Preferably, the at least one vehicle function includes or relates to a driver monitoring function, in particular a driver attention monitoring function.

[0031] Preferably, at least one control module is configured to issue at least one driver instruction based on the gaze direction of the primary eye. This avoids unnecessary driver instructions due to insufficient or inaccurate gaze direction detection.

[0032] Preferably, at least one driver instruction includes, or is, a warning message, in particular an attention warning.

[0033] Preferably, the at least one driver instruction includes, or is, a visual and / or acoustic and / or haptic driver instruction.

[0034] Preferably, at least one control module is configured to activate a driver assistance function based on the gaze direction of the primary eye. For example, a braking assistance system, such as an emergency braking assistant, can be activated based on the gaze direction of the primary eye. This avoids unnecessary system interventions due to insufficient or inaccurate gaze direction detection.

[0035] According to another independent aspect of the present disclosure, a vehicle, in particular a motor vehicle, is specified. The vehicle comprises the assistance system according to the embodiments of the present disclosure.

[0036] The term "vehicle" includes cars, trucks, vans, buses, motorhomes, motorcycles, etc., used for the transport of people, goods, etc. In particular, the term includes motor vehicles for passenger transport.

[0037] According to another independent aspect of the present disclosure, an assistance method for a vehicle, in particular a motor vehicle, is specified. The assistance method comprises: - Detecting, by means of an occupant detection module, an occupant of the vehicle and providing corresponding detection data; - Determine, using an analysis module, a first gaze direction of the occupant's first eye and a second gaze direction of the occupant's second eye based on the captured data; - Determine, using the analysis module, one of the occupant's primary eyes if there is a discrepancy between the first and second gaze directions; and - Control, by at least one control module, of at least one vehicle function based on a viewing direction of the primary eye.

[0038] The assistance procedure can implement the aspects of the assistance system described in this document.

[0039] According to another independent aspect of the present disclosure, a software (SW) program is specified. The SW program can be configured to run on one or more processors and thereby execute the assistance procedure for a vehicle described in this document.

[0040] According to another independent aspect of the present disclosure, a storage medium is specified. The storage medium may include a software program configured to run on one or more processors and thereby execute the assistance procedure for a vehicle described in this document.

[0041] According to another independent aspect of the present disclosure, software with program code is specified. The software is designed to carry out the assistance procedure for a vehicle when the software runs on one or more software-controlled devices.

[0042] According to another independent aspect of the present disclosure, a system is specified. The system comprises one or more processors; and at least one memory connected to the one or more processors and containing instructions that can be executed by the one or more processors to perform the vehicle assistance procedure described in this document.

[0043] A processor or processor module is a programmable computing unit, i.e., a machine or an electronic circuit that controls other elements according to given instructions and thereby advances an algorithm (process). Brief description of the drawings

[0044] Examples of the manifestation of the revelation are shown in the figures and are described in more detail below. They show: Fig. 1 schematically a vehicle with an assistance system according to embodiments of the present disclosure, Fig. 2 schematically a determination of a primary eye according to embodiments of the present disclosure, Fig. 3 schematically a determination of a primary eye according to further embodiments of the present disclosure, and Fig. 4 a flowchart of an assistance procedure for a vehicle according to embodiments of the present disclosure. Implementations of the revelation

[0045] Unless otherwise noted, the same reference symbols are used for identical and equivalent elements in the following.

[0046] Fig. Figure 1 schematically shows a vehicle 10 with an assistance system 100 according to embodiments of the present disclosure.

[0047] The assistance system 100 comprises an occupant detection module 110, which is configured to detect an occupant of the vehicle and provide corresponding detection data; an analysis module 120, which is configured to determine, based on the detection data, a first gaze direction of the occupant's first eye and a second gaze direction of the occupant's second eye, wherein the analysis module 120 is further configured to determine a primary eye of the occupant in the event of a deviation between the first gaze direction and the second gaze direction based on at least one context factor; and at least one control module 130, which is configured to control at least one vehicle function based on a gaze direction of the primary eye.

[0048] Typically, the occupant of the vehicle is the driver. However, the present disclosure is not limited to this, and it is conceivable to apply the embodiments of the present disclosure to other vehicle occupants, such as a front passenger or rear passenger.

[0049] In some embodiments, the occupant detection module 110 includes interior sensors. The interior sensors can include at least one optical sensor, such as a camera, in particular an interior camera. Optionally, the occupant detection module can include at least one light source. In some embodiments, the light source can be an infrared light source, and the at least one optical sensor can be an infrared sensor, such as an infrared camera.

[0050] Determining gaze direction can be achieved through various methods, such as observing eye movements, pupil position, and / or head orientation. In particular, gaze direction can be determined using eye tracking. The term "eye tracking" specifically refers to the recording of eye movements, which are detected, for example, by an indoor sensor camera. This allows insights to be gained into where a person is looking.

[0051] Analysis module 120 is designed to detect deviations between the first and second gaze directions, and thus eye misalignment. Eye misalignment can be detected when the first and second gaze directions deviate from parallel alignment by more than a threshold value. This ensures that only deviations that actually impair visual perception are considered, while deviations that do not affect visual perception or only minimally do so are disregarded. Preferably, the threshold is 3° or more, or 5° or more. Typically, the threshold is 5°.

[0052] Analysis module 120 is further configured to determine an occupant's primary eye based on at least one contextual factor when an eye misalignment is detected. Exemplary approaches for determining the primary eye will be described later with reference to the... Fig. 2 and Fig. 3 explained.

[0053] In some embodiments, the assistance system 100 is configured to determine, either once (e.g., with a new driver) or repeatedly, e.g., at predetermined intervals (e.g., once a month) and / or in specific situations (e.g., at red lights), which eye is the primary eye. In particular, it is not necessary to perform this determination continuously. Instead, the determination can be carried out selectively, stored, and subsequently used to control various vehicle functions.

[0054] The at least one control module 130 is configured to control at least one vehicle function based on the gaze direction of the primary eye. The at least one vehicle function can be, for example, a driver attention monitoring function or a driver assistance function; however, the present disclosure is not limited to these.

[0055] For example, at least one control module 130 can be configured to issue at least one driver instruction based on the gaze direction of the primary eye. This avoids unnecessary driver instructions due to insufficient or inaccurate gaze direction detection. Typically, the at least one driver instruction includes, or is, a warning message, in particular an attention warning.

[0056] In another example, a braking assistance system, such as an emergency braking assistant, can be controlled based on the gaze direction of the primary eye. This avoids unnecessary system interventions due to insufficient or inaccurate gaze direction detection.

[0057] Fig. Figure 2 schematically shows a determination of a primary eye according to embodiments of the present disclosure.

[0058] The illustration shows a driver whose left eye is aligned in a first direction of view, BR1, and whose right eye is aligned in a second direction of view, BR2. The directions of view BR1 and BR2 are not parallel, resulting in a misalignment of the eyes.

[0059] The driver's primary eye is determined based on a contextual factor. In the example of the Fig. 2. The context factor is a situationally relevant object, such as a traffic light 20. The situationally relevant object is typically detected by means of environmental sensors in the vehicle.

[0060] This allows the eye whose gaze is directed towards the situationally relevant object to be determined as the occupant's primary eye. (Left in the) Fig. 2 is the right eye, and to the right in the Fig. 2 the left eye.

[0061] In an illustrative example, the vehicle is stopped at traffic light 20, and the driver observes the traffic light 20. The driver slightly turns their head towards the traffic light 20 while their primary eye focuses on the signal. Simultaneously, the vehicle's external cameras detect the red traffic light 20 and estimate its position. The viewing directions of both eyes are then compared with the coordinates of traffic light 20, and the smaller delta is selected. In this way, the driver's primary eye can be determined.

[0062] Fig. Figure 3 schematically shows a determination of a primary eye according to further embodiments of the present disclosure.

[0063] The illustration shows a driver whose left eye is aligned in a first direction of view, BR1, and whose right eye is aligned in a second direction of view, BR2. The directions of view BR1 and BR2 are not parallel, resulting in a misalignment of the eyes.

[0064] The driver's primary eye is determined based on a contextual factor. In the example of the Fig. 3 is the context factor a user interface module, such as a touchscreen 30 with a control element 32.

[0065] This allows the eye whose gaze is directed towards control element 32 to be determined as the occupant's primary eye. (Left in the) Fig. 3 is the right eye, and to the right in the Fig. 3 the left eye.

[0066] In an illustrative example, the driver operates the control element 32 in the interior, whereby the gaze direction of both eyes is recorded and compared with the known coordinates of the control element 32. The slight deviation between the gaze directions is used to determine the primary eye. A particularly advantageous aspect of operating control elements is the precise determination of the moment the driver looks at the display. This is especially effective with touchscreens, as the objects on the display have exact coordinates, thus enabling precise detection.

[0067] Fig. Figure 4 schematically shows a flowchart of an assistance method 400 for a vehicle according to embodiments of the present disclosure. The assistance method 400 can be implemented by appropriate software that can be executed by one or more processors (e.g., a CPU).

[0068] According to the invention, the gaze directions of both of the occupant's eyes are recorded separately in order to detect any eye misalignment. If such a misalignment is detected, a primary eye—that is, the eye predominantly responsible for visual perception and information processing—is determined by taking contextual factors into account. For example, the contextual factor could be a red traffic light, with the eye whose gaze is directed towards the red light being identified as the primary eye. Such precise detection of the gaze direction enables, for example, improved driver attention monitoring, which in turn can increase road safety.

[0069] The assistance procedure 400 comprises, in block 410, the detection of a vehicle occupant by an occupant detection module and the provision of corresponding detection data; in block 420, the determination by an analysis module of a first gaze direction of the occupant's first eye and a second gaze direction of the occupant's second eye based on the detection data; in block 430, the determination by the analysis module of a primary eye of the occupant if there is a deviation between the first gaze direction and the second gaze direction; and in block 440, the control by at least one control module of at least one vehicle function based on a gaze direction of the primary eye.

[0070] Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Images