Controlling a parking assistance system of a vehicle on the basis of gaze behavior

EP4466176A1Pending Publication Date: 2024-11-27MERCEDES BENZ GROUP AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
EP2024710383
Authority / Receiving Office
EP · EP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-11
Filing Date
2024-03-07
Publication Date
2024-11-27

AI Technical Summary

Technical Problem

Modern parking assistance systems in vehicles often mistakenly offer parking options due to driver gaze behavior misinterpretation, especially in slow city traffic, leading to driver irritation and unreliable intention detection.

Method used

A system using a computing unit and driver observation camera to analyze gaze behavior by distinguishing between fixations and saccades, projecting gaze directions onto the road environment, and excluding road-related gazes to accurately detect the driver's intention to park, thereby activating the automatic parking aid only when a genuine intention is recognized.

Benefits of technology

Improves the reliability of detecting the driver's intention to park, reducing false positives and enhancing the human-machine interface by using existing vehicle components without additional sensors, ensuring accurate and efficient activation of the automatic parking aid.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2024055993_17102024_PF_FP_ABST
    Figure EP2024055993_17102024_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a system for controlling an automatic parking aid of a vehicle (1), comprising a computing unit (3) and a driver observation camera (5) which is used to capture gaze behavior of the driver, wherein the computing unit (3) is designed to infer an intention to park the vehicle (1) in a parking space from determined patterns of fixations and saccades in the gaze behavior of the driver and to activate the automatic parking aid when an intention is recognized. The computing unit (3) is also designed to determine a planar road course (7), to assign gaze directions of the driver to the current surroundings of the vehicle (1), and to exclude, from an examination of the patterns in the gaze behavior for an intention to park, fixations with a gaze direction at the planar road course (7) and / or saccades with gaze directions over at least part of the planar road course (7).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Gaze behavior-based parking assistance system control of a vehicle

[0002] The invention relates to a system for controlling an automatic parking aid of a vehicle, as well as a vehicle with such a system.

[0003] The following information does not necessarily arise from a specific document representing the state of the art, but rather from practical considerations and practical experience: Modern vehicles often have a parking assistance system that is typically capable not only of virtually marking parking spaces for the driver of the vehicle, but, particularly in modern passenger cars, of independently parking the vehicle in a selected parking space. To ensure that the driver of the vehicle knows exactly where the parking space for automated parking is located, the respective space must typically be marked accordingly; in this case, it is advisable to offer the driver a variety of possible parking spaces to choose from. First, however, it must be determined whether the driver of the vehicle even wants to park the vehicle.For example, if a large number of possible parking spaces are presented to the driver, particularly visually, below a certain speed threshold, this can lead to undesired behavior on the part of the vehicle. Parking spaces can be marked and presented even when the driver has no intention of parking, for example, when the vehicle is traveling in a constantly changing but slow speed range in heavy city traffic. This would irritate the driver. In the prior art, it is known to automatically analyze the driver's gaze behavior and, based on this, to use parking systems accordingly.

[0004] DE 102012 221 036 A1 relates to a method for automatically activating a parking and / or maneuvering assistance system in a motor vehicle, comprising the steps of: determining the driver's gaze behavior; evaluating the gaze behavior; and automatically activating the parking and / or maneuvering assistance system based on the evaluation of the gaze behavior. In particular, DE 10 2012 221 036 A1 determines fixations with a minimum fixation duration and saccades based on the driver's gaze behavior.

[0005] The object of the invention is to improve the use of a parking assistance system depending on the detected gaze behavior of a driver of a vehicle.

[0006] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0007] A first aspect of the invention relates to a system for controlling an automatic parking aid of a vehicle, comprising a computing unit and a driver observation camera connected to the computing unit, wherein the driver observation camera serves to detect a gaze behavior of the driver of the vehicle and the computing unit is designed to infer an intention to park the vehicle in a parking space from certain patterns of fixations and saccades in the driver's gaze behavior and, upon detection of the intention, to at least partially activate the automatic parking aid, characterized in that the computing unit is designed to determine a planar road course of a road section lying in front of the vehicle in the direction of travel, to assign the driver's gaze directions to the respective current surroundings of the vehicle,and to exclude from a review of patterns in gaze behavior for an intention to park fixations with a gaze direction towards the planar road course and / or saccades with gaze directions over at least partially the planar road course.

[0008] To detect an intention to park using corresponding patterns, only fixations in gaze behavior above a predefined minimum fixation duration are preferably counted. Shorter fixations are more likely to indicate scanning of the surroundings in order to form a spatial image of the environment. This is typical human behavior when perceiving the environment, as the point of sharpest vision is limited to a narrow angular range and thus a large number of eye movements are necessary to scan the surroundings in order to mentally piece together the numerous detailed sections of the environment. In contrast, a fixation above the predefined minimum fixation duration is more likely to indicate that a parking space that the driver of the vehicle is looking at with his gaze for longer than the minimum fixation duration is viewed as a potential parking space.

[0009] In contrast to the fixations in the driver's gaze, saccades are rapid eye movements that are typically performed by both eyes simultaneously and occur between two phases of fixation. Because the speed of the eye movements exceeds the range typical for humans during saccades, correspondingly less detailed information about the environment is perceived.

[0010] The driver's gaze behavior is recorded using the driver observation camera. This is used in particular to detect the driver's eye movements and gaze directions. The driver observation camera is advantageously positioned within the vehicle interior so that its detection area is directed toward the driver's head area. With appropriate eye tracking modules (so-called "eye trackers"), it is then possible to continuously observe the driver's gaze behavior and analyze it with the computing unit. Fixations and saccades are advantageously determined from the recorded gaze behavior only through appropriate image analysis by the computing unit, and in particular, the driver's gaze is classified into exclusively one of the categories of fixations and saccades to facilitate pattern recognition.

[0011] The processing unit is specifically designed to check the sequence of fixations and saccades for patterns in which saccades and fixations occur in rapid succession. However, further criteria can be specified in the requirements for the gaze behavior patterns, in particular certain geometric trajectories of the saccades and thus the relative positions of the fixations to one another.

[0012] However, according to the invention, this pattern recognition does not take into account the driver's gaze behavior at all times, meaning that all fixations and saccades in the driver's gaze behavior are not evaluated; instead, gazes directed at the section of road in front of the vehicle are disregarded. If the pattern recognition is carried out in such a way that the driver's gaze and eye movements recorded by the driver observation camera are projected onto the area around the vehicle, the area around the road section ahead and, preferably, other possible roads to be traveled are masked out of this projection, and gazes directed towards those areas are disregarded. In particular, the image from the driver observation camera is thus compared with an image of the course of the road or the surrounding area projected onto it.with sections of the road transformed into a coordinate system of the driver observation camera masked; in other words, the corresponding image of the road section is masked out of the driver observation camera image. This means that the streets are known in the driver observation camera coordinate system and act as a filter or

[0013] Mask to mask out the saccades / fixations in this area. These streets are each treated as a flat road surface, i.e. an elongated extension along the lanes on the road surface as well as the width of the road are taken into account, which in combination describe the surface of the road. It is assumed that the driver's glances at this road surface do not coincide with an intention to park there. For this reason, this road surface is expediently linked to the fixations and saccades in the driver's gaze behavior using appropriate projections and / or coordinate system transformations in such a way that glances directed at the flat road surface in the form of fixations or saccades that at least partially, or in particular only, cross this surface cannot be interpreted as an intention to park and are therefore excluded from the analysis of gaze behavior.

[0014] If such a pattern is detected, which, according to the specified criteria, indicates the driver's intention to park, the computer unit activates the automatic parking assistance. Activating the parking assistance preferably means that the driver is presented with a variety of possible parking spaces to choose from. After the selection has been made and the request for parking assistance has been confirmed (optionally automatically) by an automated control system of the vehicle, the vehicle can park independently or at least with assistance in the selected parking space. This is particularly advantageous in tight parking spaces.

[0015] An advantageous effect of the invention is that physical devices typically installed in modern vehicles, such as an interior camera for driver observation (functioning as a driver observation camera) and a computing unit with sufficient computing power for image analysis, can already be used without requiring additional devices. Only an adaptation on the software side is necessary to achieve the improvement of the system for controlling an automatic parking aid. Thus, the human-machine interface can be improved depending on customer needs. If the vehicle's position is also determined using satellite technology and the road section in front of the vehicle is determined using a pre-stored digital map, no external sensors on the vehicle are necessary.The vehicle driver's intention recognition when searching for a parking space will also be improved and made significantly more reliable, as false positive detections of parking intentions by the driver will be better prevented. Such false detections could easily occur without the masking of the road ahead if the vehicle is in a traffic jam, for example on the motorway, where the driver is tracking the lane markings applied at long intervals on the road, or if road boundaries such as fences, plants, emergency call boxes, warning beacons or other objects are being tracked on winding roads, in particular, or if in so-called "stop-and-go" traffic, particularly in city traffic, a high level of driver attention is required and the driver would therefore, for example, alternately observe a road user ahead and lane markings at short intervals.

[0016] According to an advantageous embodiment, the computing unit is designed to determine the planar road course of the road section in the direction of travel in front of the vehicle by determining the current position of the vehicle and by assigning the determined position to a digital map with stored road courses.

[0017] The current position of the vehicle is preferably determined using satellite-based positioning, but can be supported by additional sensors, such as inertial sensors (preferably a compass), environmental detection, and others. Preferably, the vehicle's orientation is also determined during the position determination process. This allows the actual alignment relative to the road section ahead to be determined not only based on the map data (which implies the assumption of an expected alignment on the road), but also to take into account vehicle inclinations, such as when avoiding potholes or other obstacles, as well as individual curve and lane guidance.

[0018] According to a further advantageous embodiment, the computing unit is designed to determine the position angle of the vehicle relative to the planar road course of the road section based on the road course according to the digital map and / or by means of an inertial measuring unit.

[0019] According to a further advantageous embodiment, the system further comprises an environmental observation camera connected to the computing unit for detecting an area in front of the vehicle, wherein the computing unit is designed to determine a vector field of an optical flow of the environment moving relative to the moving vehicle from current data of the environmental observation camera, and to take into account a relationship of the gaze behavior to the vector field of the optical flow when checking patterns in the gaze behavior for an intention to park.

[0020] According to this embodiment, it is taken into account that, in particular, a rapid sequence of consecutive fixations and saccades alternating along an optical flow of objects in the environment indicates the intention to park, since the driver's gaze then wanders from one potential parking space to the next, and due to the driver's own driving speed, the optical flow causes the sequence of fixations and saccades to occur on average with a certain minimum frequency, depending on the driving speed; in this respect, the driving speed and thus the optical flow of the environment relative to the driver's own vehicle and its effects on the driver's gaze behavior when searching for a parking space are taken into account in the pattern recognition.

[0021] According to a further advantageous embodiment, the computing unit is designed to determine values ​​of at least one of the parameters "orientation", "repeatability" and "connection" by analyzing the saccades, wherein the "orientation" parameter indicates a respective orientation of the saccades relative to the direction vectors of the vector field of the optic flow, wherein the "repeatability" parameter characterizes a sequence of the saccades among each other, and wherein the "connection" parameter indicates the length of the individual saccades, and wherein the computing unit is designed to infer an intention to park depending on the respectively determined values ​​of the at least one parameter.

[0022] The "Orientation" parameter specifies the respective orientation of the saccades relative to the direction changes of the optic flow vector field. For comparison, the nearest optical flow orientations are used here. The "Repeatability" parameter, on the other hand, characterizes the sequence of saccades. For example, the saccades follow a line or a zigzag pattern is recognizable. The "Connection" parameter can be used to check whether saccades span the length of a parking space or are caused by road boundaries such as a fence or barrier. These parameters are therefore particularly suitable for characterizing the shape of the saccades, especially with regard to the determined optical flow of the environment relative to the vehicle.

[0023] According to a further advantageous embodiment, the computing unit is designed to determine the respective value of the at least one parameter by means of a respective histogram, wherein the respective histogram indicates occurrence frequencies or occurrence frequencies of the individual values ​​of the parameters in order to form parameter mean values ​​as the final value of the respective parameter from the respective histogram.

[0024] For example, to suppress false positives along a horizontal ladder, etc., the search primarily seeks values ​​that encompass the typical width of parking spaces. These values ​​are preferably derived repeatedly for each of a number n of frames from the driver observation camera. For each of a number m of frames, three histograms (for the three parameters) are preferably created on this basis. These histograms define a range relevant for detecting the driver's intention, particularly based on previous application drives. The frequency is averaged and evaluated within this range.

[0025] According to a further advantageous embodiment, the computing unit is designed to determine a weighted sum of the final values ​​of the parameters "orientation", "repeatability", and "connection" and to compare the result of the sum with a predetermined limit value and to conclude an intention to park if the limit value is exceeded.

[0026] According to a further advantageous embodiment, the computing unit is designed to adapt weighting factors of the weighted sum and / or the limit value based on learned field data.

[0027] The weighting factors of the weighted sum of the parameters can be determined through calibration runs or adjusted based on customer behavior in the field. The limit value can also be adaptively adjusted during subsequent regular vehicle operation, particularly by recording a history of the parameters and determining a limit value based on this during manual parking maneuvers.

[0028] According to a further advantageous embodiment, the computing unit is designed to carry out the adaptation differently for individual drivers.

[0029] A further aspect of the invention relates to a vehicle with a system as described above and below.

[0030] Advantages and preferred developments of the proposed vehicle result from an analogous and analogous transfer of the statements made above in connection with the proposed system.

[0031] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0032] It shows:

[0033] Fig. 1 A system for activating an automatic parking aid after driver observation in a vehicle according to an embodiment of the invention and

[0034] Fig. 2 optical flow and exemplary saccades 2, 2'

[0035] The representations in the figures are schematic and not to scale.

[0036] Fig. 1 shows a system for conditionally activating an offer to execute an automatic parking assistance system of a vehicle 1. The system has a computing unit 3 and a driver observation camera 5 connected to the computing unit 3 in the interior of the vehicle 1. This driver observation camera 5 has its detection area directed towards a headrest and thus towards a head of the driver of the vehicle 1 and serves to detect the gaze behavior of the driver of the vehicle 1. Eye movements performed by the driver of the vehicle 1, in particular so-called saccades 2, as well as fixations of objects in the surroundings of the vehicle, during which the driver looks at an object in the surroundings of the vehicle for a certain period of time, can thus be detected by the driver observation camera 5 and transferred to the computing unit 3 for analysis.The computing unit 3 is further designed to infer an intention to park the vehicle 1 in a potential parking space 4 from certain sequence patterns of fixations and saccades 2, 2' in the driver's gaze behavior, see Fig. 1 and Fig. 2. For this purpose, an environment observation camera connected to the computing unit 3 is also used to capture an environment in front of the vehicle 1. The computing unit 3 of the vehicle 1 uses its data to determine a vector field of an optical flow 6a, 6b shown in Fig. 2 of the environment moving relative to the moving vehicle 1. Fig. 2 shows the optical flow in the areas relevant for parking on the left side of the road 6a and the right side of the road 6b from Fig. 1, which is extracted by the computing unit.

[0037] Furthermore, a relationship between gaze behavior and the vector field of optic flow 6a is considered when examining patterns in gaze behavior as follows: By analyzing the saccades (see example saccades 2, 2' in Figure 2), final values ​​of the parameters "orientation," "repeatability," and / or "connectivity" are determined. The "orientation" parameter indicates the respective orientation of the saccades relative to the direction vectors of the vector field of optic flow. The orientation is preferably determined as the angle of the saccades 2, 2' between the fixations represented as points and represented by vector arrows.

[0038] The parameter "repeatability" characterizes a sequence of saccades among each other, ie preferably represented by angles of the vector arrows to each other, for example the saccades follow a line or a zigzag pattern.

[0039] The "connection" parameter specifies the length of the individual saccades. To exclude, for example, false positives along a horizontal ladder, etc., we primarily search for values ​​that encompass the common dimensions of parking spaces. The aforementioned parameters are repeatedly derived for each frame of the driver observation camera 5. The saccades 2' shown in Fig. 2, for example, exhibit small angular deviations from the optical flow 6a; at the same time, the saccades 2' exhibit small angular deviations from each other. If the length of the saccades 2' correlates with the corresponding parking space dimensions, it can be unequivocally concluded that the user is parking and is looking for a parking space. In contrast, the values ​​also shown in Fig.1 corresponding saccades 2 are disorganized, deviate from the optical flow 6a and have different dimensions that do not correlate with the parking space dimensions, so that a parking intention by the user can be excluded.

[0040] For evaluation, the final values ​​of the parameters are determined by the computing unit 1 using a respective histogram from the continuous frames. The respective histogram indicates the occurrence frequencies of individual values ​​or value classes of the parameters determined in a history. In order to calculate parameter mean values ​​from the respective histogram as the respective final value of the respective parameter, for example, by multiplying the parameter values ​​of a class or their mean by the respective frequencies, summing them, and dividing them by the total number of parameter values. After determining a respective final value of a respective parameter, the computing unit 3 calculates a weighted sum with weighting factors for each of the final values ​​of these parameters "orientation," "repeatability," and "connection," and compares the result of the weighted sum with a specified limit.As long as the limit value lies within a given range on this side of a given threshold, e.g., if the threshold is exceeded, an intention to park is inferred, preferably if the saccades with this characteristic occurred for a certain minimum duration. In a simplified version, final values ​​can be calculated from the histograms for only one or two of the parameters and calculated using weighting factors.

[0041] However, this analysis of fixations and saccades to determine whether the driver of the vehicle has an intention to park is only carried out for those fixations that are directed at the surroundings of the vehicle 1 outside of a flat road course 7 of a road section lying in front of the vehicle 1 in the direction of travel, and only for those saccades that at least partially, in particular completely, only cover areas of the surroundings of the vehicle 1 outside of the flat road course 7. The flat road course describes a surface of at least one drivable road in the surroundings ahead of the vehicle 1. Side streets can also be included in the flat road course. The flat road course is therefore determined, on the one hand, by the elongated road course in a respective direction of travel, which includes straight sections, curves, etc.includes, and on the other hand, on the respective width at the respective location of the elongated road course, wherein the width is indicated in Fig. 1 by two dashed boundary lines of a two-lane road with a center marking. The assignment of the viewing directions of the fixations into the surroundings of the vehicle 1 or the areas of the surroundings of the vehicle 1 covered by the saccades on the basis of the fixations and saccades in the driver's gaze behavior determined by the driver observation camera 5 is carried out by the computing unit 1 through coordinate system transformations.For this purpose, a position and orientation in the form of attitude angles of the vehicle 1 relative to the road are determined by determining a respective current position of the vehicle 1 by means of satellite-supported positioning, and the immediately ahead course of the road with its respective width is determined from information from a digital map, wherein the coordinate system transformations are based on the attitude angles, which represent the only transformation variable, since it can be assumed that the driver observation camera 5 is arranged in a body-fixed constant manner relative to the rest of the vehicle 1.If the intention is detected, the automatic parking assistance is activated to the extent that the driver of the vehicle is shown possible parking spaces marked on a head-up display on the windshield in front of the driver's seat using augmented reality. Thus, by selecting one of these available parking spaces, the driver of the vehicle can initiate the automatic parking of vehicle 1 in this selected parking space by issuing a command, for example, a voice input or other control action. Alternatively or additionally, the driver is offered a visualization of an activation option for an automatic parking function on an instrument cluster, a switch, or a head unit.

[0042] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims

Patent claims 1. A system for controlling an automatic parking aid of a vehicle (1), comprising a computing unit (3) and a driver observation camera (5) connected to the computing unit (3), wherein the driver observation camera (5) serves to detect a gaze behavior of the driver of the vehicle (1), and the computing unit (3) is designed to infer an intention to park the vehicle (1) in a parking space from certain patterns of fixations and saccades in the driver's gaze behavior and, upon detection of the intention, to at least partially activate the automatic parking aid, characterized in that the computing unit (3) is designed to determine a planar road course (7) of a road section located in front of the vehicle (1) in the direction of travel, to assign the driver's gaze directions to the respective current surroundings of the vehicle (1),and to exclude from a review of the patterns in gaze behavior for an intention to park fixations with a gaze direction towards the planar road course (7) and / or saccades with gaze directions over at least partially the planar road course (7).

2. System according to claim 1, wherein the computing unit (3) is designed to determine the planar road course (7) of the road section in the direction of travel in front of the vehicle (1) by determining the current position of the vehicle (1) and by assigning the determined position to a digital map with stored road courses.

3. System according to claim 2, wherein the computing unit (3) is designed to determine the position angle of the vehicle (1) relative to the planar road course (7) of the road section based on the road course (7) according to the digital map and / or by means of an inertial measuring unit.

4. System according to one of the preceding claims, further comprising an environmental observation camera connected to the computing unit (3) for detecting an area in front of the vehicle (1), wherein the computing unit (3) is designed to determine a vector field of an optical flow of the environment moving relative to the moving vehicle (1) from current data of the environmental observation camera, and to take into account a relationship of the gaze behavior to the vector field of the optical flow when checking patterns in the gaze behavior for an intention to park.

5. System according to claim 4, wherein the computing unit (3) is designed to determine values ​​of at least one of the parameters: "orientation", "repeatability" and "connection" by analyzing the saccades, wherein the "orientation" parameter indicates a respective orientation of the saccades relative to the direction vectors of the vector field of the optical flow, wherein the "repeatability" parameter characterizes a sequence of the saccades among each other, and wherein the "connection" parameter indicates the length of the individual saccades, and wherein the computing unit (3) is designed to infer an intention to park depending on the respectively determined values ​​of the at least one parameter.

6. System according to claim 5, wherein the computing unit (3) is designed to determine the respective value of the at least one parameter by means of a respective histogram, wherein the respective histogram indicates occurrence frequencies or occurrence frequencies of the individual values ​​of the parameters in order to form parameter mean values ​​as the final value of the respective parameter from the respective histogram.

7. System according to claim 6, wherein the computing unit (3) is designed to determine a weighted sum of the final values ​​of the parameters "orientation", "repeatability", and "connection" and to compare the result of the sum with a predetermined limit value and to conclude an intention to park if the limit value is undershot.

8. System according to claim 7, wherein the computing unit (3) is designed to adapt weighting factors of the weighted sum and / or the limit value based on learned field data.

9. System according to claim 8, wherein the computing unit (3) is designed to carry out the adaptation differently for individual drivers.

10. Vehicle (1) with a system according to one of the preceding claims.