A system and vehicle for controlling automatic parking assistance.

Abstract

The present invention relates to a system for controlling automatic parking assistance for a vehicle (1), comprising a computing unit (3) and a driver observation camera (5) used to detect the driver's gaze behavior, wherein the computing unit (3) is designed to estimate the driver's intention to park the vehicle (1) in a parking space from specific patterns of gaze and saccades in the driver's gaze behavior, and to activate automatic parking assistance when such intention is recognized. The computing unit (3) is further designed to determine a planar road layout (7), assign the driver's gaze direction to the current surrounding environment of the vehicle (1), and to exclude gaze direction gaze toward the planar road layout (7) and / or gaze direction saccades at least partially across the planar road layout (7) from the examination of patterns in gaze behavior regarding the intention to park.

Description

Technical Field

[0001] The present invention relates to a system for controlling automatic parking assistance of a vehicle and a vehicle equipped with such a system.

Background Art

[0002] The following information is not necessarily obtained from specific prior art documents, but is obtained from practical considerations and practical experiences. Recently, vehicles can often not only virtually show the driver of the vehicle the available parking spaces, but especially recent passenger cars have a parking assistance system that can also autonomously park the vehicle in the selected available parking space. In order for the driver of the vehicle to clearly recognize where the parking space for automatic parking is, usually each parking space needs to be correspondingly indicated, and in this case, it is recommended that the driver be presented with options for a plurality of possible parking spaces. However, first, it is necessary to confirm whether the driver of the vehicle actually wants to park the vehicle in the first place. For example, if a large number of possible available parking spaces are particularly visually presented to the driver when the vehicle speed drops below a certain speed threshold, this may lead to undesirable behavior of the vehicle. For example, when the vehicle is traveling in a constantly changing low-speed area in a congested urban area, even if the driver has no intention of executing the parking process, there is a possibility that an available parking space will be presented. This will irritate the driver. In this regard, in the prior art, it is known to automatically analyze the driver's line-of-sight behavior, evaluate the line-of-sight behavior, and then use the parking system accordingly.

[0003] German Patent Application Publication No. 102012221036 relates to a method for automatically turning on a parking and / or steering assistance system of an automobile, comprising the steps of determining the gaze behavior of the driver, evaluating the gaze behavior, and automatically turning on the parking assistance system and / or steering assistance system in accordance with the evaluation of the gaze behavior. Specifically, German Patent Application Publication No. 102012221036 particularly describes how minimum gaze time and saccades are determined based on the driver's gaze behavior. [Overview of the project] [Problems that the invention aims to solve]

[0004] The objective of the present invention is to improve the use of a parking assist system in response to the detected gaze behavior of the vehicle driver. [Means for solving the problem]

[0005] The present invention is evident from the features of the independent claims. Advantageous developments and embodiments are the subject of the dependent claims.

[0006] A first aspect of the present invention relates to a system for controlling automatic parking assistance for a vehicle, comprising a computing unit and a driver observation camera connected to the computing unit, wherein the driver observation camera is used to detect the gaze behavior of the vehicle's driver, and the computing unit is designed to estimate the intention to park the vehicle in a parking space from specific patterns of gaze and saccades in the driver's gaze behavior, and to activate automatic parking assistance at least partially when such intention is recognized, wherein the computing unit is designed to determine the planar road layout of a road section located in front of the vehicle's direction of travel, to assign the driver's gaze direction to the vehicle's current surrounding environment, and to exclude gaze in the direction toward the planar road layout (flaechige Strassenverlauf) and / or gaze in the direction toward the planar road layout at least partially from the examination of patterns in the gaze behavior regarding the intention to park.

[0007] In this case, it is preferable that only gazes exceeding a predetermined minimum gaze time in eye movement behavior be evaluated in order to recognize the intention to park based on the corresponding pattern. Shorter gaze times more strongly suggest that the driver is exploring the surrounding environment to obtain an image of the surrounding space. This is typical human behavior when perceiving the surrounding environment, as the most clearly visible parts are limited to a relatively narrow angular range, and therefore, numerous eye movements are required to explore the surrounding environment in order to mentally assemble the numerous detailed parts of the perceived surrounding environment. In contrast, gazes exceeding a predetermined minimum gaze time suggest that the parking space that the driver looks at for longer than the minimum gaze time is considered a potential available parking space.

[0008] In contrast to fixation in a driver's gaze behavior, saccades are rapid eye movements that are usually performed simultaneously by both eyes and occur between two fixation phases. Because the speed of eye movements in saccades exceeds the range of typical human eye movements, information about the surrounding environment is perceived in less detail.

[0009] The driver's gaze behavior is detected using a driver observation camera. This is used specifically to detect the driver's eye movements and gaze direction. In this case, it is advantageous that the driver observation camera is positioned in the vehicle cabin so that its detection area is directed towards the driver's head area. In this case, it is possible to continuously observe the driver's gaze behavior using a corresponding gaze tracking module (a so-called "eye tracker") and analyze it in a computing unit. Advantageously, from the detected gaze behavior, the computing unit first determines fixation and saccades through corresponding image analysis, and to facilitate pattern recognition, the driver's gaze is assigned to only one of the categories consisting of fixation and saccades.

[0010] The computational unit is designed to examine sequences of gaze and saccades, particularly those with rapid alternations between saccades and gazes. However, further criteria can be set regarding the pattern requirements in gaze behavior, especially the specific geometric transitions of saccades and, along with the relative positions of gazes.

[0011] According to the present invention, in this pattern recognition, gaze behavior is not continuously considered, and therefore, not all gazes and saccades in the driver's gaze behavior are evaluated; rather, gazes directed towards the road section ahead of the vehicle are not considered. When pattern recognition is performed such that the driver's gaze and eye movements detected by the driver observation camera are projected onto the surrounding environment around the vehicle, the area around the road section located ahead, preferably other drivable road areas, is made invisible from this projection, and gazes directed thereto are not considered. Therefore, in particular, the image from the driver observation camera is masked by the area transformed into this projected road layout image, or the area of ​​the road layout transformed into the coordinate system of the driver observation camera; in other words, the correspondingly relevant image of the road section is made invisible from the image from the driver observation camera. Thereafter, the road path is displayed in the driver observation camera coordinate system and functions as a filter or mask for masking saccades / gazes in this area. These road contours are considered as planar road layouts, namely as elongated extensions along the lanes on the road layout and as road widths, which are combined to represent the road surface. The premise is that a driver's line of sight directed towards this road surface does not coincide with an intention to park there. Therefore, relating this road surface to gaze and saccades in the driver's line of sight behavior through corresponding projections and / or coordinate system transformations helps to achieve the objective. In this process, lines of sight in the form of gaze directed towards the planar road layout, or saccades that pass at least partially, especially through, the planar road layout, should not be interpreted as an intention to park and are therefore excluded from the analysis of line of sight behavior.

[0012] If a pattern indicating the driver's intention to park is recognized according to predetermined criteria, the calculation unit activates automatic parking assistance. Activation of parking assistance preferably means that a number of possible parking spaces are presented to the vehicle driver for selection, and after a selection is made and the request for parking assistance is approved by the vehicle's automatic control system (optionally automatically), the vehicle is automatically, or at least assisted, parked in the selected parking space. This is particularly advantageous when parking spaces are limited.

[0013] One of the advantageous effects of the present invention is that physical equipment typically installed in modern vehicles, such as an in-vehicle camera for driver observation (which functions as a driver observation camera) and a computing unit with sufficient computing power for image analysis, can be used without the need for additional equipment. Only software adjustments are required to achieve improvements to the system for controlling automatic parking assistance. Therefore, the human-machine interface can be improved according to customer needs. Furthermore, if vehicle positioning is performed on a satellite basis and the road layout of the road section ahead of the vehicle's direction of travel is determined using a pre-stored digital map, an external sensor system is unnecessary. In addition, false-positive perception of the driver's intention to park is appropriately prevented, improving the recognition of the vehicle driver's intention to search for a parking space and significantly increasing reliability. For example, when a vehicle is caught in traffic congestion on a highway, and the driver is following lane markings that are widely spaced on the road, or when driving along road boundaries such as fences, hedges, emergency call boxes, and warning lights, especially on roads with many curves, or when a high level of attention is required from the driver in so-called "stop-and-go" traffic conditions, particularly in urban areas, and therefore the driver is observing, for example, the road users ahead and the lane markings at short intervals, such misperceptions are likely to occur unless the road layout ahead is hidden.

[0014] In a favorable embodiment, the computing unit is designed to determine the planar road layout of a road section located ahead of the vehicle's direction of travel by identifying the vehicle's current position, which is constantly being updated, and assigning the determined position to a digital map in which the road layout is stored.

[0015] The current position of each vehicle is preferably determined by satellite positioning, but it can also be based on other sensor systems such as inertial sensors (particularly preferably a compass) and ambient environment detection. Preferably, the orientation of the vehicle is also determined during the positioning process, thereby determining its actual orientation relative to the road section ahead. This allows for consideration of the vehicle's tilting posture, such as when avoiding potholes or other obstacles in the road, as well as individual curve driving and lane guidance, rather than relying solely on map data (meaning assuming the expected orientation on the road).

[0016] In a more advantageous embodiment, the calculation unit is designed to determine the position angle of a vehicle relative to the planar road layout of a road section based on a digital map and / or by an inertial measurement unit.

[0017] According to another advantageous embodiment, the system further comprises an ambient environment observation camera connected to a computing unit to detect the area in front of the vehicle. The computing unit is designed to determine a vector field of optical flow of the ambient environment moving relative to the moving vehicle from the current data of each ambient environment observation camera, and to consider the relationship between the optical flow vector field and the gaze behavior when examining patterns in gaze behavior regarding the intention to park.

[0018] In this embodiment, a rapid sequence of alternating gazes and saccades, particularly along the optical flow of objects in the surrounding environment, is considered to suggest an intention to park. This is because the driver's gaze moves from one potential parking space to the next, and the optical flow resulting from the vehicle's speed causes a sequence of gazes and saccades to occur at a specific minimum frequency on average, depending on the speed. In this case, pattern recognition takes into account the influence of the speed, and consequently the optical flow of the surrounding environment relative to the vehicle, and the driver's gaze behavior when searching for a parking space.

[0019] In a more advantageous embodiment, the computing unit is designed to determine the value of at least one of the parameters “orientation,” “reproducibility,” and “connection (Verbindung)” by analyzing the saccades. The parameter “orientation” indicates the orientation of each saccade with respect to the direction vector of the optical flow vector field. The parameter “reproducibility” characterizes the continuity of the saccades with respect to each other. The parameter “connection” indicates the length of the individual saccades. The computing unit is designed to estimate the intention to park, depending on the determined value of each of the at least one parameter.

[0020] The parameter "orientation" indicates the orientation of each saccade in relation to changes in the direction of the optical flow vector field. For comparison, the nearest orientation of the optical flow is used here. In contrast, the parameter "reproducibility" characterizes the continuity of the saccades with respect to each other. For example, it can be recognized that the saccades follow a straight line or extend in a zigzag pattern. The parameter "coupling" allows for the determination of whether the saccade includes the length of the parking space or, in some cases, is caused by a road boundary such as a fence / barrier. Therefore, the above parameters are particularly suitable for characterizing the shape of the saccade with respect to the determined optical flow of the surrounding environment to the vehicle.

[0021] In another advantageous embodiment, the computing unit is designed to determine the value of each of at least one parameter using its respective histogram, each histogram showing the frequency or occurrence of individual values ​​of the parameter, and the parameter mean is calculated from each histogram as the final value of each parameter.

[0022] For example, to suppress false positives along a horizontal ladder, values ​​are primarily obtained that include the general width of the parking space. These values ​​are preferably repeatedly derived for each of the n frames of the driver observation camera. For all m frames, three histograms are preferably obtained based on these values ​​(for the three parameters). In these histograms, regions related to detecting the driver's intent are identified, particularly based on prior application driving. In these regions, the frequency is averaged and evaluated.

[0023] In another advantageous embodiment, the computing unit is designed to calculate a weighted sum of the final values ​​of the parameters “orientation,” “reproducibility,” and “joint,” compare the result of the sum to a predetermined limit, and estimate the intention to park if it falls below the limit.

[0024] According to another advantageous embodiment, the computing unit is designed to adjust the weight coefficients of the weighted sum and / or limit values ​​based on the learned field data.

[0025] The weighting coefficients for the weighted sum of parameters can be determined by application driving or adjusted on-site based on customer behavior. Limit values ​​can also be adaptively adjusted in subsequent normal vehicle operation, particularly by recording the parameter history and determining the limit values ​​during manual parking operations based on this history.

[0026] According to another advantageous embodiment, the computing unit is designed to perform adjustments in different ways for individual drivers.

[0027] Another aspect of the present invention relates to a vehicle equipped with the system described above and described below.

[0028] Advantageous and preferred developments of the proposed vehicle are obtained in the same way and by an application consistent with the meaning of the explanations made above in connection with the proposed system.

[0029] Other advantages, features, and details will become apparent from the following description in which at least one embodiment is described in detail, with reference to the drawings as necessary. Identical, similar, and / or functionally identical parts are designated by the same reference numerals.

Brief Description of the Drawings

[0030] [Figure 1] It is a diagram of a system for activating automatic parking assistance after observing a driver in a vehicle according to an embodiment of the present invention. [Figure 2] It is a diagram of optical flow and exemplary saccades 2, 2'.

Mode for Carrying Out the Invention

[0031] The figures are schematic and not to scale.

[0032] Figure 1 shows a system that conditionally activates an offer to perform automatic parking assistance for vehicle 1. This system comprises a computing unit 3 and a driver observation camera 5 connected to the computing unit 3 inside the vehicle 1. The driver observation camera 5 has its detection area directed towards the headrest, and therefore towards the driver's head in vehicle 1, and is used to detect the driver's gaze behavior in vehicle 1. Thus, eye movements performed by the driver of vehicle 1, in particular the so-called saccade 2 and fixation on objects in the vehicle's surrounding environment, where the driver looks at an object in the vehicle's surrounding environment for a specific period of time, can be detected by the driver observation camera 5 and transmitted to the computing unit 3 for analysis. The computing unit 3 is further designed to estimate the intention to park vehicle 1 in a potential parking space 4 from the fixation in the driver's gaze behavior and specific sequence patterns of saccade 2, 2' (see Figures 1 and 2). For this purpose, an ambient environment observation camera connected to the computing unit 3 is further used to detect the surrounding environment in front of vehicle 1. Using this data, the calculation unit 3 of vehicle 1 determines the vector fields of optical flow 6a and 6b of the surrounding environment moving relative to the moving vehicle 1, as shown in Figure 2. 3 Extracted by this method, the optical flow in the parking-related areas on the left side of the road 6a and the right side of the road 6b in Figure 1 is shown.

[0033] Furthermore, when examining patterns in gaze behavior, the relationship of gaze behavior to the vector field of optical flow 6a is considered as follows: By analyzing the saccades (see exemplary saccades 2 and 2' in Figure 2), the final values ​​of the parameters "orientation," "reproducibility," and / or "coupling" are determined. In this case, the parameter "orientation" indicates the orientation of each saccade with respect to the direction vector of the optical flow vector field. Preferably, the orientation is determined as the angle of saccades 2 and 2', represented by vector arrows between the gaze points shown as points.

[0034] The parameter "reproducibility" characterizes the sequence of saccades with respect to each other, i.e., preferably represented by the angle between the vector arrows, for example, that the saccades follow a straight line or extend in a zigzag pattern.

[0035] The parameter "coupling" indicates the length of individual saccades. For example, to eliminate false positives along a horizontal ladder, a value is sought here that includes, among other things, the general dimensions of the parking space. The above parameter is repeatedly derived for each frame of the driver observation camera 5. For example, saccade 2' shown in Figure 2 shows a slight angular deviation with respect to the optical flow 6a, and at the same time, saccades 2' show slight angular deviations from each other. If the length of saccade 2' correlates with the dimensions of the corresponding parking space, the user's parking intention and search for a parking space can be clearly estimated. In contrast, saccades 2 corresponding to the illustration in Figure 1 are disordered from each other, deviate from the optical flow 6a, and have different dimensions that do not correlate with the dimensions of the parking space, so the user's parking intention can be ruled out.

[0036] For evaluation purposes, the final values ​​of the parameters are calculated using the calculation unit. 3 This is determined using the histograms from each of the consecutive frames. Each histogram shows the frequency of occurrence of individual values ​​or value classes of the parameters determined in the history. For example, the parameter mean is obtained from each histogram as the final value of each parameter by multiplying the parameter value or its mean by each frequency, summing them up, and dividing by the total number of parameter values. After determining the final value of each parameter, the calculation unit 3 calculates a weighted sum using weight coefficients for each of the final values ​​of these parameters "orientation," "reproducibility," and "coupling," and the result of the weighted sum is compared with a predetermined limit value. As long as the limit value is on this side of the predetermined threshold within the predetermined range, i.e., for example, below the threshold, preferably when a saccade with this characteristic occurs over a specific minimum duration, the intention to park is estimated.

[0037] In one variant, the final value can be obtained from a histogram of only one or two of the parameters in a simplified form, and then calculated using weighting coefficients.

[0038] However, this analysis of gaze and saccades for determining whether the driver of a vehicle intends to park is performed only for gaze directed towards the surrounding environment of vehicle 1 outside the planar road layout 7 of the road section located ahead of vehicle 1 in the direction of travel, and only for saccades that pass at least partially, and especially completely, through only the area of ​​the surrounding environment of vehicle 1 outside the planar road layout 7. The planar road layout represents the surface of at least one drivable road in the surrounding environment located ahead of vehicle 1. In this case, side roads may also be included in the planar road layout. Thus, the planar road layout is determined on the one hand by the elongated road layout in each direction of travel, which includes straight sections, curves, etc., and on the other hand by the respective width of each location of the elongated road layout, the width of which is indicated in Figure 1 by the two dashed boundary lines of a two-lane road including a center marking. Based on the gaze and saccades in the driver's eye gaze behavior detected by the driver observation camera 5, the allocation of the region of the surrounding environment of vehicle 1 that has been passed through by the gaze direction or saccade of the gaze of vehicle 1 to the surrounding environment of vehicle 1 is calculated by the calculation unit 3This is performed using coordinate system transformation. For this purpose, the current position of each vehicle 1 is determined by satellite positioning, and the position and orientation are determined in the form of the position angle of vehicle 1 relative to the road, and the road layout immediately ahead, including its width, is determined from the information on the digital map. The coordinate system transformation is performed based on the position angle, which is the only transformation parameter. This is because it can be assumed that the driver observation camera 5 is fixed to the vehicle body and positioned at a constant distance from the rest of vehicle 1. When a parking intention is recognized, the automatic parking assist is activated so that the driver of the vehicle is shown a head-up display on the windshield in front of the driver's seat, marking possible parking spaces using augmented reality. The driver of the vehicle can then initiate automatic parking of vehicle 1 into the selected parking space by selecting one of these presented parking spaces, for example by outputting a command such as voice input or other operational action. Alternatively or additionally, the driver is provided with a visual indication of the availability of the automatic parking function on the instrument cluster, switches, or head unit.

[0039] Although the present invention has been illustrated and described in detail with reference to preferred exemplary embodiments, the present invention is not limited by the disclosed examples, and those skilled in the art can derive other variations therefrom without departing from the scope of protection of the present invention. Therefore, it is clear that numerous possible variations exist. Similarly, it is clear that the exemplary embodiments are merely examples and should not be understood as any limitation on the scope, applicability, or configuration of the present invention, for example. Rather, the foregoing description and the illustrations enable those skilled in the art to concretely implement the exemplary embodiments, and in doing so, they can make various modifications, for example, with respect to the function or arrangement of the individual elements given in the exemplary embodiments, without departing from the scope of protection defined by the claims and their legal equivalents, for example, the extensive description in the specification, while being aware of the disclosed inventive concept. [Prior art documents] [Patent Documents]

[0040] [Patent Document 1] German Patent Application Publication No. 102012221036

Claims

1. A system for controlling automatic parking assistance for a vehicle (1), In a system comprising a computing unit (3) and a driver observation camera (5) connected to the computing unit (3), wherein the driver observation camera (5) is used to detect the gaze behavior of the driver of the vehicle (1), and the computing unit (3) is designed to estimate the driver's intention to park the vehicle (1) in a parking space from specific patterns of gaze and saccades in the driver's gaze behavior, and to activate the automatic parking assist at least partially when such an intention is recognized, The calculation unit (3) is designed to determine the planar road layout (7) of the road section located in front of the vehicle (1) in the direction of travel, to assign the driver's line of sight to the current surrounding environment of the vehicle (1), and to exclude gaze fixation toward the planar road layout (7) and / or gaze saccades across the planar road layout (7) at least partially from the examination of patterns in the line of sight behavior regarding the intention to park. A system characterized by the following features.

2. The calculation unit (3) is designed to determine the planar road layout (7) of the road section located ahead of the direction of travel of the vehicle (1) by identifying the current position of the vehicle (1) and assigning the determined position to a digital map in which the road layout is stored. The system according to claim 1.

3. The calculation unit (3) is designed to determine the position angle of the vehicle (1) relative to the planar road layout (7) of the road section based on the road layout (7) corresponding to the digital map and / or using an inertial measurement unit. The system according to claim 2.

4. The computing unit (3) is further equipped with an ambient environment observation camera connected to it for detecting the area in front of the vehicle (1), and the computing unit (3) is designed to determine a vector field of optical flow of the ambient environment moving relative to the moving vehicle (1) from the current data of the ambient environment observation camera, and to consider the relationship of the gaze behavior to the vector field of optical flow when inspecting the pattern of gaze behavior regarding the intention to park. The system according to any one of claims 1 to 3.

5. The calculation unit (3) is designed to determine the value of at least one of the parameters "orientation," "reproducibility," and "coupling" by analyzing the saccades, wherein the parameter "orientation" indicates the orientation of each saccade with respect to the direction vector of the optical flow vector field, the parameter "reproducibility" characterizes the continuity of the saccades with respect to each other, and the parameter "coupling" indicates the length of each of the saccades, and the calculation unit (3) is designed to estimate the intention to park depending on the determined value of each of the at least one parameter. The system according to claim 4.

6. The calculation unit (3) is designed to determine the respective values ​​of the at least one parameter using its respective histogram, the histogram showing the frequency of occurrence of the individual values ​​of the parameter in order to determine the parameter mean as the final value of the parameter from the respective histogram. The system according to claim 5.

7. The calculation unit (3) is designed to calculate a weighted sum of the final values ​​of the parameters "orientation," "reproducibility," and "coupling," compare the result of the sum with a predetermined limit value, and estimate the intention to park if it falls below the limit value. The system according to claim 6.

8. The calculation unit (3) is designed to adjust the weight coefficients of the weighted sum and / or the limit value based on the learned field data. The system according to claim 7.

9. The calculation unit (3) is designed to perform the adjustments in different ways for each individual driver. The system according to claim 8.

10. A vehicle (1) comprising the system described in any one of claims 1 to 9.

Images