Calibration method for accommodation-supported depth reconstruction and motor vehicle

Abstract

The invention relates to a method for calibrating a system for accommodation-supported depth reconstruction of a gaze of an occupant (1) of a vehicle, comprising the steps of: providing (S1) an interior model of the vehicle; determining (S2) a gaze direction of the occupant (1) and determining an associated eye position and an accommodation value of an eye associated with the respective gaze direction; determining (S3) whether a respective gaze direction passing through the eye position has an intersection point (3) with a surface in the interior of the vehicle according to the interior model; and if this is the case: determining (S4) a respective distance between at least one eye of the occupant (1) and the respective intersection point (3) of the gaze direction with the surface; from a respective pair of accommodation value and associated determined distance: generating (S5) a respective support point for a continuous connection;

Description

The invention relates to a method for calibrating a system for accommodation-assisted depth reconstruction of a gaze of an occupant of a vehicle, and to a motor vehicle having a computing unit for carrying out the method.It is known in the prior art to understand by detecting the eyes of a driver of a vehicle not only in which direction the driver is looking, but also on which object in the environment his gaze is focused. For this purpose, depth information is necessary which specifies not only the viewing direction but also the focus, to which distance both eyes are aligned, in order to be able to see an object located at this distance sharply.US 2015 / 0160033 A1 relates in this context to a navigation system which is configured such that a sequence of driving instructions is generated in order to guide a driver of a vehicle from a current location to a final destination. The navigation system is also configured to acquire sensor data reflecting the field of view of the driver and to identify objects in that field of view. The navigation system then adjusts the driving instructions to specific objects and thus generates context-specific driving instructions. The navigation system is also configured to identify particular objects on which the driver is focused and generate context-specific driving instructions with respect to such objects of focus. The navigation system may also give correction information to the driver when it is determined that the driver erroneously focuses on an object other than one in a context-specific driving instruction.Thus, US 2015 / 0160033 A1 discloses gaze sensors which are configured to determine a focal point which represents a specific depth to which the driver is looking along the focal direction.DE 10 2015 116 160 A1 discloses a head-up display for generating a virtual image in the field of view of a user on a windshield. The representation of the virtual image is influenced by a controlled movement of an emission unit and a modification of an emitted image signal. The modification of the image signal is provided for compensating for distortion caused by curvature of the windshield or for short-term changes in the eye position in the vehicle longitudinal direction.For recognizing the depth of focusing of the eyes of a driver of a vehicle, so-called depth reconstruction based on the eye characteristics of the driver may be applied. The result can be helpful in numerous ADAS functionalities, for example in plausibility checking ADAS environment data, the so-called "blindness" detection of camera sensors, or even in implementations simpler relative thereto, such as for implementations for a windshield cleaning system. In the conventional approaches, the depth of the driver's direction of view is reconstructed from a stereo vision of a camera unit. This approach takes into account the so-called disparity between viewing an object by two or more cameras towards the eyes on their image plane.This has the disadvantage that both eyes must always be viewed, which can lead to problems, especially in the case of strong head rotations about a vertical axis. This approach is also susceptible to errors, especially for focusing on higher distances, caused by the lower disparity. Moreover, one of the major weaknesses of this approach is the so-called correspondence analysis, i.e. the search of a feature on both image planes of the eyes.One approach to avoiding this is so-called accommodation-assisted depth reconstruction. Accommodation refers to a dynamic adaptation of the refractive power of the eye. By means of accommodation, it is possible to sharply image an object which is located at any distance between the individual near and far points onto a retinal plane. The accommodation of the eye therefore correlates with the respective viewing depth. However, the use of this correspondence can only be carried out individually to a great extent and the need for an initial calibration.It is the object of the invention to provide a calibration approach that is individually adapted to a respective occupant of a vehicle in order to enable an individual accommodation-supported depth reconstruction for determining a depth of a viewing focus of the occupant with the result thereof.The invention results from the features of the independent claims. Advantageous refinements and refinements are the subject matter of the dependent claims.A first aspect of the invention relates to a method for calibrating a system for accommodation-assisted depth reconstruction of a gaze of an occupant of a vehicle, comprising the steps of:providing an interior model of the vehicle, wherein the interior model comprises position information about the surfaces of the interior of the vehicle;determining a viewing direction of the occupant or a plurality of viewing directions at different points in time, and determining an eye position associated with the respective viewing direction and an accommodation value associated with the respective viewing direction of at least one eye of the occupant in each case;determining whether a respective viewing direction with progression through the eye position has an intersection point with a surface in the interior of the vehicle according to the interior model; and if this is the case:determining a respective distance between at least one eye of the occupant and the respective intersection point of the viewing direction with the surface;from a respective pair of accommodation values and a distance determined in association therewith: generating a respective support point for a continuous relationship between accommodation values and respective associated focused depths of a gaze of an occupant, whereinthe computing unit is designed to determine the continuous relationship between accommodation values and associated focused depths of a gaze of an occupant from the support values and to apply the continuous relationship, such that an associated focused depth of a gaze of an occupant of the vehicle is determined from an determined accommodation value, and to transfer the determined depth to a driver assistance system of the vehicle, and wherein the driver assistance system comprises a parking assistance system which is designed to use the determined depth for generating signals for a visual and / or acoustic output device and / or for controlling actuators.The interior model of the vehicle comprises a data record about the respective positions on surfaces of components of the interior of the vehicle. For example, a CAD model of the interior of the vehicle can be used for this purpose, in which individual reference points are specified and, for example, straight lines which connect these reference points to one another are specified. Geometric primitives such as circular arcs can also be parametrically defined-the type of storage of positions of the surfaces is practically of no importance, as long as a viewing direction of the occupant under consideration, preferably defined as a vector or straight line, can be analyzed to determine whether the viewing direction meets a surface in the interior of the vehicle defined in this way.Here, the assumption is made that when a gaze of an occupant is made on a surface in the interior of the vehicle, the occupant focuses this surface, so that the surface appears visually sharp for him. Accordingly, accommodation is formed in the eye of the occupant, in particular in both eyes of the occupant, i.e. by shape change and the changing refractive power of the lens of an eye accompanying this, it is possible for the eye to focus this surface sharply.For this purpose, the eye position and the viewing direction of the occupant proceeding therefrom and the provided information regarding the positions of the surfaces of components in the interior of the vehicle are expediently determined in the same coordinate system; the eye position is preferably determined in a coordinate system which is fixed to the body of the interior model of the vehicle.The method for calibrating the system for depth reconstruction at a detected gaze of an occupant in a vehicle serves to store a relationship, which accommodation value results depending on the depth to which a gaze of the occupant focuses. The depth corresponds to a distance between a focused object, for example a point on a surface in the interior of the vehicle, and one or both eyes of the occupant who is wearing this gaze.The more such interpolation points, each consisting of an accommodation value and a distance determined in association (i.e. the focused depth of a respective gaze), are determined, the more high-resolution a continuous relationship can be established between accommodation values and associated focused depths of a gaze of an occupant. It does not matter whether only the interpolation points are stored and interpolation is carried out by means of these interpolation points during operation of the system, which takes the result of the method, or whether the continuous relationship is already determined when applying the method for calibration, so that this continuous relationship can be used as a continuous function by the system in order to be able to determine the associated focused depth of a gaze of the occupant as an output value with a determined accommodation value as an input value.The method can therefore advantageously already have the further step: generating the continuous relationship between accommodation values and associated focused depths of a gaze of an occupant with the aid of at least one support point, preferably with the aid of a multiplicity of support points.The method for calibrating the system for accommodation-assisted depth reconstruction of a gaze of an occupant allows an individually adapted relationship to be determined between accommodation values of at least one eye and associated focused depths of a gaze. The occupant is in particular a driver, since the determination of his gaze is typically of significantly higher importance than other passive riders, but the method and the system for applying the results of the method can also be applied to other occupants. While such a calibration can already be carried out by the manufacturer without reference to an individual occupant, this is not necessary. Current weather conditions and light conditions and properties of the current vehicle environment are also of no importance for this purpose and do not influence the method for calibration, since the interior model is independent thereof and the detection of the viewing direction can likewise be carried out independently.In order to be able to determine the distance between a supporting point of a viewing direction with the surface of the interior model, the position of the impact of the viewing direction on the surface is known on the one hand, and the position of the eye or the eyes performing the viewing is known on the other hand. The position of the eye is preferably determined by a 3D driver observation camera. The position of the eye or eyes is in each case a point through which the straight line of the viewing direction has to run, or is a starting point of a vector of the viewing direction. If the viewing direction passing through the position of the eye is thus known, the point of intersection of the viewing direction with the surface of the modeled interior space can be calculated. From this, the distance between the eye or eyes and the point of intersection with the surface is also known. This distance corresponds to a depth that is focused at the glance.For this depth, an individually dependent accommodation value is to be set by the eye or by the eyes of the occupant. This in turn means that when the accommodation value is detected, a focused depth of the gaze can be calculated, provided that a relationship calibrated accordingly for an individual occupant is known. This in turn means that in the method for calibrating, such a pair of accommodation value and the distance determined in association therewith serves to be able to determine, during operation of the system for accommodation-assisted depth reconstruction, from an accommodation value, an expected focused depth analogous to the distance (which is determined in the method for calibrating as assumed focused depth).The viewing direction is preferably also determined by such a driver observation camera; in particular with the aid of a Purkinje or corneal reflection. An accommodation value can also be determined with such a driver observation camera.According to an advantageous embodiment, in order to generate a further supporting point for the continuous relationship, a further viewing direction which differs from a previously determined viewing direction, and an intersection point with the surface associated with the further viewing direction and an associated accommodation value are determined.According to a further advantageous embodiment, the continuous relationship is determined by at least two interpolation points by a predefined mapping rule of accommodation values being adapted to focused depths by the at least two interpolation points.According to a further advantageous embodiment, the continuous relationship is determined and adapted to sensor data of a depth sensor. The depth sensor is used in particular for driver or gesture observation.According to a further advantageous embodiment, respective interpolation points for the continuous relationship and / or a continuous relationship between accommodation values and associated focused depths of a gaze of an occupant determined from the interpolation points are stored for an individual occupant in order to be retrievable in a personalized manner.Alternatively, a continuous, so-called "on the fly" adaptation of the calibration curve can be carried out to take account of currently changed boundary conditions such as tiredness, weather, etc.According to a further advantageous embodiment, the accommodation value is determined by an interior camera, which can be a driver observation camera.A further aspect of the invention relates to a motor vehicle having a computing unit which is designed to carry out the method according to one of the preceding steps.According to a further advantageous embodiment, the driver assistance system comprises a holographic display and / or a system for controlling the gaze and / or a system for cleaning the windshield.In this case, a distance to the object can also be determined on the basis of a view from a side window onto an object, for example a pillar.Advantages and preferred refinements of the proposed motor vehicle result from an analogous and analogous transfer of the statements made above in connection with the proposed method.Further advantages, features and details are evident from the following description, in which--possibly with reference to the drawing--at least one exemplary embodiment is described in detail. Identical, similar and / or functionally identical parts are provided with the same reference numerals.The following are shown: FIG. 1 : shows a method for calibrating a system for depth reconstruction according to an exemplary embodiment of the invention. FIG. 2 : shows a motor vehicle interior for the application of the method of FIG. 1. FIG. 3 : shows an exemplary correlation between accommodation values and respectively associated focused depths.The representations in the figures are schematic and not to scale.FIG. 1 shows a method for calibrating a system for a vehicle, which serves to carry out an accommodation-assisted depth reconstruction of a gaze of an occupant 1 of the vehicle. The purpose of calibration is to determine a relationship to which accommodation value is adjusted by one or both eyes of the occupant to focus a certain depth with his gaze. If such a relationship is known, a depth of the gaze into which a gaze of the occupant focuses can be estimated from the measurement of the respective accommodation value. The depth can be directed into the interior region, but also into the exterior region of the vehicle, i.e. into the environment outside the vehicle interior. However, the calibration method only occurs for eyes of an occupant who are directed into the interior of the vehicle. By providing a corresponding interior model, such as from CAD data, coordinates of surfaces to which potential eyes of an occupant can be directed are known. The following method steps are carried out in the interior or with reference to the interior, as outlined in FIG. 2. For this reason, in the course of the description of the following method steps, FIG. 2 can also be used for further understanding. In a first step of the method, an interior model is provided S 1. This is a parametric model and originates from a CAD modeling of components in the interior of the vehicle. A viewing direction of the occupant 1 is now determined S 2 using an interior camera 5, and an accommodation value of at least one eye of the occupant 1 associated with the respective viewing direction. If, in addition, the positions of the eyes or of at least one eye of the occupant 1 are determined by means of the interior camera 5, a starting point can also be assigned to the viewing direction. However, first of all, S 3 checks whether the viewing direction has an intersection point 3 with a surface of the interior model. Then, it can be considered that the occupant 1 focuses his gaze at this point on the surface of a component of the interior of the vehicle. If the test runs positively, then with the aid of the position of the eye or eyes of the occupant 1 and with the position of the intersection point 3 of the viewing direction with a surface of a component in the interior, a distance between the focusing eye and the focal location, namely the intersection point 3, is determined S 4. For this purpose, the position of the eye or eyes and the position of the intersection point 3 are advantageously indicated in the same coordinate system, for example one which is given by the CAD model for the interior space. A support point can thus be generated S 5 for each data pair of accommodation values and distance, wherein such a support point can be used to generate a continuous relationship between accommodation values and respective associated focused depths of a gaze of an occupant 1. This is advantageously done by interpolation between two interpolation points, or by establishing a continuous function, which is determined, for example, by nonlinear regression. With the interpolation nodes or the continuous relationship determined on the basis of the latter, it is possible for the system in its later operation to determine an associated focused depth of the gaze from a measurement of a current accommodation value.FIG. 2 shows an interior of a vehicle by way of example, in which an interior camera 5 is arranged, which can currently determine the viewing direction of the occupant's eye or eyes 1. In addition, the interior camera 5 determines a position of the eye or eyes present at the same time. By comparing the viewing direction, which must pass through the position of the eye or eyes, with the position information relating to the surface of the interior of the vehicle, an intersection point 3 can be determined, and a distance of the focused intersection point 3 to the eye or eyes of the occupant 1 can be determined from the relative position of the intersection point 3 to the position of the eye or eyes of the occupant 1. While only one exemplary intersection point 3 is indicated by way of example in FIG. 2, it is expedient for a high-quality calibration method to repeat the abovementioned method steps S 2 to S 5 as many times as possible in order to obtain data points for different distances d.FIG. 3 shows an exemplary diagram with data points, also called interpolation points, wherein the horizontal axis of the diagram indicates an accommodation value A, while the vertical axis indicates a measured distance in the interior of the vehicle d. Here, by way of example, a multiplicity of interpolation nodes is obtained by repeating method steps S 2 to S 5, from which a continuous curve is determined by using a nonlinear regression method. This continuous curve represents the desired continuous relationship between accommodation values and respectively associated focused depths of a gaze of an occupant, so that by reading the curve at a measured accommodation value, an associated focused depth d of a current gaze can be determined, which can also be guided into the surroundings of the vehicle.Although the invention has been illustrated and explained in more detail by preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by the person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a large number of possible variations exist. It is also clear that embodiments mentioned by way of example represent only examples which are not to be understood in any way as limiting, for example, the scope of protection, the possible applications or the configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, wherein the person skilled in the art, knowing the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims

Method for calibrating a system for accommodation-assisted depth reconstruction of a gaze of an occupant (1) of a vehicle, comprising the steps of: - providing (S1) an interior model of the vehicle, wherein the interior model comprises position information about the surfaces of the interior of the vehicle; - determining (S2) a viewing direction of the occupant (1) or of a plurality of viewing directions at different times, and determining an eye position associated with the respective viewing direction and an accommodation value associated with the respective viewing direction of at least one eye of the occupant (1) in each case; - determining (S3) whether a respective viewing direction with course through the eye position has an intersection point (3) with a surface in the interior of the vehicle according to the interior model; and if this is the case: - determining (S 4) a respective distance between at least one eye of the occupant (1) and the respective intersection point (3) of the viewing direction with the surface; - from a respective pair of accommodation values and a distance determined in association therewith: generating (S 5) a respective supporting point for a continuous relationship between accommodation values and respective associated focused depths of the gaze of the occupant (1), wherein a computing unit is configured to determine the continuous relationship between accommodation values and associated focused depths of the gaze of the occupant (1) from supporting values and to apply the continuous relationship, such that an associated focused depth of the gaze of the occupant (1) of the vehicle is determined from a determined accommodation value, and to do so, the determined depth to be transferred to a driver assistance system of the vehicle, and wherein the driver assistance system comprises a parking assistance system which is designed to use the determined depth for generating signals for a visual and / or acoustic output device and / or for actuating actuators.Method according to claim 1, wherein for generating a further interpolation point for the continuous relationship, a further viewing direction, which differs from a previously determined viewing direction, and an intersection point (3) with the surface and the associated accommodation value associated with the further viewing direction are determined.Method according to one of the preceding claims, wherein the continuous relationship is determined by at least two interpolation points by a predefined mapping rule of accommodation values being adapted to focused depths by the at least two interpolation points.Method according to one of the preceding claims, wherein the continuous relationship is determined and adapted with sensor data of a depth sensor.Method according to one of the preceding claims, wherein respective interpolation points for the continuous relationship and / or a continuous relationship between accommodation values and associated focused depths of the gaze of the occupant (1) determined from the interpolation points are stored for an individual occupant (1) in order to be retrievable in a personalized manner.Method according to one of the preceding claims, wherein the accommodation value is determined by an interior camera (5).Motor vehicle having a computing unit which is designed to carry out the method according to one of the preceding steps.Motor vehicle according to Claim 7, wherein the driver assistance system comprises a holographic display and / or a system for controlling gaze and / or a system for windshield cleaning.

Images