Method and apparatus for determining a center point and an eye opening of an eye of an occupant of a vehicle

Abstract

Method for determining a center point (504) of an eye (202) of an occupant of a vehicle (100), the method comprising the following steps: Determining (803) reference pupil pixel coordinates (140) and reference eye angle pixel coordinates (142) using a reference image signal (110) representing a reference image (300) captured by a camera (106) of at least a portion of a face of the occupant exhibiting a reference head pose, wherein the reference pupil pixel coordinates (140) represent a position of a center point (504) of a pupil (506) of the eye (202) of the occupant in the reference image (300) and the reference eye angle pixel coordinates (142) represent a position (312) of an eye angle of the eye (202) of the occupant in the reference image (300);Determine (805) homogeneous reference pupil coordinates (144) in a camera coordinate system (120) of the camera (106) using the reference pupil pixel coordinates (140) and homogeneous reference eye angle coordinates (146) in the camera coordinate system (120) of the camera (106) using the reference eye angle pixel coordinates (142); Determine (809) reference eye angle camera coordinates (148) using the homogeneous reference eye angle coordinates (146) and an eye angle associated depth information (588), wherein the reference eye angle camera coordinates (148) represent a position (312) of the eye angle in the camera coordinate system (120);and determine (811) reference eye center camera coordinates (150) assigned by the reference head pose as the intersection of a reference vector formed from the homogeneous reference pupil coordinates (144) and the homogeneous reference eye angle coordinates (146) and a reference sphere, wherein a center of the reference sphere is defined by the reference eye angle camera coordinates (148) and a diameter of the reference sphere by a mean eye diameter of a human eye (202).

Description

State of the art The invention relates to a device or a method according to the preamble of the independent claims. The present invention also relates to a computer program. In the context of modern driver assistance systems, which, for example, temporarily take over complete vehicle control on the motorway, driver monitoring is becoming increasingly important. This is necessary because the driver assistance systems offered only serve to increase comfort and do not constitute a function for automated driving; therefore, the driver must be able to continuously monitor the surrounding traffic and their vehicle at all times. The known prior art documents are DE 10 2010 044 449 A1 , DE 10 2015 204 283 A1 , DE 10 2015 210 090 A1 , DE 10 2016 201 939 A1 , DE 10 2017 212 446 A1 , DE 11 2011 105 435 T5 and US 7 401 920 B1 . Disclosure of the invention Against this background, the approach presented here comprises a method for determining a center point and a method for determining the eye opening of an occupant of a vehicle, a device that uses these methods, and finally a corresponding computer program according to the main claims. Advantageous further developments and improvements of the device specified in the independent claim are possible through the measures listed in the dependent claims. Advantageously, a spherical model of a vehicle occupant's eye can be used to determine the position of the eye's center point within the camera coordinate system of a camera capturing the eye. Knowing the eye's center point, in turn, allows for the reliable determination of the eye's aperture using the spherical model. Additionally or alternatively, the center point can be used to determine the occupant's gaze direction. For example, the gaze direction can be determined by a vector from the eye's center point to the pupil's center point. A method for determining the center point of an occupant's eye comprises the following steps: determining reference pupil pixel coordinates and reference eye angle pixel coordinates using a reference image signal representing a reference image captured by a camera of at least a portion of the occupant's face in a reference head pose, wherein the reference pupil pixel coordinates represent the position of the center point of a pupil of the occupant's eye in the reference image and the reference eye angle pixel coordinates represent the position of an eye angle of the occupant's eye in the reference image; determining homogeneous reference pupil coordinates in a camera coordinate system of the camera using the reference pupil pixel coordinates and homogeneous reference eye angle coordinates in the camera coordinate system of the camera using the reference eye angle pixel coordinates;Determining reference eye angle camera coordinates using the homogeneous reference eye angle coordinates and depth information associated with the eye angle, wherein the reference eye angle camera coordinates represent a position of the eye angle in the camera coordinate system; and determining reference eye center camera coordinates associated with the reference head pose as the intersection of a reference vector formed from the homogeneous reference pupil coordinates and the homogeneous reference eye angle coordinates and a reference sphere, wherein a center point of the reference sphere is defined by the reference eye angle camera coordinates and a diameter of the reference sphere is defined by a mean eye diameter of a human eye. The method can be used in conjunction with a vehicle's driver monitoring system. Alternatively, it can be used for monitoring systems deployed outside of vehicles, for example, in factories. The reference head pose can be understood as a predetermined position of the occupant's head. The camera can be an image acquisition device located in a vehicle, such as a stereo camera. The center of the eye and the center of the eye can be understood as the center point of the eyeball. The camera can be associated with the camera coordinate system. Pixel coordinates can be understood as coordinates relating to an image plane of the reference image. Homogeneous coordinates can be understood as homogeneous coordinates known from projective geometry.The homogeneous coordinates can be determined from the pixel coordinates using a suitable transformation rule. According to one embodiment, homogeneous coordinates defining the pupil center in the camera coordinate system can at least approximately correspond to the homogeneous coordinates defining the eye center. Thus, the homogeneous reference pupil coordinates can be equated with homogeneous reference eye center coordinates. The depth information can be contained in the reference image signal or determined from it, for example, by depth image analysis. The mean eye diameter can represent a predetermined value. The process can comprise a step of capturing the reference image using the camera and a step of providing the reference image signal. Advantageously, a camera that is part of the vehicle's driver monitoring system can be used for this purpose. This eliminates the need for an additional camera. The reference image signal can consist of raw data captured by the camera or already processed data. The reference image can be captured when the occupant is looking directly into the camera lens. This state can define the reference head pose. This state can be automatically detected using suitable image analysis. In this way, the reference eye center camera coordinates can be automatically determined as soon as the occupant is looking directly into the camera, even if only by chance. The procedure may include a step of determining the depth information associated with the periphery of vision using the reference image signal. For example, a disparity measurement can be performed for this purpose. Advantageously, the reference eye center camera coordinates determined in the first instance can then be used to determine the eye center in further images taken by the same camera. These additional images may have been taken at times when the occupant moved their head, thus showing the head in different poses, which may also differ from the reference head pose. The respective eye center can then be used to determine whether the eye is open or closed in these additional images. A corresponding method for determining the eye opening of an occupant's eye comprises the following steps: reading in reference eye center camera coordinates, which represent coordinates determined according to a said method for determining the center of an occupant's eye; determining first eyelid pixel coordinates and second eyelid pixel coordinates using a camera image signal representing a camera image captured by the camera of at least a portion of the occupant's face in a head pose, wherein the first eyelid pixel coordinates represent the position of a point of an upper eyelid of the occupant's eye in the camera image and the second eyelid pixel coordinates represent the position of a point of a lower eyelid of the occupant's eye in the camera image;Determining eye-center camera coordinates associated with the head pose using the reference eye-center camera coordinates and a transformation rule that converts the reference head pose into the head pose; projecting the first eyelid pixel coordinates onto a sphere to obtain first eyelid camera coordinates, and the second eyelid pixel coordinates onto the sphere to obtain second eyelid camera coordinates, where a center point of the sphere is defined by the eye-center camera coordinates and a diameter of the sphere by the mean eye diameter of the human eye; determining an eye-opening signal representing the eye opening of the eye using the first eyelid camera coordinates and the second eyelid camera coordinates. The eye opening can indicate, for example, whether the eye is fully open, partially open, or closed. This can be achieved using pre-determined reference eye center camera coordinates, which define the position of the eye center when the occupant's head is held in the reference head pose. The occupant's head pose as depicted by the camera image may differ from the reference head pose, and therefore the position of the eye center in the head pose may also differ from the position of the eye center in the reference head pose. At least one point on the upper eyelid and one point on the lower eyelid can be predefined. Advantageously, several points assigned to the upper eyelid and several to the lower eyelid can be predefined, and the corresponding pixel coordinates of these points can be determined.The transformation rule that converts the reference head pose into the actual head pose can, for example, define a movement—such as a rotation and / or translation—of the reference vector formed from the homogeneous reference pupil coordinates and the homogeneous reference eye angle coordinates. This movement transforms the reference vector into a vector formed by homogeneous pupil and eye angle coordinates relative to the camera image. The transformation rule can be predetermined or determined using a suitable method, such as comparing the camera image with the reference image using the camera image signal and the reference image signal. The eye opening signal can be an electrical signal indicating the eye's opening state at a given time of image acquisition.The eye-opening signal can be used, for example, by a vehicle's driver assistance system to assess the driver's current level of attention. The steps involved in determining the eye-opening status can be repeated continuously to monitor the eye's open state. The procedure can include a step of determining the distance between the first and second eyelid camera coordinates. The eye opening signal can then be determined using this distance. Since the eyelid camera coordinates, due to the spherical model, very accurately represent the actual position of the eyelids, the distance between the points defined by the eyelid camera coordinates allows for a very precise determination of whether, and if so, how far the eye is open. The methods mentioned can be implemented, for example, in software or hardware, or in a hybrid form of software and hardware, for example in a control unit. The approach presented here further creates a device designed to perform, control, and implement the steps of a variant of the method presented here in appropriate facilities. This embodiment of the invention in the form of a device also allows the problem underlying the invention to be solved quickly and efficiently. For this purpose, the device may have at least one processing unit for processing signals or data, at least one storage unit for storing signals or data, at least one interface to a sensor or actuator for reading sensor signals from the sensor or for outputting data or control signals to the actuator, and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The processing unit may, for example, be a signal processor, a microcontroller, or the like, and the storage unit may be flash memory, EEPROM, or a magnetic storage device.The communication interface can be configured to read or output data wirelessly and / or via wired connections, whereby a communication interface that can read or output wired data can, for example, read this data electrically or optically from or output it into a corresponding data transmission line. In this context, a device can be understood as an electrical device that processes sensor signals and outputs control and / or data signals accordingly. The device may have an interface, which can be implemented in hardware and / or software. In the case of a hardware-based interface, the interfaces can, for example, be part of a so-called system ASIC, which incorporates various functions of the device. However, it is also possible that the interfaces are separate integrated circuits or consist at least partially of discrete components. In the case of a software-based interface, the interfaces can be software modules, which, for example, are present on a microcontroller alongside other software modules. Also advantageous is a computer program product or computer program with program code that can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and is used to carry out, implement and / or control the steps of the method according to one of the embodiments described above, in particular if the program product or program is executed on a computer or device. Exemplary embodiments of the approach presented here are shown in the drawings and explained in more detail in the following description. Figure 1 shows a schematic representation of a vehicle with a device according to one embodiment; Figure 2 shows a first head pose according to one embodiment; Figure 3 shows a second head pose according to one embodiment; Figure 4 shows a reference image of an eye with markings of the pupil and the corner of the eye according to one embodiment; Figure 5 shows a schematic representation of an eye according to one embodiment; Figure 6 shows a camera image of an eye with markings of the eyelids according to one embodiment; Figure 7 shows a schematic representation of an eye model according to one embodiment; Figure 8 shows a flowchart of a method according to one embodiment. In the following description of favorable embodiments of the present invention, the same or similar reference numerals are used for the elements shown in the various figures and acting similarly, without repeating these elements. Fig. 1 shows a schematic representation of a vehicle 100 with a device 102 according to an exemplary embodiment. The vehicle 100 contains an occupant 104, who may, for example, be the driver of the vehicle 100. The vehicle 100 has a camera 106 for observing the occupant 104. The camera 106 is designed to capture images of the face or at least part of the eye area of ​​the occupant 104. The camera 106 may be a suitable image capture device, for example, in the form of a stereo camera, such as those already used as driver observation cameras. According to this embodiment, the camera 106 is configured to capture a reference image of the occupant's face 104 at a time when the occupant's head 104 is in a specific position, referred to as the reference head pose. For example, in the reference head pose, the occupant 104 is looking directly into the camera 106. The camera 106 is configured to provide a reference image signal 110 that represents the reference image of the occupant's face 104. The reference image signal 110 can include raw image data or pre-processed data. According to one embodiment, the device 102 is configured to determine the center point of an occupant's eye in a camera coordinate system 120 assigned to the camera 106 or in a world coordinate system 124 assigned to the vehicle 100. For this purpose, the device comprises detection devices 130, 132, 134, 136. The detection device 130 is designed to determine reference pupil pixel coordinates 140 and reference eye angle pixel coordinates 142 from the reference image signal 110 using a suitable evaluation method. The reference pupil pixel coordinates 140 represent the position of the center point of a pupil in the eye of occupant 104 within the reference image, and the reference eye angle pixel coordinates 142 represent the position of an eye angle of occupant 104 within the reference image. The pixel coordinates refer, for example, to an image plane of the reference image. The detection device 132 is designed to determine, for example, homogeneous reference pupil coordinates 144 from the reference pupil pixel coordinates 140 and homogeneous reference eye angle coordinates 146 from the reference eye angle pixel coordinates 142, using a suitable conversion rule. The homogeneous reference pupil coordinates 144 and the homogeneous reference eye angle coordinates 146 represent homogeneous coordinates related to the camera coordinate system 120. The detection device 134 is designed to determine, for example, reference eye angle camera coordinates 148 from the homogeneous reference eye angle coordinates 146 and depth information associated with the eye angle, using a suitable transformation rule. The reference eye angle camera coordinates 148 represent a position of the eye angle in the camera coordinate system 120. According to one embodiment, the depth information is determined using the reference image signal 110, for example, by the detection device 130 or the detection device 134. The detection device 136 is configured to determine and provide reference eye center camera coordinates 150 associated with the reference head pose. These reference eye center camera coordinates 150 can be used to reliably determine the center of the eye in subsequent camera images from camera 106. To determine the reference eye center camera coordinates 150, the detection device 136 is configured to generate a reference vector from the homogeneous reference pupil coordinates 144 and the homogeneous reference eye angle coordinates 146. Furthermore, the detection device 136 is configured to span a reference sphere whose center is defined by the reference eye angle camera coordinates 148 and whose diameter corresponds to the mean diameter of a human eye.Furthermore, the investigation device 136 is designed to determine the reference eye center camera coordinates 150 as the intersection point of the reference vector and the reference sphere. According to one embodiment, the device 102 is configured to determine the eye opening of an occupant's eye using the reference eye center camera coordinates 150 and a camera image captured by the camera 106. According to this embodiment, the camera 106 is configured to capture a camera image of the occupant's face 104 at a later time. At this later time, the occupant's head 104 may have a head pose that differs from the reference head pose. For example, the occupant 104 may be looking sideways past the camera 106. The camera 106 is configured to provide a camera image signal 152 representing the camera image of the occupant's face 104. The camera image signal 152 may include raw image data or pre-processed data. In order to determine the eye opening of the occupant's eye at the further time at which the camera image was captured, the device 102 according to this embodiment has detection devices 160, 162, 164. The detection device 160 is designed to read the camera image signal 152 and, using a suitable evaluation rule, to determine the first eyelid pixel coordinates 170 and the second eyelid pixel coordinates 172 from the camera image signal 152. The first eyelid pixel coordinates 170 represent the position of a point on the upper eyelid of the eye of occupant 104 in the camera image, and the second eyelid pixel coordinates 172 represent the position of a point on the lower eyelid of the eye of occupant 104 in the camera image. The pixel coordinates 170 and 172 refer, for example, to an image plane of the camera image. The detection device 162 is designed to read the first eyelid pixel coordinates 170, the second eyelid pixel coordinates 172, and the reference eye center camera coordinates 150 and to use them to determine the eye center camera coordinates assigned to the head pose. The detection device 162 can use a transformation rule to convert the reference head pose into the head pose. The detection device 162 is configured to mount a sphere whose center point is defined by the eye-center camera coordinates and whose diameter corresponds to the mean diameter of the human eye. The detection device 162 is further configured to project the first eyelid pixel coordinates 170 and the second eyelid pixel coordinates 172 onto the sphere, for example, using a suitable projection rule, and to obtain the first eyelid camera coordinates 174 and the second eyelid camera coordinates 176 from the projection. The detection device 164 is configured to determine an eye-opening signal 178 representing the eye opening of the eye using the first eyelid camera coordinates 174 and the second eyelid camera coordinates 176. For example, the detection device 164 is configured to determine a distance between the first eyelid camera coordinates 174 and the second eyelid camera coordinates 176 and to determine the eye-opening signal 178 using this distance. According to one embodiment, the detection device 160 is designed to determine further eyelid pixel coordinates assigned to further points of the eyelids, which can be used by the further detection devices 162, 164 to determine the eye opening more precisely. According to an alternative embodiment, the investigative devices 130, 132, 134, 136 and the investigative devices 160, 162, 164 are not implemented together in the device 102 but in separate devices. Fig. 2 shows a first head pose of a head 200 according to an exemplary embodiment. This can be the head of the occupant shown in Fig. 1, which is held upright in the first head pose. A nose and an eye 202 are shown schematically. The eye 202 is captured by a camera 106. The camera 106 can, for example, be a vehicle camera as shown with reference to Fig. 1. From the perspective of the camera 106, the upper and lower eyelids of the eye 202 have a distance d. Fig. 2 also shows the camera coordinate system 120 associated with the camera 106, here with an x-axis, a y-axis and a z-axis, as well as the world coordinate system 124, here also with an x-axis, a y-axis and a z-axis. Fig. 3 shows a second head pose of the head 200 shown in Fig. 2 according to an exemplary embodiment. In the second head pose, the head 200 is angled downwards. From the perspective of the camera 106, the upper and lower eyelids of the eye 202 have a distance d'. According to one embodiment, camera 106 is part of a so-called driver monitoring system, which is implemented as a camera-based system and directed towards the driver. The driver's face is of particular interest. The camera images from camera 106 are processed by suitable algorithms and provide, for example, the head pose 200, the direction of gaze, or the opening of the eyes 202 as a dimension in millimeters. The temporal progression of this eye opening plays a central role in detecting the driver's fatigue level. To validate the corresponding "estimation algorithms" for driver modeling, reference methods are used that enable higher accuracy.The exemplary implementations described here deal with the development of a camera-based reference method for generating ground truth data for eye opening, which primarily takes into account the compensation of the influence of head pose on the eye opening result. For this purpose, a model-based method is used to compensate for head pose when determining eye opening using image data from one or more driver observation cameras. This method can be used additionally or alternatively to a procedure in which the driver's head 200 is tracked using a stereo camera system and a target. The position of the target is thus known in the world coordinate system 124, which is set up in the vehicle. With an additional tool, distinctive points on the driver's face can be measured once. Because the position of these points relative to the head 200 remains constant, their position in the world coordinate system 124 can also be calculated at any given time. Knowing the extrinsic parameters of the camera 106, all points can also be converted into the camera coordinate system 120 of the camera 106. In this alternative approach, the labeled eyelids, as shown below in Fig. 6, are projected onto a plane 204 that lies parallel to the xy-plane of the world coordinate system 124 in the vehicle.The labeling process can be understood as marking relevant features, such as the corner of the eye, the center of the pupil, or points on the eyelids, in an image from camera 106. The pixel coordinates of these features relative to the image can then be determined. For example, the labeled points shown in Fig. 6 would be projected onto the plane 204. This may be sufficiently accurate if the driver's head 202 is facing straight ahead, as shown in Fig. 2. However, if the driver's head 202 is tilted downwards, as shown in Fig. 3, this results in a "reduction" of the eye opening. Thus, the distance d' shown in Fig. 3 appears smaller than the distance d shown in Fig. 2, meaning that the result calculated for the second head pose shown in Fig. 3 with respect to the eye opening would no longer correspond to reality.The model-based method described here avoids such an error that occurs when determining the eye opening during projection onto the static plane 204. Fig. 4 shows a reference image 300 of an eye 202 with a pupil center position 310 and an eye angle position 312 according to an exemplary embodiment. The reference image 300 can, for example, have been acquired using the camera described with reference to Fig. 1. The eye 202 can look directly into the camera, thus establishing the reference head pose. The pupil center position 310 and the eye angle position 312 can be represented by the reference pupil pixel coordinates and reference eye angle pixel coordinates described with reference to Fig. 1. The pixel coordinates can include x and y values ​​with respect to the camera coordinate system. The pixel coordinates are determined by labeling the pupil center position 310 and the eye angle. Fig. 5 shows a schematic representation of an eye 202 according to an exemplary embodiment. This could be the eye shown in Fig. 4. Shown are the eyeball 502 of the eye 202, the center 504 of the eye 202, and the pupil 506 of the eye 202. The position 312 of the canthus of the eye 202 is also shown. According to this embodiment, the occupant's head has a reference head pose in which the occupant looks directly into the camera 106 with eye 202, as indicated by vector 580, also referred to as vector 580. Vector 580 passes through the center point of the pupil 506 and an origin of the camera coordinate system 120. A visual axis 582 runs parallel to vector 580. An optical axis 584 passes through the center point 504 of eye 202. A vector 586, also referred to as vector 586, extends from position 312 of the eye's periphery to the center point 504 of eye 202. Depth information 588 indicates a distance z_css from position 312 of the eye's periphery to the origin of the camera coordinate system 120 in the z-direction of the camera coordinate system 120. If the position of the eye center 504 in the camera coordinate system 120 is known, labeled points, such as those shown below in Fig. 6, can be directly projected onto a spherical model of the eye 202, as described below with reference to Fig. 7. It is assumed that the diameter of a human eyeball is relatively constant in adulthood. This model-based approach allows for a precise determination of the eye opening from a geometric point of view. If the position of the eye center 504 and the center of the pupil 506 are known, the eye's gaze direction can be determined by passing a vector from the position of the eye center 504 through the center of the pupil 506. A signal indicating the gaze direction can be provided and used, for example, by a vehicle assistance system. Figure 3 illustrates the determination of a vector for estimating the center of the eye. A method for estimating the center of the eye (504) is discussed. The position 312 of the eye angle can be determined by measurement and is therefore known in the camera coordinate system 120. Suitable image analysis methods can be used to determine the position 312 of the eye angle. If a vector is known that points from this eye angle in any head pose to the center point 504 of the eyeball 502, this vector can be rotated with each head pose. This vector, together with the position information of the eye angle, can then provide the center point 504 of the eye 202 in the camera coordinate system 120. An estimate is made for the center point 504 of the eye 202, which is explained below. After measuring the landmarks, here the angle of the eye and the center of the pupil 506, the test subject, for example the occupant of a vehicle, must look directly into the lens of the camera 106. Subsequently, the center of the pupil 506 and an eye corner, in this case the outer eye corner, are labeled and the pixel coordinates are stored. The pixel coordinates of the eye center 504, also called peyecenter, and of the eye corner, also called peyecorner, are converted into homogeneous coordinates in the camera coordinate system 120. The position 312 of the peyecorner, which has also been labeled, is transformed into the camera coordinate system 120. Using the depth information 588 in the camera coordinate system 120, also referred to as z_ccs, and the homogeneous coordinates of the peyecorner, the position 312 of the peyecorner in the camera coordinate system 120 is calculated. It is now assumed that the center point 504 of the eye 202, in the case that the person is looking into the camera 106, has the same homogeneous coordinates in the camera coordinate system 120 as the center of the pupil 506. This assumption is not entirely correct, but the error will be neglected. If a sphere with the mean diameter of the human eye 202 is placed around position 312 of the peyecorner and the intersection point with the vector 580, also referred to as , which is formed from the homogeneous coordinates, is sought, then the estimated position of the eyeball 502 in camera coordinates 120 is obtained. For this purpose, the vector 580, also referred to as , can first be used to determine the vector 586, also referred to as . Now both points, i.e., the corner of the eye and the center of the eye 504 in the camera coordinate system 120, are known for the frame in which the labeling was done. This information can now be used to reconstruct the position of the eye center 504 from the position 312 of the corner of the eye in each frame, as described below with reference to Fig. 7. Fig. 6 shows a camera image 600 of an eye with eyelid markings according to an exemplary embodiment. A position 312 of the outer corner of the eye and a position 612 of the inner corner of the eye are shown. Furthermore, the upper eyelid's contour is indicated by points 620, 622, 624, 626, and the lower eyelid's contour is indicated by points 630, 632. The labeled eyelids are defined by points 620, 622, 624, 626, 630, and 632. Fig. 7 shows a schematic representation of an eye model according to an exemplary embodiment. The eye model is realized as a sphere 702 with the mean diameter of the human eye. A projection 740, for example of the labeled points 312, 612, 620, 622, 624, 626, 630, 632 shown in Fig. 6, onto the sphere 702 of the eye model is shown in sketch form. The points of intersection between the projection lines of the projection 740 and the surface of the sphere 702, which result from the projection, are interpolated. The eye opening is then calculated based on this interpolation. Fig. 8 shows a flowchart of a method for determining the center point of an eye and for determining the aperture of the eye according to an exemplary embodiment. The method can be carried out, for example, using a device as shown in Fig. 1. In an optional step 801, a reference image is acquired using a camera, and a reference image signal representing the reference image is provided. The reference image signal represents a reference image, captured by a camera, of at least a portion of a person's face while they are in a reference head pose, for example, looking directly into the camera. In step 803, reference pupil pixel coordinates and reference eye angle pixel coordinates are determined using the reference image signal. In step 805, homogeneous reference pupil coordinates are determined in a camera coordinate system using the reference pupil pixel coordinates, and homogeneous reference eye angle coordinates are determined in the camera coordinate system using the reference eye angle pixel coordinates.Optionally, in step 807, depth information associated with the angle of vision is determined using the reference image signal. In step 809, reference angle-of-vision camera coordinates are determined using the homogeneous reference angle-of-vision coordinates and depth information associated with the angle of vision. In step 811, reference eye center camera coordinates associated with the reference head pose are determined as the intersection of a reference vector formed from the homogeneous reference pupil coordinates and the homogeneous reference angle-of-vision coordinates, and a reference sphere. In step 821, reference eye center camera coordinates, as determined in step 811, are read in. In step 825, first eyelid pixel coordinates and second eyelid pixel coordinates are determined using a camera image signal. The camera image signal represents a camera image captured by the camera of at least a portion of the person's face, which now has a head pose that may differ from the reference head pose. In step 827, eye center camera coordinates associated with the head pose are determined using the reference eye center camera coordinates and a transformation rule that converts the reference head pose into the head pose. In step 829, the first eyelid pixel coordinates are projected onto a sphere to obtain first eyelid camera coordinates.Furthermore, the second eyelid pixel coordinates are projected onto the sphere to obtain second eyelid camera coordinates. Optionally, the procedure includes a step 831 in which a distance between the first eyelid camera coordinates and the second eyelid camera coordinates is determined. In a step 833, an eye-opening signal representing the eye opening is determined using the first eyelid camera coordinates and the second eyelid camera coordinates, or using the distance determined in step 831. Steps 821, 823, 825, 827, 829, 831, and 833 can be performed independently of steps 801, 803, 805, 807, 809, and 811. Specifically, steps 821, 823, 825, 827, 829, 831, and 833 can be performed once to determine suitable reference eye center camera coordinates for the individual. Steps 821, 823, 825, 827, 829, 831, and 833 can be repeated for each captured camera image to determine the individual's eye opening at the time the respective camera image was captured. If an embodiment includes an “and / or” connection between a first feature and a second feature, this is to be read as meaning that the embodiment according to one embodiment has both the first feature and the second feature, and according to another embodiment either only the first feature or only the second feature.

Claims

Method for determining a center point (504) of an eye (202) of an occupant of a vehicle (100), the method comprising the following steps: Determining (803) reference pupil pixel coordinates (140) and reference eye angle pixel coordinates (142) using a reference image signal (110) representing a reference image (300) captured by a camera (106) of at least a portion of a face of the occupant exhibiting a reference head pose, wherein the reference pupil pixel coordinates (140) represent a position of a center point (504) of a pupil (506) of the eye (202) of the occupant in the reference image (300) and the reference eye angle pixel coordinates (142) represent a position (312) of an eye angle of the eye (202) of the occupant in the reference image (300);Determine (805) homogeneous reference pupil coordinates (144) in a camera coordinate system (120) of the camera (106) using the reference pupil pixel coordinates (140) and homogeneous reference eye angle coordinates (146) in the camera coordinate system (120) of the camera (106) using the reference eye angle pixel coordinates (142); Determine (809) reference eye angle camera coordinates (148) using the homogeneous reference eye angle coordinates (146) and an eye angle associated depth information (588), wherein the reference eye angle camera coordinates (148) represent a position (312) of the eye angle in the camera coordinate system (120);and determine (811) reference eye center camera coordinates (150) assigned by the reference head pose as the intersection of a reference vector formed from the homogeneous reference pupil coordinates (144) and the homogeneous reference eye angle coordinates (146) and a reference sphere, wherein a center of the reference sphere is defined by the reference eye angle camera coordinates (148) and a diameter of the reference sphere by a mean eye diameter of a human eye (202). Method according to claim 1, comprising a step (801) of acquiring the reference image (300) using the camera (106) and a step of providing the reference image signal (110). Method according to claim 2, wherein the step (801) of detection is carried out at a time when the occupant is looking directly into a lens of the camera (106) with the eye (202). Method according to one of the preceding claims, comprising a step (807) of determining the depth information (588) associated with the angle of the eye using the reference image signal (110). Method for determining an eye opening of an eye (202) of an occupant of a vehicle (100), wherein the method comprises the following steps: Reading (821) reference eye center camera coordinates (150) which represent coordinates determined according to a method for determining a center point (504) of an eye (202) of an occupant of a vehicle (100) according to one of claims 1 to 4;Determine (825) first eyelid pixel coordinates (170) and second eyelid pixel coordinates (172) using a camera image signal (152) representing a camera image (600) captured by the camera (106) of at least a portion of the face of the occupant in a head pose, wherein the first eyelid pixel coordinates (170) represent the position of a point (620, 622, 624, 626) of an upper eyelid of the eye (202) of the occupant in the camera image (600) and the second eyelid pixel coordinates represent the position of a point (630, 632) of a lower eyelid of the eye (202) of the occupant in the camera image (600); Determine (827) eye center camera coordinates associated with the head pose using the reference eye center camera coordinates (150) and a transformation rule that converts the reference head pose into the head pose;Projecting (829) the first eyelid pixel coordinates (170) onto a sphere (702) to obtain first eyelid camera coordinates (174), and the second eyelid pixel coordinates onto the sphere (702) to obtain second eyelid camera coordinates (176), wherein a center point of the sphere (702) is defined by the eye center camera coordinates and a diameter of the sphere by the mean eye diameter of the human eye (202); Determining (833) an eye opening signal representing the eye opening of the eye (202) using the first eyelid camera coordinates (174) and the second eyelid camera coordinates (176). Method according to claim 5, comprising a step of determining a distance between the first eyelid camera coordinates and the second eyelid camera coordinates, wherein the eye opening signal is determined using the distance. Device that is configured to perform and / or control the steps of the method according to one of the preceding claims in corresponding units. A computer program configured to execute and / or control the steps of the method according to any one of claims 1 to 6 when the computer program is executed on a computer. Machine-readable storage medium on which the computer program according to claim 8 is stored.

Images