Methods for detecting visual impairment based on vehicle assistance systems

Abstract

Method for detecting visual impairment based on sensors of a vehicle assistance system, comprising the following steps: a) Acquisition of the area ahead in the direction of travel using vehicle-owned sensors, b) Generation of a 2D image plane (RI1 - RI5) from the sensor data, hereinafter referred to as a range image, c) Recording of a driver image by a driver observation camera (18), d) Detection of the driver's gaze direction from the driver image, e) Projection of the driver's gaze direction onto the range image (RI1 - RI5), f) Initial determination of one of the edge points closest to the driver's gaze direction in the first range image (RI1) as a fixed point (F) or acquisition of an already determined fixed point (F) in the subsequent range images (RI2 - RI5), g) Detection of the deviation of the driver's gaze direction (B1 - B5) from the fixed point (F), h) Repetition of steps a) to e) and g).provided that the deviation of the fixed point from the driver's line of sight in the range image remains within a defined tolerance range, i) creating a distribution (22) of the detected deviations (D1 - D5) from all repetition steps h), k) comparing the distribution (22) with correspondence data that includes a relationship between the distribution and a visual impairment, i) determining the visual impairment from the comparison, and m) adapting vehicle functions and / or the vehicle assistance system to the determined visual impairment.

Description

Advanced driver assistance systems (ADAS) are auxiliary devices in motor vehicles that use sensor systems such as radar, laser, lidar, ultrasound, or a combination of these sensors to monitor the area, at least in front of the vehicle, while driving. These systems are capable of detecting and locating objects in the front of the vehicle. They can also track the driver's eyes using a driver-monitoring camera, making it possible, for example, to detect increasing driver fatigue and issue a timely warning. WO 2018 / 066 695 A1 discloses a display device system for a vehicle that controls a head-up display (HUD) to show information to the driver as a virtual image in the driver's field of vision. Furthermore, JP 2021-59 242 discloses a head-up display system in which the display is individually adapted to the driver's visual perception based on data obtained through occupant monitoring. The present invention relates to a novel method by which it is possible to detect and counteract a driver's visual impairment using conventional vehicle assistance systems such as those mentioned above, for example by displaying a field of view on a screen which the driver cannot perceive as well due to his visual impairment. The method according to the invention is characterized by the features of claim 1. Advantageous further developments of the method are the subject of the dependent claims. Advantageous further developments and embodiments of the method according to the invention are also described in the description and in the drawing. According to the invention, in a first step a) the vehicle's surroundings are captured using the vehicle's own sensors. These are the sensors of the conventional vehicle assistance system already described above. In a second step b) from the data thus acquired, a 2D image plane is generated, which represents the surrounding environment data. This 2D image plane is hereinafter referred to as the range image. Simultaneously, in a further step c), the driver's image is captured by a driver observation camera. From this driver image, the driver's gaze direction is detected in a subsequent step d) and, in the next step e), projected into the range image, taking into account the position and orientation of the driver observation camera. In the next step (f), an edge point in the range image closest to the driver's line of sight is selected as an initial fixed point. Such an edge point is a high-contrast point that, for example, distinguishes an object, such as a vehicle ahead, from its surroundings. This initial fixed point is used for the subsequent determination of the visual impairment. In step (g), the deviation of the driver's line of sight from the fixed point is then recorded. This deviation includes both the distance of the driver's line of sight from the fixed point and the orientation of the vector between the driver's line of sight and the fixed point in the plane of the range image. The aforementioned steps a) to e) and g) are repeated cyclically according to procedure step h), preferably in increments of 100 milliseconds to 1 second. In each repetition, according to step g), the deviation of the driver's gaze direction from the initial fixed point determined in step f) is recorded, provided that the distance of the driver's gaze direction from the fixed point in the range image remains within a defined tolerance range. This repetition can also be carried out until a time limit is reached, for example, between 1 second and 10 seconds, in order to limit the duration of the visual impairment assessment to a reasonable extent. Slight wandering of the fix point during successive repetitions of the steps according to h) can be easily compensated for by software control of the ADAS. Such wandering of the fix point can occur, for example, if a vehicle ahead moves slightly laterally, such as when changing lanes. If the move of the fix point is too great in successive range images, the visual impairment assessment may be aborted. The deviations recorded in the repetition steps are statistically analyzed in step i) to determine their distribution, for example, as a variance. This distribution is then compared in step k) with correspondence data that establish a relationship between the distribution of deviations and visual impairment. This correspondence data can be obtained, for example, by determining the distribution of deviations for individuals with known visual impairment in experimental runs and creating a correspondence table that correlates the distribution of deviations with the visual impairment, represented, for example, by the diopter value. In this way, the driver's visual impairment can be easily determined in the next step l) by comparing the data in step k). Now, vehicle functions or display functions on a vehicle assistance system screen, or other functions of the vehicle assistance system in step m), can be adapted to the detected visual impairment. For example, the distance range that the visually impaired person cannot perceive well can be displayed on a vehicle screen to help them perceive the area ahead. Alternatively or additionally, warning signals can be emitted to alert the driver to the visual impairment. In summary, the method according to the invention enables the detection and determination of visual impairment using conventional advanced driver assistance systems (ADAS) and can initiate appropriate actions based on this, such as those mentioned above. It is clear that the ADAS, for its various tasks, monitors not only the area in front of the vehicle but also the side and rear. Today's ADAS systems aim to support the driver in many ways in their task of vehicle control. Examples include ACC, BSM, and navigation assistance. In most cases, these functions must be explicitly activated by the driver. The driver must therefore decide whether or not the function is needed. In recent years, there has been an increasing trend to measure the driver's vital signs and other parameters, and to suggest, based on available ADAS and comfort features, which functions might be relevant or helpful in which scenarios. To suggest such a pre-selection to the driver, it is necessary to be able to measure the driver's health status. In addition to this requirement, the ability to quantify and log the driver's health status (for example, using an EDR - Event Data Recorder) is also necessary for insurance and legal purposes.Such measurements also include determining visual impairments, such as myopia (nearsightedness) and hyperopia (farsightedness). This impairment can lead to an increased risk while driving – here, the MB ADAS function or Navistance could provide support. The latter can zoom in on the critical distance range – for example, 50–100 m – upon detection of myopia, thus enabling the system to provide visual support for distant scenarios that are difficult for the driver to judge due to their visual impairment. This visualization is illustrated below as an example and conceptual representation. The present invention disclosure therefore describes an approach that enables the detection of myopia / hyperopia in the vehicle by comparison with ADAS objects and, in turn, provides the input for downstream system and customer functions. Preferably, a 3D model of the surroundings is generated from the sensor data of the vehicle's own sensors when the area ahead is detected. This model is then computer-aidedly projected onto a 2D image plane, thus obtaining the range image of the area ahead. The driver's line of sight can then easily be projected onto such a 2D image plane, and the nearest edge points can be identified and determined as fixed points from this line of sight. In step f), an area of ​​high contrast closest to the driver's line of sight is initially determined as a fixed point in the range image. This area represents an object depicted in the range image. It is assumed that, at least for a certain period of time, the driver's gaze will remain fixed on this area of ​​high contrast, which could, for example, reflect the side edge or some feature of a vehicle ahead, a road sign, or the like. This object fixation of the driver's line of sight can then be effectively used to determine the visual impairment. The acquisition of the range image, i.e., the 2D image plane, has a certain resolution depending on the computing power and design of the vehicle's own sensor systems, for example, 1,000 pixels x 2,000 pixels. The tolerance range for detecting a correlation between the fixed point and the driver's line of sight is preferably between 3 and 100 pixels, preferably between 5 and 50 pixels, and particularly between 10 and 30 pixels. Such a deviation has proven to be very advantageous for evaluating the correlation between the driver's line of sight and the fixed point. The repetition rate of the repetitions according to step h) can be between 10 milliseconds and 1 second, preferably between 30 milliseconds and 200 milliseconds, depending on the computer-based system used. As previously described, the repetition stops if the correlation between the fixed point and the driver's gaze direction is lost. However, if the correlation persists for too long, the maximum recording time of the correlation can preferably be limited to a range between 0.5 and 3 seconds. Within this timeframe, enough repetitions are possible to obtain a statistically significant figure, such as the variance, from the distribution of deviations. This variance can then be compared with the corresponding data, thus providing a usable basis for determining the visual impairment. Determining the deviations within the repetitions allows for the determination of a fixation metric, provided there is a correspondence between the gaze point and the initial fixation point, namely by measuring the distance and its variance between the gaze focus and the fixation point for each image (frame) used within the repetitions, and determining the resulting metric (deviations within the tolerance range). As briefly described above, the correspondence data is preferably in the form of a mapping table, which translates the distribution of deviations in the gaze direction from the fixed point into a refractive error in diopters. In this way, the diopter value corresponding to the refractive error can be directly deduced from the distribution of the deviations. Preferably, the driver observation camera is calibrated relative to the range image before capturing the driver's image. This calibration aligns the driver observation camera and its image beam onto the driver's eyes with the 3D coordinate system of the space in which the vehicle is located. Since the exact position and orientation of the driver observation camera are thus known, the correct viewing direction of the driver in the 2D image plane can be directly determined from the identified driver's gaze direction. Such a calibration can be performed, for example, once before the journey, or before each capture of the driver's gaze direction, as it is possible that the driver may move relative to the camera during the course of repetitions, thus requiring the spatial alignment to be re-established with each image acquisition. In an advantageous embodiment of the invention, a 3D model of the surrounding environment is generated by the vehicle's own sensors, and this 3D model is subdivided into several 2D image planes, each reflecting a different depth of the 3D model. The driver's gaze direction is then correlated to the 2D image plane that has a fixed point correlated to that gaze direction. In this way, visual impairment can even be detected with respect to distance, which ultimately makes it possible to display a field of vision that is not perceived as well by the driver on a screen in the vehicle. In this case, for example, the different 2D image layers can be displayed in different colors to make them easier to identify during any subsequent processing or preparation of the data. The following terms are used synonymously: Vehicle assistance system - ADAS; Vehicle - Motor vehicle; Deviation - Fixation offset; Correspondence data - Mapping table; 2D image plane - Range image; It is understood that the embodiments described above can be combined with one another in any way according to the invention. The invention is described below by way of example with reference to the schematic drawing. In this drawing: Fig. 1 shows a side view of a motor vehicle with ADAS sensors, a driver's gaze direction camera, and a display; Fig. 2 shows a flowchart of the method according to the invention; Fig. 3 shows a representation of the detection of the deviation of the driver's gaze direction from a fixed point in 5 successive range images; Fig. 4 shows a detailed representation of the tolerance range of the first range image from Fig. 3; and Fig. 5 shows the distribution of the deviation of the driver's gaze direction from the fixed point in the five range images shown in Fig. 3, which distribution is then used to calculate the visual impairment by comparison with corresponding data. In the figures, identical or functionally equivalent elements are provided with the same reference symbols. Fig. 1 shows a motor vehicle 10 with a driver 12. The motor vehicle 10 is equipped with ADAS sensors 14, which can be, for example, ultrasonic, radar, lidar, or other known sensors. These ADAS sensors 14 detect the area ahead while the vehicle is driving. The ADAS sensors 14 are connected to a controller 16, which is capable of evaluating this environmental data in a manner specified in more detail. The motor vehicle 10 also has a driver observation camera 18, which is also connected to the controller 16. This driver observation camera 18 is directed towards the head of the person 12 and is thus able to detect the eyes and therefore the direction of the person 12's gaze. A display 20 is also connected to the controller 16, which can display various vehicle data, including data provided by the ADAS system. A flowchart of a preferred embodiment of the method according to the invention is shown in Fig. 2. In the method according to the invention, in a first step a), the area in front of the vehicle is detected by means of the sensors 14. In the next step b1), a 3D environment model is generated from the sensor data obtained according to step a), which in the subsequent step b2) is projected onto a 2D image plane or range image. In the following step b3), the driver observation camera is calibrated in relation to the person being recorded or their eyes, thereby enabling the system to determine the exact viewing direction in the coordinate system of the motor vehicle 10 from the detected viewing direction. In the following step c), the driver image is now captured by the driver observation camera and in step d) the driver's viewing direction is extracted from the driver image using the calibration data. In the following step e), the determined driver's gaze direction is projected into the 2D plane from step b2), which allows the exact viewpoint of the driver to be determined in the range image. Step f) now checks whether the range image created in step b2) is the first range image of the visual impairment evaluation procedure. If so, an edge point—that is, a point of high contrast—is initially determined from the range image within an evaluation area corresponding to the viewing direction. This edge point indicates that an object stands out from its surroundings. This edge point within the area is used as a fixed point for the subsequent assessment of the visual impairment. The evaluation area is defined and can range from 10 to 100 pixels in the range image, depending on the resolution. If it is not the first range image, the fixed point already determined in a previous range image is used. In the following step g), the deviation of the driver's viewing direction from the fixed point is recorded. In the subsequent first decision step h1), it is checked whether the distance between the fixed point and the driver's line of sight is within a predefined tolerance range. If this is the case, the procedure branches back to a second decision step h2), in which it is checked whether a maximum time duration for the assessment of the visual impairment or a maximum number of range images (frames) has been created during the procedure. If this is not the case, the procedure branches back to step a), otherwise the procedure continues to step i). If, in the first decision step h1), the tolerance range between the fixed point and the direction of gaze is exceeded, the procedure branches to step i), in which a statistical distribution of the deviations recorded in step g) in the frames is created, for example as variance. In the subsequent step k), this distribution is compared with corresponding data, for example a mapping table, which correlates the distribution with a refractive error. In step l), the refractive error is determined directly from this comparison. In the final step (m), the vehicle function or the ADAS function can then be adapted to the detected visual impairment. For example, the distance range to the vehicle in which the visual impairment is particularly pronounced can be displayed. Alternatively or additionally, in the event of excessive visual impairment, warning signals can be issued to encourage the driver to have their vision checked and corrected by a doctor. The visual impairment data can also be stored or used in other ways. This method thus allows for an effective and simple check of visual impairment within the capabilities of a conventional advanced driver assistance system (ADAS). Fig. 3 schematically shows the representation of a partial area of ​​an object, in this case a vehicle, in a section of five consecutively acquired range images or frames (RI1 - RI5), which are, for example, recorded at intervals of 100 ms. The initial fixed point F was determined at the bend of two mutually inclined outer lines of the vehicle's silhouette as a prominent point of high contrast within the evaluation area of ​​the driver's gaze direction B. This point is then captured as a fixed point in the subsequent range images. In the course of the further process, a tolerance range T, preferably circular, with a radius of 30 pixels, is defined around this fixed point F by the software, and the driver's gaze direction B1 - B5 is captured and stored with each repetition of process steps a) to g). For better illustration, the tolerance range T from the first range image RI1 of Fig. 3 is shown enlarged in Fig. 4. The fixed point F at the center of the tolerance range T and the driver's viewing direction B1 are visible. The distance D1 between the viewing direction B1 and the fixed point F, as well as its orientation in the plane of the range image, indicate the deviation of the driver's viewing direction from the fixed point. This deviation D1-D5 is recorded and stored with each iteration, i.e., in each successively created range image RI1-RI5. The distribution 22 of gaze directions B1–B5 from the five range images RI1–RI5 in Fig. 3 is schematically represented in Fig. 5. The tolerance range T around the fixed point F is again shown, along with the gaze directions B1–B5 from all five consecutively recorded range images or frames RI1–RI5 from Fig. 3 and the corresponding deviations D1–D5 of gaze direction B1–B5 from the fixed point B. This distribution 22 of deviations D1–D5 takes into account both their magnitude and their orientation within the tolerance range T. This distribution is then recorded and statistically evaluated, e.g., as variance. This statistical distribution is used, as described above, to determine the refractive error by comparison with corresponding data, such as a mapping table that correlates the statistical distribution of deviations with a diopter value of a refractive error.Following this, appropriate measures can be taken, as described in step m, such as displays, warnings, etc. The invention is not limited to the illustrated embodiment, but can be varied within the scope of protection of the attached claims.

Claims

Method for detecting visual impairment based on sensors of a vehicle assistance system, comprising the following steps: a) Acquisition of the area ahead in the direction of travel using vehicle-owned sensors, b) Generation of a 2D image plane (RI1 - RI5) from the sensor data, hereinafter referred to as a range image, c) Recording of a driver image by a driver observation camera (18), d) Detection of the driver's gaze direction from the driver image, e) Projection of the driver's gaze direction onto the range image (RI1 - RI5), f) Initial determination of one of the edge points closest to the driver's gaze direction in the first range image (RI1) as a fixed point (F) or acquisition of an already determined fixed point (F) in the subsequent range images (RI2 - RI5), g) Detection of the deviation of the driver's gaze direction (B1 - B5) from the fixed point (F), h) Repetition of steps a) to e) and g).provided that the deviation of the fixed point from the driver's line of sight in the range image remains within a defined tolerance range, i) creating a distribution (22) of the detected deviations (D1 - D5) from all repetition steps h), k) comparing the distribution (22) with correspondence data that includes a relationship between the distribution and a visual impairment, i) determining the visual impairment from the comparison, and m) adapting vehicle functions and / or the vehicle assistance system to the determined visual impairment. Method according to claim 1, characterized in that a 3D environment model is generated from the sensor data of step a), and that the 3D environment model is projected onto the at least one 2D image plane c. Method according to claim 1 or 2, characterized in that a fixed point (F) in the range image (RI1 - RI5) is determined as an area of ​​high contrast closest to the driver's direction of view (B1 - B5) due to an object reproduced in the range image (RI1 - RI5). Method according to one of the preceding claims, characterized in that the tolerance range for the fixed point and / or the driver's viewing direction is between 3 px and 100 px, preferably between 5 px and 50 px, in particular between 10 px and 30 px. Method according to one of the preceding claims, characterized in that the correspondence data, e.g. a mapping table, translates the dispersion of the deviations into diopters. Method according to one of the preceding claims, characterized in that the driver observation camera (16) is calibrated in relation to the vehicle coordinate system before step c). Method according to one of the preceding claims, characterized in that the repetition of steps d) to g) according to step i) takes place at a time interval of 10 ms to 100 ms. Method according to one of the preceding claims, characterized in that the repetitions of steps d) to g) according to step i) are carried out until a predetermined time limit is reached. Method according to one of the preceding claims, characterized in that the projection of the 3D environment model according to step c) is carried out onto several 2D image planes, wherein the content of the different 2D image planes reflects the different depth of the 3D environment model. Method according to claim 9, characterized in that different colors are assigned to the different 2D image planes. Method according to one of the preceding claims, characterized in that the vehicle's own sensors (14) are ADAS sensors, in particular vision, lidar, radar, USS or a combination of several of these sensors. Method according to one of the preceding claims, characterized in that mapping tables are used as correspondence data which have an empirically determined distribution, in particular variance, of the deviations depending on a visual impairment parameter, in particular the diopter number.

Images