Methods for detecting astigmatism and vehicle

Abstract

Method for detecting astigmatism based on a visual detection of the eyes (1) of a person, characterized in that an in-vehicle computing unit determines a degree of astigmatism by comparing a reflection (3) of the environment extracted from a corneal reflection image (CRI) of a vehicle occupant (2) with an environment camera image (4), wherein the corneal reflection image (CRI) and the environment camera image (4) show at least partially the same section of the environment, wherein the computing unit performs the following process steps: - Recording of the eye area of ​​the vehicle occupant (2) using an interior vehicle camera; - Extracting the corneal reflection image (CRI) from the camera image captured by the internal camera; - Determining the position of the eyes (1) of the vehicle occupant (2) relative to the vehicle interior and determining the direction of gaze (5) of the vehicle occupant (2) by processing the camera image recorded by the interior camera; - Generating the surround view camera image (4) using a vehicle-integrated surround view camera (10); - Taking into account the installation position and orientation of the interior camera and the surround camera (10) on the vehicle as well as the position of the eyes (1) and the direction of gaze (5): Overlay at least a subset of the image content of the corneal reflection image (CRI) and the surround camera image (4); - Segmentation of the corneal reflection image (CRI) into a multitude of image segments (6); - Performing an image registration for each image segment (6) onto the superimposed image content of the ambient camera image (4), whereby each image segment (6) is assigned a similarity value; and - Determining the degree of astigmatism depending on the similarity values ​​assigned to each image segment (6).

Description

[0001] The invention relates to a method for detecting astigmatism of the type defined in the preamble of claim 1 and to a vehicle of the type defined in the preamble of claim 6.

[0002] Astigmatism, also known as astigmatic vision, is a specific refractive error of the eye. It results from an astigmatism of the cornea. The light rays emanating from a viewed object are not focused at a single point on the retina, but rather along a focal line, which can cause the viewed objects to appear blurry or distorted.

[0003] Astigmatism can be detected by mapping the corneal topography. This is typically done by projecting numerous concentric circles onto the cornea and recording the projection path with a camera. The distortion of the circles reveals information about the corneal topography.

[0004] Depending on the severity of a person's astigmatism, corrective measures are necessary. Otherwise, there is a risk that a person with astigmatism will not be able to adequately perceive their surroundings, other road users, and the vehicle's instruments while driving.

[0005] German patent application DE 10 2017 202 257 A1 discloses a display device for a motor vehicle and a method for individually adjusting a display. The document describes a display device in a vehicle that can be adapted to the individual visual requirements of different vehicle occupants. A rotary control allows the display properties to be adjusted so that even people with myopia, hyperopia, and / or astigmatism can perceive the displayed content without other aids, such as glasses. The actual degree of visual impairment is not directly determined; rather, the display is simply modified by manually operating the rotary control until the user can see the content clearly.Each manual user setting can be assigned to a personal user profile, allowing for quick, easy and repeatable changes to the display device settings.

[0006] DE 10 2019 208 474 A1 discloses a method for the optical measurement of an object, in which a pattern generator produces a planar pattern that varies in at least one optical property such that, at least in some areas, a multitude of different points or a multitude of different groups of points are distinguishable from one another. At least parts of the pattern are reflected by a reflective surface of the object as a reflected pattern onto a detector of a camera unit, the reflected pattern being converted into a camera image by the detector. A relationship between points of the camera image and corresponding points of the pattern can be described by a correspondence function that depends on geometric properties of the object's reflective surface.One of the geometric properties of the object's reflecting surface is determined by using differential geometric properties of a transformation given by the correspondence function.

[0007] From US Patent 7,401,920 B1, an eye-tracking system is described that determines a user's line of sight based on the relative position between the center of the pupil and a reference point. The system includes an image detector that captures an image of the eye, a pupil-illuminating light source that illuminates the user's pupil, a reference light source that illuminates another part of the user's face as a reference point, and an image processor that analyzes the captured eye image. The image processor can detect physiological conditions such as fatigue, loss of consciousness, strabismus, or astigmatism based on pupil movement.

[0008] The present invention is based on the objective of providing a method for detecting astigmatism, which makes it possible to make the operation of a vehicle even safer.

[0009] According to the invention, this problem is solved by a method for detecting astigmatism with the features of claim 1. Advantageous embodiments and further developments, as well as a vehicle produced by carrying out the method, are described in the dependent claims.

[0010] A generic method for detecting astigmatism based on a visual detection of a person's eyes is further developed according to the invention in that a vehicle-internal computing unit determines a form of astigmatism by comparing a reflection of the environment extracted from a corneal reflection image of a vehicle occupant with an environment camera image, wherein the corneal reflection image and the environment camera image show at least partially the same section of the environment.

[0011] The method according to the invention makes it possible to detect whether a vehicle occupant suffers from astigmatism and how severe the visual impairment is. It is not necessary to project patterns onto the occupant's eye, such as the aforementioned concentric circles, since the reflection of the surroundings is used for this purpose. The method according to the invention can therefore be carried out while the person is using the vehicle, as the occupant, for example, the driver, is thus not dazzled by dedicated light pattern projections. The information obtained using the method according to the invention, namely whether the vehicle occupant suffers from astigmatism or not, and if so, how severe the astigmatism is, can be used in the vehicle context to provide various vehicle functions or driver assistance systems, which will be discussed in more detail below.

[0012] The method according to the invention provides that the vehicle's internal computing unit performs the following process steps: - Recording of the vehicle occupant's eye area with an in-vehicle interior camera; - extracting the corneal reflection image from the camera image captured by the internal camera; - Determine the position of the vehicle occupant's eyes relative to the vehicle interior and determine the direction of the vehicle occupant's gaze by processing the camera image captured by the interior camera; - generating the surround view camera image using a vehicle's own surround view camera; - taking into account the installation position and orientation of the interior camera and the surround camera on the vehicle as well as the position of the eyes and the direction of gaze: to superimpose at least a subset of the image content of the corneal reflection image and the surround camera image; - Segmentation of the corneal reflection image into a large number of image segments; - perform an image registration for each image segment onto the superimposed image content of the surrounding camera image, whereby each image segment receives a similarity value; - Determining the degree of astigmatism depending on the similarity values ​​assigned to each image segment.

[0013] The process steps can be executed by a computing unit in a vehicle equipped with standard hardware components. Modern vehicles are typically fitted with interior and exterior cameras. Only the necessary program code needs to be implemented in the computing unit to control the interior and exterior cameras and process the respective camera images. The invention also includes the corresponding computer program and a computer-readable storage medium containing the computer program.

[0014] The processing unit can control the interior camera and / or the surrounding camera itself to capture the respective camera images. However, the interior camera and / or the surrounding camera can also be used in the context of various driver assistance systems and already record corresponding camera images. These camera images can be used by the processing unit to execute the method according to the invention.

[0015] To detect astigmatism, the processing unit compares the reflection of the surroundings detected in the corneal reflection image with the corresponding representation of the respective area captured by the surround-view camera. Due to the uneven corneal topography, distortions occur in the reflection of the surroundings in the corneal reflection image. These distortions can be detected by the processing unit by comparing the representation of the surroundings from the surround-view camera image. The processing unit is able to locate the corresponding distortions in the corneal reflection image, and thus on the eye of the vehicle occupant, and also determine the degree of distortion. This allows the processing unit to recognize the severity of the astigmatism. To compare the corneal reflection image and the corresponding section from the surround-view camera image, the area of ​​intersection of both images must first be aligned.The image content of the corneal reflection image can be mapped onto the image from the surrounding camera, or vice versa.

[0016] The corneal reflection image is then divided into numerous image segments. Each image segment represents a section of the surface of the eye of the vehicle occupant being examined. The segmentation of the corneal reflection image can follow various geometric patterns. For example, the image segments can be square or rectangular and arranged two-dimensionally in rows and columns. The individual image segments can also be formed by circular segments that extend from an inner to an outer radius over a portion of the circumference. Other geometric shapes and arrangements are also possible.

[0017] During image registration, the processing unit compares the image content of each image segment with the surrounding camera image, or a section thereof. Various established metrics can be used for this comparison to determine visual similarity or differentiation between the image content, which will be discussed in more detail later. This allows the processing unit to determine which part of the surrounding camera image a given image segment corresponds to and the degree of similarity. This, in turn, allows the aforementioned similarity values ​​to be assigned to the corresponding regions of the surrounding camera image. Based on the spatial distribution of these similarity values ​​across the surrounding camera image, or the corneal reflection image, the processing unit then determines the degree of astigmatism.For this purpose, reference distributions can be predefined in the processing unit, which are used to evaluate the respective application. Thus, if certain similarity values ​​are exceeded or fallen below in specific image regions, this is associated with a specific degree of astigmatism.

[0018] According to a further advantageous embodiment of the method according to the invention, the processing unit maps the similarity values ​​assigned to the image segment onto a heatmap and generates a topographic representation of the corneal surface based on the heatmap. A heatmap thus enables a particularly fast and intuitive assessment of the degree of astigmatism.

[0019] According to a further advantageous embodiment of the method according to the invention, the processing unit transforms the corneal reflection image and the image from the surrounding camera into a common reference coordinate system. This facilitates the location of the corresponding sub-areas of the environment that are depicted in the corneal reflection image and in the image from the surrounding camera, as well as the superimposition of the respective image content. Preferably, a coordinate system fixed relative to the vehicle is used as the reference coordinate system.

[0020] According to a further advantageous embodiment of the method according to the invention, the similarity values ​​are determined based on the structural similarity index. The structural similarity index is also referred to as the "Structural Similarity Index Measure (SSIM)." This is a proven method for estimating the perceived quality of image content. SSIM is used to measure the similarity between two images. The SSIM index can be generalized, taking into account, in particular, the luminance, contrast, and structure of the image content of the images being compared when comparing similarity. Using SSIM, the similarity values ​​can be reliably determined.

[0021] A further advantageous embodiment of the method according to the invention provides that the corneal reflection image is rectified after generation and before further processing to compensate for a spherical projection. Human eyes are approximately spherical. This results in the reflection of the surroundings on the eyes being correspondingly curved. This distortion can be compensated for to better align the image content of the corneal reflection image with the image from the surroundings camera. Distortions of the image content arising from differing corneal topography are retained.

[0022] In a vehicle comprising an interior camera, an environment camera and a processing unit, the interior camera, the environment camera and the processing unit are, according to the invention, configured to carry out a method for detecting astigmatism based on a visual detection of a person's eyes, in which an in-vehicle processing unit determines the degree of astigmatism by comparing a reflection of the environment extracted from a corneal reflection image (CRI) of a vehicle occupant with an environment camera image, wherein the corneal reflection image (CRI) and the environment camera image (4) show at least partially the same section of the environment, wherein the processing unit is also configured to transmit the degree of astigmatism as an input variable to a control unit, wherein the control unit is configured to adapt its behavior taking into account the degree of astigmatism.As already described at the beginning, the degree of astigmatism determined using the method according to the invention can be transmitted as an input variable to a control unit.

[0023] The vehicle in question can be any road vehicle such as a car, truck, van, bus, or similar. Generally, it could also be a rail vehicle, watercraft, or aircraft.

[0024] An advantageous further development of the vehicle according to the invention provides that the control unit is configured to control a display device in order to adjust the display content shown on the display device to compensate for astigmatism and / or to control an environmental sensor of the vehicle in order to selectively direct it to a region of the environment or to increase its detection sensitivity in the region where, taking into account the direction of the vehicle occupant's gaze, their visual impairment is present. This means that the vehicle is automatically able to recognize whether a vehicle occupant, in particular the driver, suffers from astigmatism and how severe the visual impairment is. The vehicle then uses this information to automatically correct the display content on the display device.Furthermore, the vehicle's environmental sensors can be specifically directed towards those areas of the vehicle that are currently not sufficiently perceived by the occupant, taking into account their line of sight. Thus, if the occupant is unable to adequately recognize objects in the corresponding area due to visual impairment, at least the vehicle itself will be able to. Suitable environmental sensors for this purpose include cameras, LiDAR sensors, ultrasonic sensors, and radar sensors.

[0025] The display device could be, for example, the instrument cluster, the head unit display, another display such as a dedicated passenger display or a head-up display (HUD).

[0026] According to a further advantageous embodiment of the vehicle according to the invention, the display device is also configured to highlight objects located in the spatial region within a virtual or augmented representation of the environment. As already mentioned, due to visual impairment, the vehicle occupant may not be able to adequately perceive objects located in the spatial region when looking directly at their surroundings. For example, the objects may be too blurred and / or distorted. According to the invention, a virtual representation or an augmented representation of the environment is displayed via the display device. Thus, a so-called "augmented reality display," also referred to as "AR," can be displayed on a display device designed as a head-up display (HUD).Objects that are not sufficiently visible to the vehicle occupant when looking around are highlighted on the display. This makes it easier for the occupant to perceive the objects, thus improving driving safety. This minimizes the risk of the occupant, especially the driver, overlooking these objects. Using a head-up display (HUD) is particularly advantageous because the objects to be highlighted can be marked using a contact-like display. This further enhances the occupant's visibility of the objects.

[0027] Preferably, the display device is further configured to make objects to be highlighted on the display device more transparent compared to objects that are not highlighted. This further improves the visibility of the objects for the vehicle occupant.

[0028] The vehicle according to the invention, using the method according to the invention, is able to detect astigmatism and to control corresponding control units for the detection and highlighting of corresponding environmental objects, such as road users, lane markings, traffic signs, road boundaries, and the like. This requires components that the vehicle already possesses, i.e., those that are standard equipment. This allows for the simple integration of the method according to the invention into suitable vehicles. Detecting the degree of astigmatism and highlighting the objects accordingly is possible regardless of the driving situation or scenario and weather conditions. Detecting the degree of astigmatism and controlling the vehicle's control units accordingly can be repeated cyclically within a control loop.The information generated by the processing unit can be used as input for a variety of different driver assistance systems. No additional human intervention is required, such as the manual adjustment of the display to compensate for imperfections, a common practice in the prior art. This simplifies vehicle operation and increases user comfort. In particular, it avoids situations where a vehicle occupant forgets to manually adjust the display to their needs, thus preventing them from perceiving the displayed information.

[0029] Further advantageous embodiments of the inventive method for detecting astigmatism and of the inventive vehicle also result from the embodiment examples, which are described in more detail below with reference to the figures.

[0030] This shows: Fig. 1 a flowchart of a method according to the invention for detecting astigmatism in a vehicle; and Fig. 2. A detailed description of the respective process steps.

[0031] Fig. 1 illustrates in conjunction with Fig. 2 describes the process of a method according to the invention for detecting astigmatism in a vehicle. In step 101, the eye area of ​​a vehicle occupant 2 is recorded using an interior camera.

[0032] In a subsequent step 102, a so-called corneal reflective image (CRI) is extracted from this camera image. In the Fig. 2b) The exemplary embodiment shown depicts a reflection 3 of the surroundings, a multi-story building. Due to the near-spherical shape of the eye, the reflection 3 appears curved. This curvature can be corrected by spherical rectification of the image content.

[0033] Subsequently, in step 103, the position of the eyes 1 of the vehicle occupant 2 relative to the vehicle interior is determined. The gaze direction 5 of the vehicle occupant 2 is also determined.

[0034] In the subsequent step 104, the vehicle uses its own surround-view camera to record 10 images. The recording and processing of the corresponding camera images from the interior camera and the surround-view camera 10 can also be carried out in reverse order, preferably simultaneously.

[0035] In step 105, the respective camera images taken by the interior camera and the exterior camera 10, or the extracted corneal reflection image CRI, are transferred into a common reference coordinate system 8.

[0036] In step 106, the processing unit, taking into account the installation position and orientation of the interior camera and the ambient camera 10 on the vehicle, as well as the position of the eyes 1 and the direction of gaze 5, aligns the overlapping areas of the image content of the corneal reflection image (CRI) and the ambient camera image 4. This is in Fig. 2f) indicated by a double arrow.

[0037] In step 107, the processing unit segments the corneal reflection image CRI into a large number of image segments 6. Fig. Figure 2g) shows an example of a scene in a row of buildings in a large city. For clarity, image segments 6 are shown particularly large. In reality, image segments 6 can be much smaller, and their number correspondingly large. The shape of image segments 6 can also be arbitrarily polygonal or circular segments.

[0038] In step 108, the processing unit performs an image registration for each image segment 6 onto the superimposed image content of the ambient camera image 4. This is preferably done using the structural similarity index to find corresponding image content. A similarity value is assigned to each image segment 6. This similarity value describes the correspondence between the respective image contents. If the vehicle occupant 2 has astigmatism, the reflection 3 of the environment is distorted in places compared to the image content of the ambient camera image 4. Different similarity values ​​are then determined for the image segment 6 exhibiting this distortion compared to undistorted image content. As shown in the lower image of the Fig. At position 2h shown in the dashed line at the top right, the best match for image segment 6 of the corneal reflection image CRI is found with the surrounding camera image 4; the similarity value S is, for example, S=0.81.

[0039] How Fig. Figure 1 shows that the process step 108 is carried out multiple times until a suitable section of the camera image 4 has been found for each image segment 6.

[0040] In step 109, the processing unit maps the similarity values ​​assigned to each image segment 6 onto a heatmap 7. The representation of the in Fig. The heatmap shown in section 2i) is based on a much finer resolution of the image segments 6 than that shown in the example image of the row of houses. This allows a topographic representation of the corneal surface to be generated based on the heatmap.

[0041] The heatmap 7, or even just information about the degree of astigmatism, particularly considering the similarity values ​​assigned to the respective image segments 6, can be transmitted from the processing unit to a downstream control unit of the vehicle as an input. This allows, for example, the display on a screen in the vehicle to be adjusted so that objects in the environment that appear blurry or distorted when viewed by the vehicle occupant 2 are clearly visible when the occupant 2 looks at the screen. Furthermore, the vehicle's environmental sensors, such as surround-view cameras, ultrasonic sensor systems, radar sensor systems, and / or LiDARe, can be specifically directed at the corresponding spatial region 9, or their detection sensitivity in these spatial regions 9 can be increased.

[0042] For example, in step 110, the display content of a HUD can be adjusted. This shows Fig. 2 j) a contact-analogous marking of other road users 11 on a corresponding HUD display 12 for the vehicle occupant 2. The vehicles ahead located in spatial region 9 are highlighted so that the driver can perceive these vehicles better. The corneal curvature of the driver's eye(s) 1 is also indicated by a dashed line 13. In the Fig. In the exemplary embodiment shown in 2j), only those vehicles that cross line 13 are highlighted.

[0043] After step 110 has been executed, the chain of effects can proceed as described in Fig. Step 101, indicated by an arrow pointing towards step 101, can be repeated. A cycle time can be predefined.

Claims

[1] Method for detecting astigmatism based on a visual assessment of a person's eyes (1), characterized by , that an in-vehicle computing unit determines the degree of astigmatism by comparing a reflection (3) of the environment extracted from a corneal reflection image (CRI) of a vehicle occupant (2) with an environment camera image (4), wherein the corneal reflection image (CRI) and the environment camera image (4) show at least partially the same section of the environment, and wherein the computing unit performs the following procedural steps: - Recording of the eye area of ​​the vehicle occupant (2) using an interior vehicle camera; - Extracting the corneal reflection image (CRI) from the camera image captured by the internal camera; - Determining the position of the eyes (1) of the vehicle occupant (2) relative to the vehicle interior and determining the direction of gaze (5) of the vehicle occupant (2) by processing the camera image recorded by the interior camera; - Generating the surround view camera image (4) using a vehicle-integrated surround view camera (10); - Taking into account the installation position and orientation of the interior camera and the surround camera (10) on the vehicle as well as the position of the eyes (1) and the direction of gaze (5): Overlay at least a subset of the image content of the corneal reflection image (CRI) and the surround camera image (4); - Segmentation of the corneal reflection image (CRI) into a multitude of image segments (6); - Performing an image registration for each image segment (6) onto the superimposed image content of the ambient camera image (4), whereby each image segment (6) is assigned a similarity value; and - Determining the degree of astigmatism depending on the similarity values ​​assigned to each image segment (6). [2] Method according to claim 1, characterized by , that the computing unit maps the similarity values ​​assigned to each image segment (6) onto a heatmap (7) and generates a topographic representation of the corneal surface based on the heatmap (7). [3] Method according to claim 1 or 2, characterized by , that the computing unit transforms the corneal reflection image (CRI) and the environment camera image (4) into a common reference coordinate system (8). [4] Method according to any one of claims 1 to 3, characterized by that the similarity values ​​are determined based on the index of structural similarity. [5] Method according to any one of claims 1 to 4, characterized by , that the corneal reflection image (CRI) is rectified after its generation and before its further processing to compensate for a spherical projection. [6] Vehicle comprising an interior camera, an exterior camera (10) and a computing unit, characterized by , that the interior camera, the environment camera (10) and the computing unit are configured to perform a method for detecting astigmatism based on a visual detection of the eyes (1) of a person, wherein an in-vehicle computing unit determines a degree of astigmatism by comparing a reflection (3) of the environment extracted from a corneal reflection image (CRI) of a vehicle occupant (2) with an environment camera image (4), wherein the corneal reflection image (CRI) and the environment camera image (4) show at least partially the same section of the environment, and the computing unit is further configured to transmit the degree of astigmatism as an input to a control unit, wherein the control unit is configured to adapt its behavior taking into account the degree of astigmatism. [7] Vehicle according to claim 6, characterized by , that the control unit is configured to control a display device in order to adjust a display content shown via the display device to compensate for astigmatism and / or to control an environment sensor of the vehicle in order to direct it specifically to a spatial region (9) of the environment or to increase its detection sensitivity in the spatial region (9) in which, taking into account the direction of gaze (5), the vehicle occupant (2) has his visual impairment. [8] Vehicle according to claim 7, characterized by , that the display device is configured to highlight objects located in the spatial region (9) in a virtual or extended representation of the environment. [9] Vehicle according to claim 8, characterized bythat the display device is configured to make objects to be highlighted on the display device more transparent compared to objects not to be highlighted.

Images