Vision correction display devices, eye-tracking systems, and methods for compensating for visual impairment.

Abstract

The present invention relates to a display device for displaying images for visually impaired users. The display device comprises a pixelated display, a processor, an eye-tracking system and a microlens array. The present invention also relates to a method for displaying images for visually impaired users on a display device. The present invention relates to a processor for operating a display device adapted to display images for visually impaired users, and to a computer program for implementing the method of the present invention.

Description

Technical Field

[0001] The present invention relates to a display device for displaying images for users with visual impairments. The present invention also relates to a method for displaying an image for a user with a visual impairment on a display device. The present invention further relates to a processor for operating a display device adapted to display an image for a user with a visual impairment, and a computer program for executing the method of the present invention.

Background Art

[0002] The world is rapidly digitizing, and life without an electronic display is unimaginable. Smart devices incorporating such electronic displays, such as mobile phones and tablets, have changed the lives of many people and enabled a new generation of workers to be highly productive in the workplace.

[0003] There are people who experience visual impairments such as myopia, presbyopia, and cataracts. A significant portion of the total population is at risk of suffering from visual impairments. For example, people in their mid-30s and older are at a higher risk of experiencing age-related visual impairments.

[0004] These visual impairments make it more difficult to focus on objects, such as on an electronic display. This is an obstacle for these people in their daily lives. Some of these electronic display operating systems have features to address some of the problems associated with visual impairments. For example, the size, brightness, and contrast may be adjusted so that the image is easier for people to see. Other features include displaying the text of the image in bold, changing the color palette, and using audio technology.

[0005] WO 2019 / 171342 discloses a digital display device for rendering input images so that they can be viewed by a visually impaired person. The display device comprises a digital display medium having an array of pixels, a microlens array disposed relative to the digital display, and a hardware processor capable of rendering an image on the digital display medium and thereby applying specified image perception adjustments to at least partially address the visually impaired person. The dimensions of each microlens are selected to minimize the spot size on the viewer's retina generated by the digital display medium.

[0006] U.S. Patent Application No. 2015 / 0262424 discloses a head-mountable device provided for enhancing a simultaneous, real-world image of an object in a real-world environment using a field-of-view display system that enables depth and focus identification. The device may include a light-emitting display engine, a viewing position element, and a microlens array.

[0007] U.S. Patent Application No. 2020 / 0272232 discloses a light field display, a modified pixel rendering method, and a computer-readable medium for the modified pixel rendering method, as well as embodiments of a vision correction system and method for addressing astigmatism or similar conditions using the light field display and the modified pixel rendering method. In one embodiment, a computer-implemented method is provided for automatically adjusting the user perception of an input image rendered on a digital display via a set of pixels of the digital display.

[0008] A known drawback of digital display devices in this field is that when a user moves relative to the display device, they view the image from different angles, which negatively impacts the viewing experience and reduces the user's ability to focus on the image. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] WO 2019 / 171342 [Patent Document 2] U.S. Patent Application No. 2015 / 0262424 [Patent Document 3] U.S. Patent Application No. 2020 / 0272232 [Overview of the project] [Means for solving the problem]

[0010] The present invention provides a display device for displaying images for visually impaired users, which mitigates the aforementioned drawbacks. The display device according to the present invention is described in claim 1.

[0011] The display device according to the present invention is adapted to display images for users with visual impairments. The display device comprises a pixel configuration display having an array of pixels. Each pixel of the pixel configuration display comprises several subpixels, for example, three subpixels, for example, red, green, and blue subpixels. By allowing the subpixels to emit light having a specific intensity, i.e., a corresponding subpixel value, the pixels may appear to emit light that changes color. The pixel configuration display of the display device is operable to display an image to the eyes of a user positioned at a viewing position relative to the location of the pixel configuration display.

[0012] The display device further comprises a processor for operating the pixels of a pixel-configured display to display an image on the pixel-configured display. For example, the processor may be the processor of a smart device such as a smartphone or smartwatch. The processor may be a processor dedicated to the functions of the display device. The processor is operable to store information about the user's eyes, such as the focal length of the eyes and the size of the eyes or the size of the pupil. For example, when the display device is first turned on, the user may be prompted to take some tests to determine the focal length and size of the user's eyes. It may also be possible to manually store and change the focal length and size of the user's eyes. In embodiments, the processor may be adapted to store additional information about the user, such as additional information about the user's visual impairment. In one embodiment, the processor may be further adapted to store information about different users. The processor may be adapted to recognize different users, for example by facial recognition, and load the correct stored focal length and eye size.

[0013] The display device further comprises an eye-tracking system adapted to determine changes in the viewing position of the user's eyes relative to the position of the pixel-configured display. The eye-tracking system may be integrated into, for example, the camera system of a smartphone or virtual reality headset equipped with the display device. The eye-tracking system may also be a dedicated eye-tracking system. The eye-tracking system may continuously monitor changes in the user's eye position, or it may monitor changes in the user's eye position at regular or irregular intervals, for example, every two seconds or every five seconds.

[0014] The display device further comprises a microlens array provided on a pixelated display. The microlens array comprises microlenses arranged in a two-dimensional plane parallel to the pixelated display. The microlens array covers, preferably completely covers, the pixelated display. The microlens array is configured to project the light emitted by the pixelated display toward the user's eyes at the viewing position.

[0015] The display device according to the present invention includes an eye-tracking system adapted to determine changes in the viewing position of the user's eyes relative to the position of a pixel-configured display. The eye-tracking system may track the position of the user's eyes and, therefore, the position of the user's head relative to the display device. This enables the display device to determine when the user's viewing position relative to the display device has changed.

[0016] The processor may determine the subpixel values ​​of a pixel configuration display to render a corrected image that appears more in focus to a visually impaired user. The processor may be adapted to determine the subpixel values ​​of a pixel configuration display based on the position of a virtual image plane.

[0017] The position of the virtual image plane is related to the focal length of the user's eye. If the user has perfect vision, the virtual image plane is located on the user's retina or, alternatively, on the pixel-based display. However, if the user does not have perfect vision, the virtual image plane is located in front of or behind the user's retina or, alternatively, in front of or behind the pixel-based display. Whether the position of the virtual image plane is assumed to be near the user's retina or near the pixel-based display depends on the method used by the processor to determine the pixel values. Either relative position may be used in this invention.

[0018] The processor is configured to determine a first position of the virtual image plane, which corresponds to the first viewing position of the user's eye. The relative position of the virtual image plane may change if the relative position of the user's eye to the pixel-based display changes. For example, the virtual image plane may move parallel to the pixel-based display when the user's eye moves parallel to the pixel-based display. If the subpixel values ​​of the display do not change in response to the translation, the corrected image may appear somewhat out of focus to the user.

[0019] The first position of the virtual image plane is determined by the processor by comparing the focal length of the user's eye with the first viewing position of the user's eye. The focal length of the user's eye is related to the user's visual impairment. The focal length may be measured by conventional means and stored in the processor for use in the present invention. The processor may be configured to determine the focal length. For example, from the focal length and the first viewing position, the processor may determine whether the lens of the user's eye projects a sharp image onto a plane in front of or behind the user's retina. The position of this plane is the position of the virtual image plane. Alternatively, from the focal length and the first viewing position, the processor may determine whether the user can focus on a plane in front of or behind the pixel-based display. In this case, the position of this plane is the position of the virtual image plane.

[0020] A virtual image is located on a virtual image plane. The virtual image corresponds to an image displayed on a pixel-based display. For example, the virtual image may be a mirror image version of the image displayed on the display. The virtual image comprises virtual pixels, each having a corresponding virtual pixel value. The virtual pixel value determines the color of each virtual pixel. The virtual pixels are located on the virtual image plane.

[0021] The processor is configured to associate subpixels of a pixel-based display with corresponding first virtual pixel values ​​on a virtual image plane. The processor may also be determined to determine the optical path of the light rays emitted by the corresponding subpixels using methods such as ray tracing. The optical path determined is the path between the corresponding subpixel, a microlens array, the retina of the user's eye, and the virtual image plane. The position on the virtual image plane where the light rays cross is the position of the first virtual pixel.

[0022] The processor is configured to determine a first value for a subpixel by comparing the subpixel's value to the value of a first virtual pixel. The value of the first virtual pixel is determined by the virtual image. The processor is configured to operate the corresponding subpixel based on the first value determined for each subpixel when the user's eyes are in a first viewing position. By repeating these steps for each subpixel of the pixel-configured display, an image that appears in focus to the user when the user's eyes are in a first viewing position may be displayed on the pixel-configured display.

[0023] The processor is preferably configured to determine a change in the user's eye position based on a change in the viewing position detected by the eye-tracking system. When the user moves away from the first viewing position, the image displayed on the pixel-based display may no longer appear in focus to the user. Therefore, the image needs to be corrected, and the subpixel values ​​of the pixel-based display need to be adjusted so that the image appears in focus at the second viewing position.

[0024] The second viewing position is determined by a processor based on a change in the viewing position detected by the eye tracking system. The second viewing position is different from the first viewing position. The processor may be further configured to determine a corresponding second position on the virtual image plane based on the second viewing position or equivalently based on the change in the viewing position from the first viewing position. The virtual image plane is between the first position and the second position when the user's eye is between the first viewing position and the second viewing position. The virtual image is located on the virtual image plane regardless of whether the virtual image is at the first position or the second position.

[0025] The processor is configured to associate the corresponding sub-pixel with the second virtual pixel on the virtual image plane when the user's eye is at the second viewing position. The processor is further configured to determine a second value of the corresponding sub-pixel of the pixel-configuration display by comparing the value of the corresponding sub-pixel with the value of the second virtual pixel.

[0026] The processor is configured to operate the corresponding sub-pixel of the display based on each second value when the user's eye is at the second viewing position. In one embodiment where these steps are repeated for each sub-pixel of the pixel-configuration display, an image that appears to be in focus for the user when the user's eye is at the second viewing position is displayed on the pixel-configuration display. Thus, the display device compensates the image displayed on the pixel-configuration display for the new position of the user's eye. Thus, the present invention enables compensating the image when the user moves to the second viewing position.

[0027] By repeating the step as the user's eyes move to a third viewing position, the image continues to appear in focus when viewed from the third viewing position. Thus, the present invention enables images displayed on a pixel-based display to appear in focus regardless of the viewing position and regardless of the user's eye movement relative to the pixel-based display. Therefore, the viewing experience is not adversely affected by movement relative to the pixel-based display.

[0028] In these embodiments, there may be two or more optical paths for the light rays emitted by a corresponding subpixel between the corresponding subpixel, the microlens array, the retina of the user's eye, and the virtual image plane. In these embodiments, one subpixel may correspond to multiple virtual subpixels. In these embodiments, the value of the subpixel may be determined based on a first optical path considered by the processor. In other embodiments, the value of the subpixel may be determined based on the values ​​of multiple virtual subpixels, for example, based on the average value of multiple subpixels.

[0029] A drawback of determining which virtual pixels correspond to which subpixels on the pixel-based display at a given viewing position by determining the optical path between the pixel-based display and the virtual image plane, for example by ray tracing, is that this calculation can be computationally intensive. As a result, the corrected image displayed on the pixel-based display may be corrected at a slower rate than the desired rate for correcting the image itself. This can negatively impact the user experience. This problem becomes more serious when the pixel-based display has more pixels and subpixels.

[0030] A well-known solution to this problem is to increase the computing power of the pixel-based display and / or the corresponding smart device. However, this is limited by the hardware and other capabilities of the pixel-based display and / or the smart device.

[0031] This problem also exists when the user moves to a second viewing position. In this case, a second virtual pixel corresponding to the corresponding subpixel is determined when the user is at the second viewing position. Determining the second virtual pixel for each subpixel of the display requires a large amount of computation and demands high processing power. This can slow down the device, and in some cases, image correction may occur at a slower speed than desired.

[0032] This problem is mitigated by the display device described in claim 2. In this embodiment, the processor is - When the virtual image plane is at a first position, determine the first pixel position of the corresponding first virtual pixel relative to the pixel configuration display. - When the virtual image plane is at the second position, for one or more virtual pixels, compare the position of the virtual pixel with the position of the first pixel. - The pixel configuration display is configured to associate the corresponding subpixel with a second virtual pixel on the virtual image plane by relating the corresponding subpixel with a virtual pixel whose position corresponds to a first pixel position when the virtual image plane is in a second position.

[0033] In this embodiment, the second virtual pixel is determined by first determining the first pixel position of the first virtual pixel, second comparing this first pixel position with the position of a virtual pixel on the virtual image plane at the second position, and third relating a subpixel of the pixel configuration display to a virtual pixel whose position corresponds to the first pixel position.

[0034] For example, if the virtual image plane is shifted in a plane parallel to the pixel configuration display, thereby preventing a change in the distance between the plane perpendicular to the pixel configuration display and the retinal surface, then a virtual pixel whose second position corresponds to the first position of the first virtual pixel may be a virtual pixel whose second position is the first position.

[0035] For example, if the virtual image plane is oriented perpendicular to the pixel configuration display, a virtual pixel whose second position corresponds to the first position of the first virtual pixel may be a pixel whose second position is located within the optical path of the first virtual pixel.

[0036] Advantageously, in this embodiment, there is no need to recalculate the optical paths of light rays emitted by subpixels in a pixel-based display. This significantly reduces computational density. As a result, the display device can determine the corrected image more quickly, thereby preventing any negative impact on the user experience.

[0037] In one embodiment, the processor is - The parallel component of the first pixel position parallel to the pixel configuration display, - The vertical component of the first pixel position perpendicular to the pixel configuration display, - The parallel component of each position of the virtual pixel when the virtual image plane is in the second position, - It is configured to determine the vertical component of each position of a virtual pixel when the virtual image plane is at a second position, The processor is configured to relate the corresponding subpixels of a pixel-configured display to a virtual pixel whose parallel component is equal to the parallel component of the first pixel position, when the vertical components are equal.

[0038] An arbitrary position may be decomposed into coordinate components perpendicular to the pixel configuration display and coordinate components parallel to the pixel configuration display. For example, it may be decomposed in a so-called Cartesian coordinate decomposition. This system may also use other types of coordinate systems, such as polar coordinates.

[0039] When the vertical component of the first pixel position is equal to the vertical component of the position of one of the virtual pixels, and the virtual image plane is at the second position, then the first virtual pixel and each of the other virtual pixels are corresponding virtual pixels if their parallel components are the same.

[0040] In one embodiment, the processor is configured to relate the corresponding subpixels of the pixel configuration display to virtual pixels whose parallel component is equal to the parallel component of a first pixel position when the eye-tracking system does not detect any change in the user's eyes perpendicular to the pixel configuration display.

[0041] The processor may determine that the vertical component of the first virtual pixel position and the vertical component of each virtual pixel position at the second position are equal when the eye-tracking system does not detect a change in position in the direction perpendicular to the pixel configuration display. Therefore, in this embodiment, the processor may not need to directly compare the vertical components, saving computation time.

[0042] In one embodiment, the display device is adapted to provide feedback to the user, such as haptic feedback, to prompt the user to change the position of the pixel configuration display. For example, the system may provide feedback when the user moves too far or too close to the pixel configuration display, for example, so that the display device can no longer properly render an image that appears to be in focus for the user. In another example, feedback may be provided to prompt the user to keep the display device in a stable position.

[0043] In one embodiment, the eye-tracking system is further adapted to track the user's line of sight to determine which part of a pixel-based display the user is fixating on. With display devices, especially those with larger displays, users often focus on a portion of the display rather than the entire display. Therefore, it may be advantageous to correct only the portion of the image the user is fixating on, rather than the entire display. Furthermore, tracking the user's line of sight may provide information about whether the user is fixating on the display. In some examples, the user may not be fixating on the display, and no image correction is necessary.

[0044] In other embodiments, the corresponding subpixel is located in the portion of the pixel-based display that the user is looking at. In this embodiment, the image is corrected in the portion of the pixel-based display that the user is looking at. More specifically, in one embodiment, the processor is configured to associate each subpixel in the portion of the pixel-based display that the user is looking at with a corresponding first virtual pixel and a corresponding second virtual pixel.

[0045] In one embodiment, the display device is adapted to track the pupil size of the user's eye. For example, as pupil size increases, more information may enter the user's eye when displayed on a pixel-based display. Therefore, it may be necessary to correct for larger portions of the image. The user's pupil size may be affected, for example, by adjusting the brightness of the display at night.

[0046] In one embodiment of the display device, the microlens width and / or microlens height are less than or equal to the pitch of eight subpixels in a pixel-based display. Smaller lens sizes may result in sharper images with less distortion due to the lens. Advantageously, the lens size is such that moiré effects are avoided, for example, when the lens size is not three times the pitch of the subpixels.

[0047] In one embodiment, the display device further comprises an optical gap layer, which is provided between the pixel-configured display and the microlens array, and which defines an optical gap between the pixel-configured display and the microlens array. The optical gap layer allows for increasing or decreasing the angle of incidence of light rays emitted by the subpixels of the display having the microlens array. Decreasing the angle of incidence may be achieved by increasing the size of the optical gap. Similarly, increasing the angle of incidence may be achieved by decreasing the size of the optical gap.

[0048] In one embodiment, the eye-tracking system is adapted to determine the user's viewing distance to a pixel-based display. This may be combined with a feedback system, for example, which may send feedback to the user prompting them to change their viewing distance if it is too far or too close. Maintaining the display device at a preferred viewing distance may increase the focus of the image observed by the user. Furthermore, maintaining the display device at a preferred viewing distance may increase the amount of information observed by the user, for example, because a larger portion of the display can be observed by the user.

[0049] In embodiments where an optical gap layer is present, the refractive index of the microlenses in the microlens array is higher than the refractive index of the optical gap layer.

[0050] In embodiments where an optical gap layer is present, for each lens of the microlens array, The relationship is smaller (microlens width, microlens height) / optical gap - pupil diameter - point projection diameter / object distance. Minimized, where the smaller of the corresponding microlens width and microlens height is indicated by smaller(microlens width, microlens height), the estimated size of the user's pupil is indicated by pupil diameter, the projected size of any point in a subpixel on the plane where the pupil of the eye is located is indicated by point projection diameter, and the distance from the pixel configuration display to the user's eye is indicated by object distance.

[0051] The above relationship allows for the projection of more subpixels onto the user's retina when the value is small but greater than or equal to zero, compared to when the relationship gives a larger value. As a result, the user perceives a display with a pixel configuration that is not only in focus but also has a higher resolution.

[0052] The microlens width, microlens height, and optical gap may all be determined during the construction of the display device. The user's pupil diameter may be approximated based on ambient light conditions or determined based on measurements of the user's eye. The point projection diameter and object distance depend on the distance from the user's eye to the display device. Therefore, this relationship may be minimized by changing the distance to the display. This determines the optimal viewing distance. The display device may provide feedback to the user, such as haptic feedback, to encourage the user to maintain the display at the optimal viewing distance.

[0053] In embodiments where an optical gap layer is present, the focal length of the lenses in the microlens array is equal to the optical gap.

[0054] In embodiments where an optical gap layer is present, the optical gap comprises a secondary microlens array.

[0055] In one embodiment, the microlens is rectangular.

[0056] In one embodiment, the microlens is a rectangular microlens. In one embodiment, the microlens is a circular or elliptical microlens. In one embodiment, the microlens is a hexagonal microlens.

[0057] In one embodiment, the display device further comprises a protective layer disposed on the microlens array. The protective layer may comprise a see-through plastic layer disposed on the microlenses to protect the microlenses and pixel configuration display from, for example, scratches and dirt.

[0058] In one embodiment, the pixel-based display is housed within a smartphone case. For example, the entire display device is integrated into the smartphone case. In this embodiment, the pixel-based display may be integrated into the smartphone case, thereby positioning the pixel-based display adjacent to or on the smartphone's display. For example, the pixel-based display may display information from the smartphone's display.

[0059] The present invention is a method for displaying images for a visually impaired user, wherein a display device according to the present invention is used, and the display device is - A pixel configuration display comprising an array of pixels, each having several subpixels, the pixel configuration display capable of displaying an image to the eyes of a user positioned at a viewing position relative to the position of the pixel configuration display, - A processor for operating the pixels of a pixel-configured display to display an image on the pixel-configured display, and a processor capable of storing information about the user's eyes, such as the focal length of the eye, - An eye-tracking system adapted to determine changes in the viewing position of the user's eyes relative to the position of a pixel-configured display, preferably comprising a processor configured to determine changes in the viewing position of the user's eyes based on changes in viewing position detected by the eye-tracking system, - A microlens array provided on a pixel configuration display, comprising microlenses arranged in a two-dimensional plane oriented parallel to the pixel configuration display, and configured to cover, preferably completely cover, the pixel configuration display and project the light emitted by the pixel configuration display toward the user's eyes at the viewing position, The processor is configured to determine a first position on the virtual image plane corresponding to the first viewing position of the user's eye by comparing the focal length of the user's eye with a first viewing position, and to determine the value of the virtual pixel on the virtual image plane, which corresponds to an image located on the virtual image plane and displayed on a pixel configuration display. The processor is configured to relate a corresponding subpixel to a corresponding first virtual pixel in a virtual image plane by determining the optical path between the corresponding subpixel, the microlens array, the retina of the user's eye, and the first virtual pixel, as the optical path of the light rays emitted by the subpixels of the pixel-configured display. The processor is further configured to determine the first value of the corresponding subpixel in the pixel configuration display by comparing the value of the corresponding subpixel with the value of the first virtual pixel. The processor is configured to operate the corresponding subpixels of the display based on each first value when the user's eyes are in a first viewing position. The processor is configured to determine a second position on the virtual image plane corresponding to the second viewing position of the user's eyes when the eye-tracking system determines a change in the user's eye viewing position from a first viewing position to a second viewing position. The processor is further configured to relate the corresponding subpixels of the pixel configuration display to the second virtual pixel of the virtual image plane when the virtual image plane is in the second position. The processor is further configured to determine the second value of the corresponding subpixel in the pixel configuration display by comparing the value of the corresponding subpixel with the value of the second virtual pixel. The processor is configured to operate the corresponding subpixels of the display based on a second value when the user's eyes are in a second viewing position.

[0060] The present invention further relates to a method for displaying images for visually impaired users, using alternative expressions for the methods described above, wherein the images are displayed on a display device according to the present invention.

[0061] In one embodiment, the method is - A step of determining a first position on a virtual image plane by comparing the focal length of the user's eye with the viewing position, wherein the virtual image is located on the virtual image plane and corresponds to an image displayed on a pixel configuration display, and a step of determining the value of a virtual pixel on the virtual image plane. - A step of relating a corresponding subpixel to a corresponding first virtual pixel in a virtual image plane by determining the optical path between the corresponding subpixel, a microlens array, the retina of the user's eye, and a first virtual pixel as the optical path of the light rays emitted by the corresponding subpixel of the pixel configuration display, - A step of determining the first value of a corresponding subpixel by comparing the value of the corresponding subpixel of a pixel configuration display with the value of a first virtual pixel, - A step of operating the corresponding subpixels of the display based on each first value when the user's eyes are in a first viewing position, - A step of determining a second position on a virtual image plane corresponding to the user's eye's viewing position, based on a change in the user's eye's second viewing position determined by the eye-tracking system. - A step of relating the corresponding subpixel of the pixel configuration display to the second virtual pixel of the virtual image plane when the virtual image plane is in a second position, - A step of determining the second value of the corresponding subpixel in a pixel configuration display by comparing the value of the corresponding subpixel with the value of the second virtual pixel, - The step of operating the corresponding subpixels of the display based on a second value when the user's eyes are in a second viewing position.

[0062] In one embodiment, this method is - A step of determining a first pixel position relative to a pixel configuration display of a corresponding first virtual pixel when the virtual image plane is in a first position, - When the virtual image plane is at a second position, the steps include comparing the position of each virtual pixel with the position of a first pixel for one or more virtual pixels, - Further includes the step of relating the position of a corresponding subpixel of a pixel configuration display to a virtual pixel corresponding to a first pixel position when the virtual image plane is in a second position.

[0063] In one embodiment, this method is - The process includes providing feedback to the user, such as haptic feedback, to prompt the user to change the position of the pixel configuration display.

[0064] In one embodiment, this method is - A step of tracking the user's eye position in a direction perpendicular to the pixel configuration display using an eye-tracking system, - The process includes the step of changing the first position of a virtual image plane based on a change in the position of the user's eyes in a direction perpendicular to the pixel configuration display.

[0065] In one embodiment, this method is - Includes the step of tracking the user's gaze using an eye-tracking system to determine which part of the pixel-based display the user is looking at, The corresponding subpixel is located in the part of the pixel configuration display that the user is focusing on.

[0066] The present invention further relates to a processor for operating a display device adapted to display images for visually impaired users.

[0067] The present invention further relates to a computer program for carrying out the method according to the present invention.

[0068] The present invention further relates to a smartphone case equipped with a pixel configuration display according to the present invention and adapted to operate the pixel configuration display according to the method of the present invention.

[0069] Next, the present invention will be described non-limitingly with reference to the accompanying drawings. In the drawings, the same parts are indicated by the same reference numerals. [Brief explanation of the drawing]

[0070] [Figure 1] This figure shows a display device equipped with a pixel-based display. [Figure 2] This is a side view of a pixel-based display covered by a microlens array. [Figure 3] This is a front view of a pixel configuration display, schematically showing some pixels and subpixels. [Figure 4] This is a schematic diagram of the optical path between the pixel-based display and the virtual image plane behind the user's eye. [Figure 5] This is a schematic diagram of the optical path between the pixel-based display, the user's eye, and the virtual image plane behind the pixel-based display. [Figure 6] These are schematic diagrams of the user's eye at the first viewing position and the user's eye at the second viewing position. [Figure 7] This is a flowchart of a method for displaying an image on a pixel-based display. [Figure 8] This is a flowchart of the method for determining the second virtual pixel. [Modes for carrying out the invention]

[0071] Figure 1 shows a display device 1 comprising a pixel configuration display 3. The pixel configuration display 3 comprises several pixels 4 schematically shown in Figure 1. By emitting light at a specific frequency, the pixels 4 render an image 2 on the pixel configuration display 4. Each pixel 4 of the pixel configuration display 3 comprises several subpixels 13, for example, three subpixels 13, which are not shown in Figure 1.

[0072] The display device 1 further comprises a processor 6 for operating pixels 4, for example, by operating subpixels 13. By operating pixels 4, an image 2 may be displayed on the pixel configuration display 3. The processor is further operable to store information about the user's eye 5, such as the focal length of the user's eye 5, the size of the user's eye 5, or the size of the pupil of the eye 5. This information may be used by the processor 6 to determine the values ​​of pixels 4, for example, subpixels 13, and to render the image 2 so that it is clearly visible to a user with a visual impairment.

[0073] The display device 1 further includes an eye-tracking system 7 adapted to determine changes in the viewing position of the user's eyes 5 relative to the position of the pixel-based display 3. In other words, the eye-tracking system 7 tracks the movement of the user's eyes 5 relative to the pixel-based display 3.

[0074] Figure 1 shows a display device 1 in which the pixel configuration display 3, processor 6, and eye-tracking system 7 all form part of the same device. This is not necessarily the case, for example, the processor 6 may be located in a different location and communicate remotely with the eye-tracking system 7 and the pixel configuration display 3, for example, via the internet or Bluetooth.

[0075] Figure 2 shows a side view of the pixel-configured display 3 covered by a microlens array 8. The microlens array 8 is separated from the pixel-configured display 3 by an optical gap layer 17, which forms an optical gap 18 between the pixel-configured display 3 and the microlens array.

[0076] The microlens array 8 comprises several microlenses 9. In Figure 3, the microlenses 9 are shown as plano-convex lenses. However, other types of microlenses 9 may be used. The shape, microlens width, and microlens height of the microlenses 9 do not have to be constant. In an embodiment, the focal length of the microlenses 9 is substantially equal to the optical gap layer 17, thereby the pixel configuration display 3 is positioned at the focal length of the microlens array 8. In an embodiment, the microlens array 8 may be covered by a protective layer 19, such as a translucent plastic layer, to protect the microlenses 9 from scratches and dust, for example.

[0077] Figure 3 shows a front view of a pixel-configured display 3 schematically showing several pixels 4 and subpixels 13. Each pixel 4 of the pixel-configured display 3 in Figure 3 comprises three subpixels 13. For example, each subpixel corresponds to a different color channel, such as red, green, and blue. By emitting different values ​​from each of these subpixels 13, the pixel 4 may appear to emit light at any wavelength. Thus, it is possible to display an image 2 on the pixel-configured display 3.

[0078] Figure 4 is a schematic diagram of the optical path 15 between the pixel-based display 3 and the virtual image plane 10 behind the user's eye 5. The pixel-based display 3 comprises pixels 4, each containing several subpixels 13. One of the subpixels 13 emits a light ray, which travels along the optical path 15 through the microlens array 8 towards the user's eye 5. The processor 6 uses techniques such as ray tracing to determine the optical path 15 from the pixel-based display 3 through the user's eye 5 to the virtual image plane 10.

[0079] The virtual image plane 10 comprises several virtual pixels whose values ​​are determined by a virtual image 11 placed on the virtual image plane 10. The position of the virtual image plane 10 is determined by the processor 6 by comparing the focal length of the user's eye 5 with the position where the eye 5 is, for example, a first viewing position.

[0080] The optical path 15 crosses the virtual image plane 10 at one of the virtual pixels 12. This allows determination of which virtual pixel 12 is the first virtual pixel 14. The value of the first virtual pixel 14 determines the value of the corresponding subpixel 13 when the user's eye is at the corresponding viewing position, for example, the first viewing position.

[0081] By determining the optical path 15 for multiple subpixels 13, the value of each of these subpixels 13 is determined by the corresponding first virtual pixel 14. In this way, the display device 1 may display a corrected image 2 on the pixel configuration display 3, which appears in focus to the user based on, for example, the focal length of the user's eye 5, and the focal length is affected by, for example, the user's visual impairment.

[0082] Figure 5 shows a schematic diagram of the optical path 15 between the pixel configuration display 3, the user's eye 5, and the virtual image plane 10 located behind the pixel configuration display 3. This embodiment illustrates an alternative method for finding the first virtual pixel 12 corresponding to the subpixel 13.

[0083] In Figure 5, the virtual image plane 10 is located behind the pixel configuration display 3. In this case, the optical path 15 is determined by the processor 6 to be the optical path 15 from the virtual image plane 10 through the pixel configuration display 3 and the microlens array 8 to the user's eye 5. This makes it possible to find the corresponding virtual pixel 12 for each subpixel 13 of the pixel configuration display 3, for example, the corresponding first virtual pixel 14 when the user's eye 5 is in a first viewing position.

[0084] Figure 6 is a schematic diagram of the user's eye 5 at a first viewing position and the user's eye 5' at a second viewing position. Changes in the positions of eyes 5 and 5' are detected by the eye-tracking system 7, and a second virtual pixel 16 is associated with the corresponding subpixel 13 by the processor 6. In one embodiment of the present invention, the second virtual pixel 16 is associated with the corresponding subpixel 13 by an optical path 15'. The optical path 15' is located between the subpixel 13 and the virtual image plane 10' at the second position.

[0085] The processor 6 is configured to determine a second value for the corresponding subpixel 13 of the pixel configuration display 3 by comparing the value of the corresponding subpixel 13 with the value of a second virtual pixel 16. The processor 6 is further configured to operate the corresponding subpixel 13 of the display 3 based on the second value when the user's eye 5' is in a second viewing position.

[0086] In another embodiment, the second virtual pixel 16 is not determined by relating it to a subpixel 13 via the optical path 15'. In this embodiment, the first pixel position of the first virtual pixel 12 is determined when the virtual image plane 10 is in a first position. The position of the virtual pixel 12 is compared to this first pixel position when the virtual image plane 10 is in a second position. The subpixel 13 of the display 3 is then associated with a virtual pixel 16 corresponding to the first pixel position when the virtual image plane 10' is in a second position.

[0087] Advantageously, in this embodiment, there is no need to recalculate the optical path 15' of the light rays emitted by the subpixels 13 of the pixel-based display. This significantly reduces computational density. As a result, the display device 1 may determine the corrected image 2 more quickly, thereby preventing the user experience from being negatively affected when the user's eyes move across the pixel-based display 3.

[0088] Figure 7 shows a flowchart of a method for displaying image 2 on a pixel-configured display 3. In the first step 101 of the method, a first position on the virtual image plane 10 is determined by comparing the focal length of the user's eye with the viewing position. A virtual image 11 may be located on the virtual image plane 10. The virtual image 11 corresponds to image 2 displayed on the pixel-configured display 3, and the virtual image 11 determines the value of the virtual pixel 12 on the virtual image plane 10.

[0089] In the second step 102 of this method, the corresponding subpixel 13 is associated with the corresponding first virtual pixel 12 of the virtual image plane 10 by determining the optical path 15 between the corresponding subpixel 13, the microlens array 8, the retina of the user's eye 5, and the first virtual pixel 14 as the optical path 15 of the light rays emitted by the subpixel 13 of the pixel configuration display 3.

[0090] In the third step 103 of this method, the first value of the corresponding subpixel 13 of the pixel configuration display 3 is determined by comparing the value of the corresponding subpixel 13 with the value of the first virtual pixel 14.

[0091] In the fourth step 104 of this method, the corresponding subpixels 13 of the display 3 are operated based on their respective first values ​​when the user's eyes 5 are in a first viewing position.

[0092] In the fifth step 105 of this method, the eye-tracking system 7 detects changes in the position of the user's eyes 5.

[0093] In the sixth step 106 of this method, a second position on the virtual image plane 10 corresponding to the second viewing position of the user's eye 5 is determined based on the change in the viewing position of the user's eye 5 determined by the eye-tracking system 7. The second viewing position of the user's eye 5 is different from the first viewing position.

[0094] In the seventh step 107 of this method, the corresponding subpixel 13 of the pixel configuration display is associated with the second virtual pixel 16 of the virtual image plane 10 when the virtual image plane 10 is in the second position.

[0095] In the eighth step 108 of this method, the second value of the corresponding subpixel 13 of the pixel configuration display 3 is determined by comparing the value of the corresponding subpixel 13 with the value of the second virtual pixel 16.

[0096] In the ninth step 109 of this method, the corresponding subpixel 13 of the display 3 is operated based on the second value when the user's eye 5 is in the second viewing position.

[0097] This method allows the user to move relative to display device 1 without any negative impact on the viewing experience as a result of viewing the corrected image 2 from different angles. The user may experience a negative impact on the viewing experience because their ability to focus on image 2 decreases with the field of view.

[0098] In an embodiment in which steps 107-109 are repeated for each subpixel 13 of the pixel-configured display 3, an image 2 that appears to be in the user's focus when the user's eye 5 is in a second viewing position may be displayed on the pixel-configured display 3. Thus, the display device 1 compensates for the image 2 displayed on the pixel-configured display 3 for the new position of the user's eye 5. Thus, the present invention makes it possible to compensate for the image 2 when the user moves to a second viewing position.

[0099] By repeating steps 105-109 of the method when the user's eye 5 moves to a third viewing position, the image 2 continues to appear in focus when viewed from the third viewing position. Thus, the present invention makes it possible for the image 2 displayed on the pixel configuration display 3 to appear in focus regardless of the viewing position and regardless of the movement of the user's eye 5 relative to the pixel configuration display 3. Therefore, the viewing experience is not adversely affected by movement relative to the pixel configuration display 3.

[0100] Figure 8 shows a flowchart of a method for determining a second virtual pixel 16, which may be performed between method steps 101-107. In the first step, when the virtual image plane 10 is in a first position, the first pixel position of the corresponding first virtual pixel 14 relative to the pixel configuration display 3 is determined.

[0101] In the second method step 111, the position of one or more virtual pixels is determined when the virtual image plane 10' is at the second position.

[0102] In the third method step 112, when the virtual image plane 10' is at the second position, for one or more of the virtual pixels 12, the position of each virtual pixel 12 is compared with the first pixel position.

[0103] In the fourth method, step 113, the corresponding subpixel 13 of the pixel configuration display 3 is associated with a virtual pixel 16 whose position corresponds to a first pixel position when the virtual image plane 10' is in a second position. [Explanation of Symbols]

[0104] 1 Display device 2 images 3-pixel configuration display 4 pixels 5 eyes 5' eye 6 processors 7 Eye-tracking system 8 Microlens Array 9 Microlenses 10 Virtual Image Plane 10' Virtual Image Plane 11 Virtual Images 12 virtual pixels 13 subpixels 14. The first virtual pixel 15 Light path 15' optical path 16. Second virtual pixel 17 Optical gap layer 18 Optical gap 19 Protective layer

Claims

1. A display device (1) that displays an image (2) to a user with a visual impairment, A pixel configuration display (3) having multiple pixels (4) in which each pixel has multiple subpixels, An eye-tracking system (7) for tracking the position of the user's eyes (5) relative to the pixel configuration display (3), A microlens array (8) is provided on the aforementioned pixel configuration display (3), Processor (6), The processor (6) is equipped with, Based on the focal length of the eye at the first viewing position, a first position on the virtual image plane (10), which includes a plurality of virtual pixels and corresponds to the viewing position of the eye (5), is determined. For each subpixel (13), the optical path of the light ray emitted by the subpixel (13) to the corresponding first virtual pixel (14) is determined, thereby relating the subpixel to the first virtual pixel (14) of the virtual image plane (10). For each subpixel, the first operating value of the subpixel (13) is determined by comparing the subpixel value relating to the color or brightness of the subpixel (13) with the value of the associated first virtual pixel (14). Based on the first value determined, each subpixel (13) is operated to display the corrected image on the pixel configuration display (3). Based on the eye-tracking system (7) detecting a change in the eye position to a second viewing position, the second position of the virtual image plane (10) is determined. When the virtual image plane (10) is in the second position, the following (a) to (c): (a) When the virtual image plane (10) is at the first position, the first pixel position of the corresponding first virtual pixel (14) relative to the pixel configuration display (3) is determined, (b) When the virtual image plane (10) is at the second position, the position of one or more virtual pixels (12) is compared with the first pixel position, (c) The corresponding subpixel (13) of the pixel configuration display (3) is associated with a virtual pixel (16) whose position corresponds to the first pixel position when the virtual image plane (10) is at the second position. By doing so, each subpixel (13) is associated with the second virtual pixel (16) of the virtual image plane (10). It is configured to do the following: For each subpixel (13), the second operating value of the subpixel is determined by comparing the value of the subpixel (13) with the value of the second virtual pixel. Based on the determined second operating value, each subpixel is manipulated, and an image corrected for the second viewing position based on the second operating value is displayed. Display device (1) configured as follows.

2. The display device (1) according to claim 1, wherein the display device (1) is configured to provide feedback prompting the user to change the position of the pixel configuration display (3).

3. The display device (1) according to claim 1, wherein the eye-tracking system (7) is configured to track the gaze of the user's eye (5) to determine which part of the pixel-based display (3) the user is fixated on.

4. The display device (1) according to claim 3, wherein the corresponding subpixel (13) is located in the portion of the pixel configuration display (3) that the user is looking at.

5. The display device (1) according to claim 1, further comprising an optical gap layer (17) between the pixel configuration display (3) and the microlens array (8), wherein the optical gap layer defines an optical gap between the pixel configuration display and the microlens array.

6. The display device (1) according to claim 5, wherein the focal length of the microlens (9) in the microlens array (8) is equal to the size of the optical gap (18).

7. The display device (1) according to claim 5, wherein the optical gap comprises a secondary microlens array.

8. The display device (1) according to claim 1, wherein the microlenses (9) in the microlens array (8) are rectangular or square microlenses (9).

9. The display device (1) according to claim 1, further comprising a protective layer (19) on the microlens array (8).

10. The pixel configuration display (3) is provided inside a smartphone case, as described in claim 1.

11. A method for displaying an image (2) to a user with a visual impairment, The steps include determining a first position on a virtual image plane (10) that corresponds to the viewing position of the eye (5) and includes a plurality of virtual pixels, based on the focal length of the user's eye at a first viewing position, For each subpixel (13) of a pixel-configured display, the steps include determining the optical path of the light ray emitted by the subpixel (13) to the corresponding first virtual pixel (14), thereby relating the subpixel to the first virtual pixel (14) of the virtual image plane (10), The steps include determining a first operating value for each subpixel (13) by comparing the subpixel value relating to the color or brightness of the subpixel (13) with the value of the associated first virtual pixel (14), The steps include: operating each subpixel (13) based on the first value determined above to display the corrected image on the pixel configuration display (3); The eye-tracking system (7) detects a change in the eye position to a second viewing position, and determines the second position of the virtual image plane (10). When the virtual image plane (10) is in the second position, the following (a) to (c): (a) When the virtual image plane (10) is at the first position, the first pixel position of the corresponding first virtual pixel (14) relative to the pixel configuration display (3) is determined, (b) When the virtual image plane (10) is at the second position, the position of one or more virtual pixels (12) is compared with the first pixel position, (c) The corresponding subpixel (13) of the pixel configuration display (3) is associated with a virtual pixel (16) whose position corresponds to the first pixel position when the virtual image plane (10) is at the second position. The steps include associating each subpixel (13) with a second virtual pixel (16) of the virtual image plane (10), The steps include determining a second operating value for each subpixel (13) by comparing the value of the subpixel (13) with the value of the second virtual pixel, The steps include: manipulating each subpixel based on the determined second operating value and displaying an image corrected for the second viewing position based on the second operating value; A method that includes this.

12. The method according to claim 11, further comprising the step of providing feedback to the user prompting the user to change the position of the pixel configuration display (3).

13. The method according to claim 11, further comprising the step of tracking the gaze of the user's eye (5) to determine which part of the pixel configuration display (3) the user is fixating on.

14. A smartphone case comprising the display device (1) described in claim 1, and adapted to operate the display device (1) according to the method described in claim 11.

Images