Image correction caused by deformation of viewing device components

Abstract

A display assembly displays a virtual object at a selected position, wherein an eye viewing the virtual object has an expected gaze direction. A deformation of the display assembly is detected. The deformation enables the virtual object to be viewable at the changed position, wherein the eye has a changed gaze direction. The virtual object may be displayed at a corrected position, wherein the eye viewing the virtual object at the corrected position has a corrected gaze direction that moves closer to the expected gaze direction than the changed gaze direction.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 643,672 filed on March 15, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates generally to viewing devices and methods of displaying rendered content, and more particularly to detecting and correcting deformation of components of viewing devices. Background Art

[0004] Viewing devices that provide rendered images have been widely used for computing, entertainment, and other purposes. Viewing devices are typically wearable devices with a display for rendering images, and may include various features, such as enabling a user to see three-dimensional images, enabling a user to see renderings within a real-world environment with or without fixing the user's position within the real-world environment, and the ability to display video or other moving renderings to a user.

[0005] The viewing device has various components that deform after a period of use. When these components deform, the rendered virtual objects may not be in their original positions when the viewing device is still new. For example, the background application can display virtual objects in a fixed position relative to the viewing device or in a fixed position relative to the real-world objects around the user. In some cases, the viewing device may have a perspective display so that the user can see the real-world objects and can perceive the rendered objects in a fixed position relative to the real-world objects. For example, the user can perceive a coffee cup rendered on a real-world table. When the components of the viewing device deform, the coffee cup may no longer be rendered as being on the table, but floating at a distance above the table. Therefore, the coffee cup will not be displayed to the user in a way that is true to reality. In addition, if the coffee cup is used as an interface element for the user to interact with the background application, there may be a mismatch between the position where the background application expects the coffee cup to be and the position where the user interacts with the coffee cup. Summary of the invention

[0006] The present invention provides a viewing device for displaying rendered content, comprising: a display component, which is configured to display a virtual object at a selected position on the display component, wherein an eye viewing the virtual object has an expected gaze direction; and a deformation detection system, which is connected to the display component and configured to detect a measured gaze direction of an eye viewing the virtual object on the display component, and calculate the deformation of the display component based on the measured gaze direction being a changed gaze direction different from the expected gaze direction.

[0007] The present invention also provides a method for displaying rendered content, comprising: displaying a virtual object at a selected position on the display component with the aid of a display component, wherein an eye viewing the virtual object has an expected gaze direction; and detecting a deformation of the display component, wherein the deformation makes the virtual object visible at the changed position, wherein the eye has a changed gaze direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention is further described by way of examples, in which:

[0009] Figure 1 is a block diagram illustrating a viewing device that displays rendered content to a user's eyes according to an embodiment of the present invention;

[0010] Figure 2A is a perspective view of an eye and an optical display of a viewing device, wherein the optical display is not deformed;

[0011] Figure 2B is similar to Figure 2A A view in which the optical display is deformed;

[0012] Figure 3A is a perspective view showing how the position of a rendered object is established;

[0013] Figure 3B and 3C is a front view showing how to move a rendered object;

[0014] Figure 3D is a top plan view showing how the position of a rendered object can be moved using different insertion angles provided by a laser projector;

[0015] Figure 4 is a perspective diagram showing possible gaze vector errors;

[0016] FIG. 5A to FIG. 5D A statistical method for determining the possible positions of a virtual object and the incremental movement of the virtual object to correct its displacement is shown;

[0017] Figures 6A to 6C shows the motion of a virtual object due to display deformation and the correction of the movement of the virtual object;

[0018] Figures 7A to 7E The movement of the virtual object and the corresponding change in the viewing angle due to the display deformation are shown ( Figures 7A to 7B ), the effect of the deformation of the eye-tracking camera on the calculated gaze angle ( Figure 7C and 7D ), and correction of the gaze angle ( Fig. 7E );

[0019] Figures 8A to 8D shows the movement of the virtual object and the corresponding change of the gaze vector ( Fig. 8A and 8B ), and the effect of the deformation of the eye-tracking camera on the calculated gaze angle ( Figure 8C and 8D );

[0020] Fig. 9 Shows a correction Fig.8D The solution for the resultant change of the viewing angle shown;

[0021] Figures 10A to 10C Shown with Fig. 9 Different corrections Fig.8D The solution of synthetic gaze angle in

[0022] Fig.11 is similar to Figure 1 The diagram further illustrates the integration Figures 10A to 10C the reference system of the viewing device for the process shown;

[0023] Fig.12 is a flow chart illustrating the functionality of a viewing device; and

[0024] Fig.13 is a block diagram of a machine in the form of a computer that may find application in the system of the present invention according to one embodiment of the present invention. DETAILED DESCRIPTION

[0025] In the attached figure Figure 1 A viewing device 20 is shown for displaying rendered content to a user's eye 22 according to an embodiment of the present invention. The viewing device 20 includes a display assembly 24, a deformation detection system 26, a user input device 28, and a correction system 30.

[0026] The display assembly 24 includes a video data receiver 32, a projector 34, and an optical display 36 connected directly or indirectly to each other. The display assembly 24 includes a structure (not shown) that can be fixed to the user's head, wherein the optical display 36 is located in front of the user's eyes 22. The optical display 36 is a transparent component that allows the eyes 22 to see objects in the real world behind the optical display 36, and can simultaneously project virtual images to the user so that light associated with real and virtual objects is visible to the user.

[0027] The video data receiver 32 is connected or can be connected to the video data channel that carries the color and intensity values ​​of the pixels. The projector 34 has a laser and a scanner that can create a two-dimensional pattern based on the video data. The optical display 36 is located at the location of the laser of the projector 34 to couple the laser into the optical display 36. The laser light then propagates through the optical display 36 and is emitted from the optical display 36 toward the eye 22 through the pupil of the optical display 36. The eye 22 thus receives light from the real world objects behind the optical display 36 and the light generated by the projector 34. An augmented reality view is then created on the retina 38 of the eye 22, which includes light from the real world scene visible to the user through the optical display 36 combined with the light created by the projector 34 to represent the virtual content.

[0028] The deformation detection system 26 includes an attention generation unit 40 , an input trigger 42 , an eye tracking camera 44 , a gaze angle calculation module 46 , a statistics system 48 , and a correction calculation unit 50 .

[0029] The attention generation unit 40 is connected to the video data receiver 32. The attention generation unit 40 is configured to provide an override function to the video data receiver 32. The attention generation unit 40 can, for example, insert a virtual object into the data stream received by the video data receiver 32, change the color of the virtual object and / or reduce the size of the virtual object, in order to cause the eye 22 to direct its gaze toward the virtual object and focus on the virtual object. Although the attention generation unit 40 is described, the virtual content that can be used as a basis for calibrating the augmented reality system is typically provided by a rendering system or module.

[0030] The input trigger 42 is connected to the user input device 28 and detects user input through the user input device 28. The user input device 28 may be, for example, one or more of a joystick, a wand, a camera that tracks a user's body part, a button, a touch pad, a glove equipped with a sensor, a mouse, a keyboard, etc. The user input device 28 provides input to the input trigger 42.

[0031] Eye tracking camera 44 is mounted in display assembly 24 at a location that captures images of eye 22. In some embodiments, one or more cameras per eye are used to image the user's eyes. Alternatively, a single camera with a wide enough angle to capture images that include both eyes of the user may be used.

[0032] The gaze angle calculation module 46 is connected to the eye tracking camera 44 and the input trigger 42. The gaze angle calculation module 46 calculates the gaze angle of the eye 22 based on the image captured by the eye tracking camera 44. The gaze angle calculation module 46 is connected to the input trigger 42 and is activated by the input trigger 42 so that when the user input is detected, the gaze angle is calculated by the gaze angle calculation module 46. Although the system is described using the calculated gaze angle, any other eye gaze orientation characteristics may be used, such as a gaze vector, a gaze coordinate, an axis orientation, or a corneal center position. In addition, the gaze data from the left eye and the right eye may be used in combination to collect information about the location where the user's eyes are focused.

[0033] The statistical system 48 is connected to the gaze angle calculation module 46 and receives a large number of gaze angle calculation measurements. Whenever the input trigger 42 activates the gaze angle calculation module 46, the statistical system 48 records the gaze angle received from the gaze angle calculation module 46. Therefore, the statistical system 48 collects multiple gaze angles over a period of time. The statistical system 48 then calculates statistically relevant gaze angles, such as median gaze angles from the gaze angles recorded by the statistical system 48. A representative gaze angle can be selected or calculated and can be used to compare with an expected gaze angle associated with the display of virtual content. If the representative gaze angle is substantially different from the expected gaze angle, it can be inferred that deformation has occurred within the system.

[0034] The correction calculation unit 50 is connected to the statistical system 48. The correction calculation unit 50 calculates the desired correction at the position of the rendered virtual object created or modified by the attention generation unit 40.

[0035] The correction system 30 is connected to the correction calculation unit 50. The correction system 30 receives correction data from the correction calculation unit 50. The video data receiver 32 is connected to the correction system 30. The correction system 30 modifies the position of the virtual object created or modified by the attention generation unit 40. The correction system 30 also modifies the positions of all other objects in the video stream by the same amount and direction as the position of the virtual object created or modified by the attention generation unit 40 is corrected.

[0036] In use, the user attaches the viewing device 20 to their head with the optical display 36 located in front of the eyes 22. The user can then see real world objects through the transmissive optical display 36 and can simultaneously view rendered virtual content.

[0037] The video data receiver 32 receives video data from a background application. The background application may be, for example, a movie application for displaying a movie, a game, a web browser, a menu, a launcher, two-dimensional content, three-dimensional content, or any other type of virtual content. The video data includes data representing an image frame received at a video data rate. The video data includes pixels having intensity and color values. The video data receiver 32 provides the video data to the projector 34. The projector 34 creates a two-dimensional pattern for each frame. The pattern may include a laser beam, each beam representing a corresponding pixel, and its intensity and color are modulated. The projector 34 couples the pattern directly or indirectly into the optical display 36 through a lens, a reflector, a grating, etc. The pattern is transmitted through the optical display 36 and emitted from the optical display 36 toward the eye 22. The light beam 54 represents the light transmitted from the optical display 36 toward the eye 22. However, it should be understood that many light beams representing virtual objects are projected from the optical display 36 so that the image received by the retina 38 is similar to the pattern created by the projector 34. In some embodiments, due to one or more wavefront shaping techniques applied to the light representing the virtual object, the user can perceive the virtual object as three-dimensional. In embodiments where the pattern changes continuously with each frame of data provided by video data receiver 32 to projector 34, the user may see a dynamic image.

[0038] The video data received by the video data receiver 32 includes data representing one or more objects to be displayed on the retina 38 of the eye 22. The virtual objects represented in the data received by the video data receiver 32 can actually be displayed on the optical display 36. However, such virtual objects may not be visible or easily discernible on the surface of the optical display 36 due to many factors, including that the optical display 36 is a see-through display and / or that the optical display 36 is primarily used as a waveguide to guide light between the projector 34 and the eye 22. Even if the virtual objects are not visible on the surface of the optical display 36, the video data provided by the video data receiver 32 to the projector 34 still includes data representing the objects. For the purpose of discussion, it will be assumed that one or more virtual objects are visible on the surface of the optical display 36. However, it should be understood that for ease of explanation, the objects are shown on the surface of the optical display 36. The present invention is primarily directed to a calibration methodology that is the same regardless of whether the objects are actually visible or not visible on the surface of the optical display 36.

[0039] During normal operation of the viewing device, the attention generation unit 40 does not provide an override function for the normal operation of the video data receiver 32. All video data is presented to the user at the video refresh rate without any interference from the deformation detection system 26. The deformation detection system 26 only performs the normal function of the video data receiver 32 a few times per day (e.g., 50 to 100 times per day during continuous use) for measurement, while the video data receiver 32 continues to operate uninterrupted the rest of the time. The user input device 28 and the eye tracking camera 44 are connected to the background application. During normal operation of the video data receiver 32, the user can use the user input device 28 to provide instructions to the background application or otherwise interact with the virtual content rendered on the optical display 36, while the one or more eye tracking cameras 44 continue to monitor the eyes 22. The user can, for example, use the user input device 28 to interact with a virtual object displayed to the user via the optical display 36, while the background application relies on the eye tracking camera 44 to know when the user is looking at the virtual object.

[0040] After factory calibration of viewing device 20, display assembly 24 has little or no deformation. Objects displayed using optical display 36 are at their expected positions relative to the viewing angle of eye 22. During use of viewing device 20, components of viewing device 20 that include or are connected to display assembly 24 begin to deform. The deformation can be manifested as deformation of optical display 36, deformation of hinges, and deformation of materials used in the structure that mounts viewing device 20 to the user's head. Deformation is usually caused by a combination of small or large stresses applied to components of display assembly 24 and the materials of components of display assembly 24. For example, stresses may be generated in the following situations: when a cable connected to optical display 36 pulls on optical display 36, when a user mounts viewing device 20 on or removes it from his head, and when a user manipulates hinges, springs, or other dynamic components to disassemble or store viewing device 20. When stresses are cycled, the material properties of components of display assembly 24 may experience fatigue, and it is known that plastic materials experience "creep" when they are stressed over a period of time. Some materials of the viewing device 20 may also be subjected to thermal loads, and such temperature changes may cause deformation of one or more components. The result of the deformation is that the actual positions of virtual objects presented to the user on the deformed viewing device 20 are not the desired rendering positions of these virtual objects. Therefore, the user's actual gaze is different from the gaze that would be expected if the user were to see the rendered virtual content at the desired rendering position. In addition, the deformation is continuous, so it can be expected that the rendering position of the virtual object perceived by the user will continue to move relative to the desired rendering position over a period of time. The display component 24 first displays the virtual object at a selected position, wherein the eye 22 has the expected gaze direction. The deformation of the display component 24 makes the virtual object viewable at the changed position, wherein the eye 22 has the changed gaze direction.

[0041] One or more components of the deformation detection system 26 can be functionally and structurally connected to the display assembly 24 to detect deformation of the display assembly. The correction system 30 is connected to the deformation detection system 26. The correction system 30 displays the virtual object at the corrected position together with the display assembly 24. When the virtual object is displayed at the corrected position, the eye 22 viewing the virtual object at the corrected position has a corrected gaze direction that moves closer to the expected gaze direction than the changed gaze direction. The deformation detection system 26 provides an override or enhancement function to the normal operation of the viewing device 20 described above. During continuous operation, the deformation detection system 26 is activated intermittently, for example, 10 to 20 times per hour. In addition, the deformation detection system 26 is activated only for a few seconds, for example, three to seven seconds, which is as long as the time required to make measurements and adjustments. During the remaining time, the viewing device 20 operates normally.

[0042] The functionality of the deformation detection system 26 is initiated by activating the attention generation unit 40 while the user input device 28 is connected to the input trigger 42 and the eye tracking camera 44 continuously monitors the eye 22. For the user, it is mainly necessary to interact with the rendered object received by the video data receiver 32 from the background application, which is small enough to minimize the gaze vector error. The attention generation unit 40 can also modify the color of the virtual object or reduce its size to attract the user's attention and help attract the eye 22 to the small point where the virtual object is located. The video data receiver 32 provides data representing the virtual object to the projector 34. The projector 34 then generates light and projects the light representing the virtual object toward the optical display 36 for viewing by the user.

[0043] The virtual object placed or modified by the attention generation unit 40 is of a type that requires the user to interact with the rendered virtual object using the user input device 28. Such a virtual object may be, for example, a play button, a target, an application launch icon, etc. When the user interacts with the rendered virtual object using the user input device 28, it is assumed that the user's eye 22 is looking at the rendered virtual object on the optical display 36. The eye tracking camera 44 captures an image of the eye 22 and provides the image data to the gaze angle calculation module 46. The user input device 28 activates an input trigger 42, which instructs the gaze angle calculation module 46 to calculate the gaze angle of the eye using the data received from the eye tracking camera 44. The gaze angle calculated by the gaze angle calculation module 46 represents the actual position of the rendered virtual object on the optical display 36 as perceived by the user.

[0044] The gaze angle calculation module 46 provides the gaze angle to the statistics system 48. The statistics system 48 stores the gaze angle. When the deformation detection system 26 is activated for the second time, the process is repeated and the statistics system 48 stores the second gaze angle. The process is repeated until the statistics system 48 has stored enough gaze angles to allow the statistics system 48 to calculate a statistically relevant changed gaze angle. The statistics system 48 can, for example, calculate an average, mean, or median of the gaze angles based on the collected gaze angles. The statistics system can calculate the statistically relevant gaze angles based on all measurements, although preferably, the statistics system only uses the gaze angles outside a predetermined error range (e.g., 90 arc minutes) as data points for calculating the statistically relevant gaze angles. Alternatively, in some embodiments, rather than accumulating multiple measurements to calculate a statistically determined gaze angle before performing an incremental correction, each measured gaze angle that is determined to be outside the tolerance error of the expected gaze angle can result in an incremental correction to the rendering position of the virtual content.

[0045] The statistics system 48 provides the changed gaze angle to the correction calculation unit 50. The correction calculation unit 50 determines the amount of correction of the gaze angle required to return the gaze angle from the changed gaze angle to the expected gaze angle. The correction calculation unit 50 calculates both the required correction direction and the correction amount, and then calculates an actual correction that is only a small fraction of the required correction. The correction calculation unit 50 then provides the actual correction to the correction system 30. The correction system 30 provides the correction to the video data receiver 32. The video data receiver 32 then moves the virtual object generated by the attention generation unit 40 and all other objects according to the direction and magnitude of the actual correction received from the correction system 30.

[0046] Before providing the changed gaze angle to the correction calculation unit 50, the statistical system 48 may require a large number of measurements, such as 50 to 100 measurements. In addition, any correction made is only a small part of the actual change in the gaze angle caused by the deformation of the display component 24. Therefore, cumulative correction is a slow process. For example, during eight hours of continuous operation in a day, only a single correction can be made, and it may take several days or weeks to correct the gaze angle within the tolerance range of the expected gaze angle. Alternatively, several small corrections can be made during the use of the device. The number of corrections can depend on the number of times the user interacts with the virtual content, which is considered to be reliable for the gaze angle calculation and rendering position calibration. This slow process allows recalculation in the event of an error, and allows the deformation of the display component 24 to continue to accelerate and decelerate, reducing the risk of exceeding the required corrections that must be made in the event of continued deformation of the display component 24.

[0047] exist Figure 2A In FIG. 3 , the undeformed shape 60 represents the optical display 36 before deformation. The rendered object 62 is placed in the optical display 36 to attract the user's attention. When the user looks at

[0048] When rendering the object 62, an actual gaze vector 64 may be defined. The actual gaze vector 64 is the direction or axis along which the eye 22 is directed in order to view the virtual object. The actual gaze vector extends from the center point 22 of the eye 22 through the center of the pupil of the eye. To calculate the actual gaze vector 64, one approach is to first calculate the actual center point 22 of the eye 22. It is also possible to calculate the actual gaze vector 64 without calculating the actual center point 22 of the eye 22, but to achieve this using other anatomical information of the eye. The expected gaze vector 64 extends through the optical display 36, and the user perceives the rendered object 62 as an augmented object 68. The augmented object 68 is located in the plane represented by the undeformed shape 70. Because the plane of the undeformed shape 70 is farther away than the plane of the optical display 36, the user perceives the augmented object 68 as being larger than the rendered object 62, and that the field defined by the undeformed shape 70 is larger than the field defined by the undeformed shape 60. The user also perceives the table 72. The user also perceives the table 72 as extending through the optical display 36. Figure 2A The rendered virtual content represented by the augmented object 68 is augmented.

[0049] A plurality of infrared light emitting diodes (LEDs) 74 may be included in the eye tracking system. Each LED 74 transmits infrared light that forms a corresponding infrared light spot ("flash") 76 on the surface of the eye 22. Reference numeral 78 represents infrared light reflected from the light spot 76 toward the eye tracking camera 44. The eye tracking camera 44 captures the positions of all the light spots produced by the LEDs 74 on the surface of the eye 22. The positions of the light spots 76 are used by the gaze angle calculation module 46 to calculate the gaze angle or gaze vector of the eye 22. In some embodiments, the eye tracking camera may also capture the position of the pupil. The pupil position data may be used in conjunction with the infrared flash to determine the position of the eye. The gaze angle or gaze vector may be calculated or otherwise determined based on the eye position information.

[0050] Figure 2A Expected gaze vector 64 is shown, propagating through the center of rendered object 62 and augmented object 68. Reference numeral 82 denotes light reflected from eye 22 captured by eye tracking camera 44. Thus, in some embodiments, eye tracking camera 44 captures the orientation of the eye and the location of pupil 80. Gaze angle calculation module 46 calculates actual gaze vector (in Figure 2A In some embodiments, the position of eye 22 relative to optical display 36 may also be used.

[0051] It may not always be necessary to measure in order to determine the gaze angle before deformation. For example, when the viewing device 20 is a new device, it can be assumed that the display component 24 is not deformed, and when the rendering object 62 is positioned as shown, the gaze angle of the eye 22 is as represented by the expected gaze vector 64. Therefore, the rendering object 62 can be positioned as shown, and the expected initial gaze vector 64 can be determined without performing any additional measurements. The expected gaze angle and the actual gaze angle will change based on the location of the rendered content. Although the "initial" gaze angle or "before deformation" gaze angle can be calculated, the expected gaze vector can be assumed from a background application or a previous calculation. The difference between the assumed expected gaze angle and the newly calculated actual gaze angle can then be calculated.

[0052] exist Figure 2B , shape 86 represents a deformation of optical display 36. The deformation of optical display 36 is represented by the difference when shape 86 is compared to undeformed shape 60. Rendered object 62 is still rendered at the same (x, y) position on the display, but the deformation of the display relative to the user causes the user to perceive the content as being at a different location than expected. Due to the deformation, rendered object 62 is no longer at Figure 2A 8. Instead of the position shown, rendered object 62 has moved a distance and direction represented by vector 88. Thus, rendered object 62 has moved in a direction caused by the deformation of optical display 36 in the area of ​​rendered object 62. An altered gaze vector 90 is defined between eye 22 and rendered object 62. The user sees rendered object 62 as augmented object 68 in the plane represented by shape 92. Shape 92 is deformed relative to undeformed shape 70 in a manner similar to the deformation between shape 86 and undeformed shape 60.

[0053] As the user looks at augmented object 68, an actual gaze vector 90 is transmitted from eye 22 to the location of augmented object 68 within deformed shape 92. In this example, actual gaze vector 90 is different from expected gaze vector 64. Eye tracking camera 44 continues to capture eye position information, which may be based on pupil position and / or glint reflection patterns within the image, so that gaze angle calculation module 46 can calculate actual gaze vector 90 and compare it to expected gaze vector 64 to infer whether display component deformation has occurred.

[0054] Figure 3A1 shows how to correct the position of the rendered object 62. The projector 34 projects image light 94 associated with a virtual image or object toward the optical display 36 at a predetermined angle. The image light 94 may encounter one or more diffractive optical elements, such as an incoupling grating 95, an orthogonal pupil expander 97, and an exit pupil expander 99, which are disposed in or on the waveguide 101 as part of the optical display 36. As the image light 94 propagates through the waveguide 101, the diffractive optical elements redirect the light in different ways, ultimately causing the image light 94 to be emitted from the waveguide toward the user's eye 22 through the exit pupil expander 99. The user's eye 22 may focus the image light 94 at a location on the retina so that the user perceives the rendered object 62.

[0055] refer to Figure 3B and 3C , two examples of images projected as image light 94 are shown. Figure 3B , a virtual content object 115 is shown in the center of image 94a. When image 94a is projected toward an undeformed optical display 36 and travels through its components, the user will see the virtual content object 115 located in the center of the field of view. However, if image 94a is projected toward a deformed display, the virtual content object 115 may be perceived by the user as having been shifted to the right, for example so that it is no longer located in the center of the field of view. In order to correct for the displacement caused by the deformation and detected by comparing the expected gaze vector and the actual gaze vector, image 94b may be projected in place of image 94a. In image 94b, the virtual content object 115 is shifted to the left of the image so that when light is emitted from the deformed display in a manner shifted to the right, the right displacement of the display cancels the left displacement of the content, and the virtual content object 115 appears to the user as being located in the center of the field of view. The pixel shifting may be achieved by illuminating different pixels of a spatial light modulator (SLM) within the projector 34 using an LED or other similar light source. In this adjustment method, it may be useful to transform the actual gaze angle or gaze vector and the expected gaze angle or gaze vector into a display coordinate system. For example, the system may expect the user's gaze to be at the center of the pixel at coordinates (x1, y1), but find that the user's gaze is actually at the center of the pixel at coordinates (x2, y2) in the display coordinate system. With this approach, the pixel shift can be adjusted by an amount equal to but in the opposite direction of the difference between the expected viewing coordinates determined by the user's gaze vector and the actual viewing coordinates. Alternative means for adjusting the rendering position of virtual content may also be used.

[0056] In some embodiments, Figure 3DAs shown, the correction can be accomplished by adjusting the projector 34 itself. To achieve full correction, the rendering position of the rendered object 62 must be moved from position 100 along the direction represented by vector 102 to the corrected position 104. To achieve full correction, the projector 34 can insert light 106 into the diffractive optical element of the optical display 36 at an adjusted angle so that the rendered object 62 is viewable closer to the expected position in the field of view. The user will perceive that the rendered object 62 has moved from position 100 along vector 102 to position 104. However, a full correction is not made. Instead, the rendered object 62 moves along the direction represented by vector 102 at a small portion of the distance between position 100 and position 104. To make a partial correction, the projector 34 therefore inserts light at an angle slightly steeper than the image light 94 but not as steep as the light 106.

[0057] Figure 4 Potential gaze vector errors 120 and 122 at two user interaction events are shown. Anatomically, when the eye is focused on a static image, such as augmented object 68, the eye is constantly moving. Natural eye movement typically results in an error or deviation of about 60 arc minutes relative to a central gaze vector, which corresponds to a straight line path from a selected anatomical landmark of the eye, such as center point 66 of eye 22, to augmented object 68. In some embodiments, the gaze vector can be defined as a straight line path from the center of the cornea of ​​the eye to augmented object 68, or can be defined as an extension of the visual axis of the eye. If the user is looking at augmented object 68, a gaze vector within an expected gaze vector range having a 60 arc minute radius relative to gaze vector 90 can be assumed to point to the location of augmented object 68. Figure 1 The gaze angle calculation module 46 in has some error for a given eye size, shape, etc., and this additional error adds about 30 arc minutes to the expected gaze vector range radius. When it is determined that the user is looking at the augmented object 68 and the calculated gaze vector is within a 90 arc minute radius of the gaze vector 90, then the system is considered to be well calibrated.

[0058] One challenge is that the relatively large 90 arc minute error radius in the gaze vector makes it difficult to determine the exact location of the actual perceived position of the augmented object 68. To account for the potential error, Figure 1 The statistics system 48 in will collect multiple data points to find trends. The trends in the determined gaze vectors are accumulated to find possible positions of the actual perceived position of the augmented object 68 as displayed by the system with a certain amount of deformation. Since it is still uncertain how much of the position difference is due to noise caused by eye movement or algorithm errors, and how much of the position difference is due to Figure 1The actual deformation of the display assembly 24 in the image may result in that a full correction of the gaze vector cannot be performed. For example, the correction increment may be between 0.01 and 0.5 arc minutes, such as 0.1 arc minutes. Alternatively, the correction increment may be based on the amount of deformation detected, with larger deformations requiring larger increments. For example, the correction increment may be two orders of magnitude smaller than the amount of deformation detected.

[0059] Figures 5A to 5D shows one of many scenarios that could unfold. Figure 5A , a number of gaze vector measurements are made that point to positions 124A through 124C. The open circles at positions 124A through 124C represent the measured gaze angles that have been calculated. The black circle at position 126 indicates that it is where the system expects the virtual object to be. The system uses the measured gaze vector positions 124A through 124C to calculate a statistically relevant position 130 where the virtual object may now be located. The measured gaze vectors at positions 124A through 124C indicate that the virtual object may have moved from the initially expected position 126 to a statistically determined changed position 130 along the direction represented by vector 128. In Figure 5B In FIG. 1 , the virtual object rendering position is moving in the direction indicated by vector 132, which is opposite to the direction indicated by vector 128 and smaller in magnitude. Assume that the virtual object is at position 130 and has moved to a new position 134 along the direction of vector 132 toward the expected rendering position 126.

[0060] exist Figure 5C In the example, other data points 124D to 124G are collected that statistically determine that the virtual object is at a changed position 136. Position 136 is located on the other side of position 126 compared to position 130. The change in the object's position is a result of the continuous deformation of the display system. When determining that the virtual object is at position 130 and / or that the virtual object has moved in the opposite direction, it may also be due to Figure 5A The noise data in the Figure 5B When adjusting the rendering position with vector 132 as shown, if Figure 5C As shown, the virtual object is actually perceived as being located at position 136, and it can be seen that by Figure 5B In the case of an incorrect recalibration, the virtual object may have been moved to a new position 138. Figure 5B The position of vector 132 represents how the system "believes" it is in Figure 5B is moving the virtual object, although you can now see that vector 132 is Figure 5C In different positions, to indicate Figure 5B How virtual objects are more likely to move when moving, because more data is now available to analyze the position of virtual objects. Figure 5B and 5C In the example, vector 132 has the same magnitude and direction. Vector 132 in Figure 5B and 5C The only difference between them is the position of vector 132. Figure 5B The position of vector 132 represents how the system believes it is moving the virtual object, while Figure 5C The position of the vector 132 in the figure indicates that when the virtual object is in Figure 5C What might happen if a is moved. Figure 5C As can be seen from FIG. 1 , the movement of the virtual object from position 136 to position 138 along the direction represented by vector 132 can be traced back to Figure 5B , is incorrect because the virtual object is moved further away from the expected position 126, rather than closer. In such examples, it is advantageous to make incremental adjustments rather than full adjustments.

[0061] exist Figure 5D In , other data points 124H to 124L are also collected, which indicate that the virtual object may be at a new position 140. The vector 132 is Figure 5D and Figure 5C in the same position to indicate that Figure 5B Position 140 has passed position 138, and the error caused by moving the virtual object to position 138 along the direction represented by vector 132 is thus eliminated.

[0062] FIG. 5A to FIG. 5D Not every possible scenario that could unfold is shown, only that moving virtual objects in smaller increments, as opposed to attempting to correct a single large movement of all movements of virtual objects due to deformation of the display assembly 24, is used to prevent a user from noticing incorrect adjustments that may be made based on noisy data.

[0063] Fig. 6A The view seen by the user is shown, including a real-world table 72 and a rendered object in the form of a coffee cup, which may be, for example, Figure 2B The coffee cup has been moved in the user's view as shown by vector 150. The label "display" at vector 150 indicates the undeformed shape, and the movement of the virtual object (as shown by vector 150) is caused by Figure 1 The deformation of the display component 24 is caused by.

[0064] Figure 6B Shown by Figure 1 The gaze angle calculation module 46 in the Figure 1 The view of the eye tracking camera 44 is used to calculate the movement of the gaze angle. Figure 1Deformation of display assembly 24 in causes movement of the gaze angle measured by eye tracking camera 44 along the direction indicated by vector 152. Fig. 6A and 6B The vectors 150 and 152 in are equal to each other. Therefore, Figure 6B The movement of the gaze angle calculated in Fig. 6A The virtual objects shown move identically within the user's field of vision, depending on the eye's tolerance and the reference Figure 4 Describe the tracking algorithm.

[0065] exist Figure 6C , the virtual object is incrementally moved in the direction indicated by vector 154. Vector 154 is opposite in direction to vector 152, but each vector is only a fraction of the length of vector 152. The sum of vectors 154 is equal to vector 152, but in opposite directions. If the virtual object moves the entire distance of all vectors 154, then over time the virtual object will return to its position before the display assembly 24 is deformed, as shown in FIG. Fig. 6A As shown, early movement of virtual objects due to display deformation is explained.

[0066] Figures 6A to 6C Shows Figure 1 4. Since the eye tracking camera 44 is not deformed, the measurements made by the eye tracking camera 44 result in measurements of the gaze angle that match the actual gaze angle of the user's eye 22 relative to the display, subject to the above-mentioned tolerances.

[0067] Figures 7A to 7E Together they show Figure 1 The deformation of the eye tracking camera 44 in combination with the deformation of the display can have an effect. Fig. 7A It shows that without considering the deformation of the camera, Figure 1 The deformation of the display component 24 in the image may result in the virtual object being fixed within the user's field of view. Fig. 7A and Fig. 6A It should be noted that any deformation of the eye tracking camera 44 will not affect the positioning of the virtual object within the user's field of view, such as Fig. 7A As shown, it affects the gaze vector calculated relative to the display. Therefore, even in the presence of camera deformation and taking into account the effect of camera deformation, Fig. 7A Still represents the movement of virtual objects within the user's field of view

[0068] Figure 7B Shown by Figure 1 The gaze angle calculation module 46 in the embodiment calculates the movement of the gaze angle based on the image data received from the eye tracking camera 44. Fig. 7AAs shown in , we do not consider the effect of camera deformation. Therefore, Figure 7B Assume that the eye tracking camera 44 is not deformed and thus corresponds to Figure 6B same.

[0069] Figure 7C 4 shows the effect of the calculated gaze angle when the effect of the deformation of the eye tracking camera 44 is taken into account but the effect of the deformation of the display is not taken into account. Figure 2A , the deformation of eye tracking camera 44 causes an error in the measurement of the position of eye 22 within the field of view of eye tracking camera 44. This error of eye tracking camera 44 causes the image provided by eye tracking camera 44 to be Figure 1 Therefore, the gaze angle calculated by the gaze angle calculation module 46 is affected by the deformation of the eye tracking camera 44. Figure 7C The calculated movement of the gaze angle due to deformation of the eye tracking camera 44 is shown, represented by vector 156. The error in the accuracy of the gaze angle caused by the camera deformation is usually relatively small relative to the display. The movement of the gaze angle caused by the camera deformation is usually on the order of about 1 / 20 of the movement of the gaze angle caused by the display deformation.

[0070] Fig.7D The resultant vector 158 of vectors 152 and 156 is shown. In this example, vector 156 representing the effect of camera deformation is at an angle less than 90° to vector 152 representing display deformation. Therefore, resultant vector 158 is longer than vector 152 representing the effect of display deformation.

[0071] exist Fig. 7E In FIG. 1 , the virtual object is moved within the user's field of view as indicated by vectors 160. The direction of each vector 160 is Fig.7D Each time the user interacts with a qualifying virtual object, the virtual object moves incrementally. Over a period of days or weeks, the correction vector 160 is accumulated to the resultant vector 158 within the anatomical eye tolerance plus any tolerance required by the tracking algorithm. As previously mentioned, the effect of camera deformation is relatively small. Fig. 7E Compared to the position shown, the virtual object is moved from a closer Fig. 7A Its position shown moves. In the case where the virtual coffee cup is intended to be presented to a user sitting at table 72, the coffee cup will therefore return to the top of the table or very close to the top of the table.

[0072] Fig. 8A and 8B and Fig. 7A and 7B Similar. Figure 8CIn FIG. 1 , the effect of the camera deformation results in a shift in the viewing angle, as shown by vector 164. The angle between vector 164 and vector 152 is greater than 90°.

[0073] Fig.8D The resultant vector 166 is shown. The resultant vector 166 represents the change in the calculated gaze angle due to the display deformation and the camera deformation. The resultant vector 166 is smaller than the vector 152 representing the effect of the display deformation. It can be said that the effect of the display deformation and the camera deformation is Fig.8D is in the "opposite" direction, while Fig.7D are in the “same” direction.

[0074] Fig. 9 A possible solution is shown when the influence of the display deformation and the influence of the camera deformation on the calculated gaze angle are in opposite directions. Figure 1 The correction system 30 in does not modify the position of the virtual object. For example, the user may not care whether the "play" button is located on the table or floating at a distance above the table. In addition, the play button should work normally as an interface element of the background application.

[0075] Figures 10A to 10C Shown as Fig. 9 An alternative solution to the solution in Fig. 10A , the change in gaze angle due to camera deformation is calculated starting when the viewing device is still new, also referred to as the "out-of-the-box" state. The change in the calculated eye position relative to the eye tracking camera due to camera deformation is periodically determined using the eye tracking camera 44 and data analysis of the generated eye tracking images. The methodology used to calculate the eye position or eye center at any particular time is similar to that described in reference Figure 2AMethodology described. The images of the eye 22 continue to be used as the basis for calculating the eye center position of each eye relative to one or more eye tracking cameras. Changes in the eye center position within the collected image frames can be attributed to changes in the position of the eye tracking camera relative to the eye. Although there are some errors associated with the fit of the device and the consistency of the user's use of the device, the use of multiple image frames in the data analysis improves the accuracy of determining eye tracking camera deformation. For example, the eye center calculated based on the eye tracking image taken during the device installation phase is determined to be at a certain coordinate location (x, y, z). Over time, eye tracking camera deformation may occur, causing the camera to tilt slightly downward. The position of the eye in the image captured by the deformed eye tracking camera will change, causing the eye center to be displayed as being at a higher y-axis position, such as coordinate location (x, y+1, z) within the eye tracking image. The change between these two coordinate locations is related to the change of the eye tracking camera, and the difference can be used to correct the rendering position of virtual content on the display. The deformation of the eye tracking camera is in Fig. 10A 170 and may occur in one, two or three dimensions.

[0076] The eye tracking camera 44 runs continuously at a rate of, for example, 30 frames per second. When determining deformations of the eye tracking camera 44, it is not necessary to wait for the user to interact with a particular virtual content rendering for data collection. A stream of image data is available for comparison to a baseline at any given time. Using a set of data points from the eye tracking camera 44 to compare to the baseline data provides a robust system. For example, using at least the last 10 images from the eye tracking camera 44 can improve accuracy compared to using only one or two images.

[0077] Fig. 10B Vector 166 is shown to be the sum of vectors 152 and 170. Vector 152 may be calculated by subtracting camera deformation vector 170 from vector 166. Since vector 152 represents the effect of the display deformation on the position of the virtual object, the virtual object may be moved in the opposite direction of vector 152.

[0078] exist Fig. 10C , the virtual object moves incrementally, as shown by vector 172. The sum of vector 172 is equal to vector 152, but in the opposite direction.

[0079] like Fig.11 As shown, viewing device 20 may also include reference system 180 to perform Figures 10A to 10CCalculations shown. The reference system 180 may include a number of baseline calculations, images, and / or measurements, such as interpupillary distance, eye center location, device fit information, eye shape information, visual axis location, and the like. The reference system 180 includes an out-of-the-box reference gaze angle 182 that is calculated when the viewing device 20 is still new. The out-of-the-box reference gaze angle 182 is stored in a memory. The reference system 180 also has an updated gaze angle 184 stored in the memory. The updated gaze angle 184 is calculated periodically and represents the most recently calculated gaze angle. The reference system 180 also includes a change 186 in the gaze angle due to camera deformation. The change 186 is the difference between the updated gaze angle 184 and the out-of-the-box reference gaze angle 182. The change 186 is in Fig. 10A The correction calculation unit 50 is used to calculate Fig. 10B Vector 152 in. Fig.11 Other aspects of the viewing device 20 in Figure 1 The viewing devices in FIG. 1 are the same, and like reference numerals indicate like parts. Although the out-of-the-box reference fixation angle is described as a baseline for comparison, any other eye feature, measurement, calculation, or image may be used instead of or in addition to the out-of-the-box reference fixation angle.

[0080] Fig.12 The method as described above is shown. At 200, a virtual object is displayed at a selected position, wherein the eye viewing the virtual object has an expected gaze direction. At 202, an expected gaze vector range is determined based on the selected position. At 204, a user interaction event is detected, wherein the user interacts with the virtual object. At 206, light reflected from the eye is captured by an eye tracking camera. At 208, a changed gaze vector is calculated based on the light captured by the eye tracking camera. At 210, a comparison is made between the determined changed gaze vector and the expected gaze vector range. Elements 200 to 210 can be combined into element 212, i.e., for detecting a deformation of a display component, wherein the deformation makes the virtual object viewable at a changed position, wherein the eye has a changed gaze direction. Element 212 can be performed without listing a specific element as element 200 to 210. In addition, although the gaze vector is calculated, a method other than the need to calculate the gaze vector can also be used to determine the gaze vector.

[0081] At 214, it is determined whether the actual gaze vector is outside the gaze vector range. If the determination at 214 is negative, the process returns to 200. If the determination at 214 is positive, the virtual content rendering position correction amount and direction amount are calculated. At 220, the virtual content rendering position is adjusted by the calculated correction amount and direction. Elements 218 and 220 can be combined into element 222, i.e., for displaying the virtual object at the corrected position, wherein the eye viewing the virtual object at the corrected position has a corrected gaze direction that moves closer to the expected gaze direction than the changed gaze direction.

[0082] Fig.13 A schematic diagram of a machine in the exemplary form of a computer system 900 is shown, having a set of instructions that can be executed to cause the machine to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine operates as a standalone device, or can be connected (e.g., networked) to other machines. Furthermore, while only a single machine is shown, the term "machine" should also be understood to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0083] The exemplary computer system 900 includes a processor 902 (e.g., a central processing unit (CPU) and / or a graphics processing unit (GPU)), a main memory 904 (e.g., a read-only memory (ROM), flash memory, a dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), and a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), which communicate with each other via a bus 908 with a laser driver chip 12 or other light source driver.

[0084] The computer system 900 may also include a disk drive unit 916 and a network interface device 920 .

[0085] The disk drive unit 916 includes a machine-readable medium 922 on which is stored one or more sets of instructions 924 (e.g., software) embodying any one or more of the methodologies or functions described herein. During execution of the software by the computer system 900, the software may also reside, in whole or in part, within the main memory 904 and / or within the processor 902, which also constitute machine-readable media.

[0086] The software may also be sent or received over a network 928 via the network interface device 920 .

[0087] Although the machine-readable medium 924 is shown as a single medium in the exemplary embodiment, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable medium" should also be taken to include any medium that can store, encode, or carry a set of instructions to be executed by a machine, as well as any medium that causes a machine to perform any one or more of the methodologies of the present invention. Thus, the term "machine-readable medium" should be taken to include, but is not limited to, solid-state memories, optical and magnetic media, and carrier signals.

[0088] The laser driver chip 12 includes a data storage area 161 and its own processor 162. The data storage area 161 is used to store instructions and data structures specific to the operation of the laser source. The processor 162 retrieves instructions from the data storage area and accesses the data structure to execute a routine to drive the laser source, thereby causing the laser source to generate laser light. The laser source forms part of a projector that receives data such as video. The scanner forms part of the projector so that the projector can display laser light in any pattern, color, saturation and other light qualities created by the projector based on the values ​​in the video data, in a two-dimensional area, and sometimes in a three-dimensional space.

[0089] Although a laser source and laser driver chip 12 have been shown and discussed, other display systems may be used. Other display systems may include, for example, displays utilizing light emitting diode (LED) technology, organic light emitting diode (OLED) technology, super luminescent diode (SLED), and the like.

[0090] While certain exemplary embodiments have been described and shown in the drawings, it is to be understood that these embodiments are merely illustrative and not restrictive of the invention, and that the invention is not limited to the specific constructions and arrangements shown and described, since various modifications may be understood by those skilled in the art.

Claims

1. A viewing device for displaying rendered content, comprising: a display assembly having a transparent optical display configured to display a virtual object at a selected location on the optical display, wherein an eye viewing the virtual object has an intended gaze direction and the virtual object is viewable by the eye at a location on a side of the optical display opposite the eye; A deformation detection system connected to the display assembly, the deformation detection system comprising: an eye tracking camera configured to capture light reflected from the eye; a gaze angle calculation module configured to calculate a measured gaze direction of the eye viewing the virtual object on the display assembly based on light captured by the eye tracking camera; and a deformation calculation unit configured to calculate a deformation of the optical display in a display deformation direction based on a measured gaze direction, the measured gaze direction being a changed gaze direction moved relative to the expected gaze direction in the display deformation direction, the changed gaze direction being caused by a deformation of at least one of the eye tracking camera and the optical display, a difference between the expected gaze direction and the changed gaze direction being in a first direction, and a change in the measured gaze direction due to the deformation of the eye tracking camera being in a second direction, and an angle of the first direction and the second direction relative to each other being less than 90 degrees; and A correction system connected to the deformation detection system to display the virtual object at a corrected position together with the display component, wherein the measured gaze direction of the eye viewing the virtual object at the corrected position is a corrected gaze direction, and the corrected gaze direction moves from the changed gaze direction toward the expected gaze direction.

2. The system according to claim 1, wherein: A difference between the expected gaze direction and the corrected gaze direction is smaller than a difference between the expected gaze direction and the changed gaze direction.

3. The system of claim 1, wherein: The change from the expected gaze direction to the altered gaze direction is in a first direction, and the measured change in gaze direction due to deformation of the eye tracking camera is in a second direction, and the first direction and the second direction are separated by more than 90 degrees relative to each other.

4. The system according to claim 1, further comprising: A reference system is provided for detecting a change in the measured gaze direction due to a deformation of the eye-tracking camera, wherein the correction calculation unit determines the change in the measured gaze direction due to the deformation of the display component by subtracting the change in the measured gaze direction due to the deformation of the eye-tracking camera from the change in the gaze angle detected by the deformation detection system.

5. The system according to claim 1, wherein: The deformation detection system comprises: A statistical system is configured to receive and analyze a plurality of measured gaze direction measurements, wherein the gaze angle calculation unit calculates a plurality of gaze directions, each gaze direction for a corresponding measurement, and the correction calculation unit determines the changed gaze direction based on the plurality of actual gaze direction measurements.

6. The system according to claim 1, wherein: The deformation detection system includes an attention generation unit configured to change the display of the virtual object and attract the user's attention to the virtual object before detecting the deformation.

7. The system according to claim 6, wherein: The attention generating unit is configured to attract the user's attention by changing a color of the virtual object.

8. The system according to claim 6, wherein: The attention generating unit is configured to attract the user's attention to the virtual object by reducing the size of the virtual object.

9. A method for displaying rendered content, comprising: displaying a virtual object at a selected location on the optical display by means of a display assembly having a transparent optical display, wherein an eye viewing the virtual object has an intended gaze direction and the virtual object is viewable by the eye at a location on a side of the optical display opposite the eye; capturing, with an eye-tracking camera, light reflected from the eye; calculating, with a processor, a measured gaze direction of the eye viewing the virtual object on the display assembly based on light captured by the eye tracking camera; calculating, with the processor, a deformation of the optical display in a display deformation direction based on a measured gaze direction, the measured gaze direction being a changed gaze direction moved in the display deformation direction relative to the expected gaze direction, the changed gaze direction being due to deformation of at least one of the eye-tracking camera and the optical display, a difference between the expected gaze direction and the changed gaze direction being in a first direction, and a change in the measured gaze direction due to the deformation of the eye-tracking camera being in a second direction, and the first direction and the second direction being at an angle of less than 90 degrees relative to each other; and With the aid of a correction system, the virtual object is displayed at a corrected position together with the display component, wherein the eyes viewing the virtual object at the corrected position have a corrected gaze direction, and the corrected gaze direction moves from the changed gaze direction toward the expected gaze direction.

Images