Concave color correction for improved color uniformity in head-mounted displays

By using eye monitoring sensors in a head-mounted display device to determine the user's gaze position and dynamically adjust the size and application area of ​​the color correction mask, the problem of inefficient color correction in the prior art is solved, and resource conservation and color uniformity are achieved Improvement.

CN112639943BActive Publication Date: 2024-11-19MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980054603.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-08-22
Filing Date
2019-06-25
Publication Date
2024-11-19
Estimated Expiration
2039-06-25

AI Technical Summary

Technical Problem

Existing head-mounted display (HMD) devices are inefficient and unnecessary to consume device resources when using software to correct color inhomogeneity, especially when users only need to view a portion of the display’s field of view, and the overall field of view is performed. Color correction.

Method used

Determine the user's gaze position through eye monitoring sensors, dynamically generate and apply a color correction mask of variable size, perform only fovea color correction in the gaze area, and perform less color correction in the user's peripheral visual area, based on gaze Weighted color correction mode can also be used for correction of larger surface areas.

Benefits of technology

Optimize device operation and user experience, save system resources, improve perceived fidelity of color rendering, and reduce battery consumption, providing a more uniform color display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112639943B_ABST
    Figure CN112639943B_ABST
Patent Text Reader

Abstract

A color correction mask programmatically generated on an HMD device in software while rendering a display is applied to a gaze area on a display field of view (FOV) on a head mounted display (HMD) device to optimize system resources. While the user operates the HMD device, an eye monitoring sensor is utilized to track the movement of the user's eyes to determine the gaze position of the user's eyes on the display FOV. Using the determined gaze position, a dynamic foveal gaze area is sized around the gaze position so that the foveal portion of the display FOV is color corrected, i.e., color non-uniformity is reduced or eliminated. In other implementations, a gaze-based weighted mode is implemented in which, relative to the foveal color correction mode, measurements of the user's full eye or eye socket are utilized to color correct a larger surface area of ​​the display FOV.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Head mounted display (HMD) devices may utilize optical systems to generate virtual images or holograms for viewing by a user. Generating full color holograms through a series of one or more diffractive waveguides results in color non-uniformity due to limitations inherent in the physical properties applicable to such waveguide-based displays. Color non-uniformity may be corrected at the hardware level through adjustments to the waveguides, or may be corrected at the software level by implementing algorithms to counteract the set color non-uniformity produced by the waveguides (or other components of the optical system).

[0002] When using software to correct color non-uniformity, device resources may be used unnecessarily and inefficiently. For example, while a device user may be viewing a portion of the display's field of view (perhaps to observe an object located to the side), color correction may be unnecessarily applied to the entire field of view (FOV). Summary of the invention

[0003] In a computing device with a display (such as an HMD device), an eye monitoring sensor is configured to enable determination of a user's gaze position to identify a specific area of ​​the display at which the user is gazing. A gaze area of ​​variable size surrounding the identified gaze position is selected, and the device generates and applies a color correction mask to the gaze area to produce a foveal color-corrected gaze area. Thus, the color correction mask can be optimized to a specific gaze area, and other portions of the display outside the gaze area may be less color-corrected due to being in the user's peripheral vision. As the user looks around the display, the color correction mask is dynamically generated and applied in real time to the gaze area around the identified gaze position.

[0004] The size of the foveal color corrected gaze region can be dynamically and variably selected based on various characteristics associated with the device and the user. For example, device characteristics can include displayed content, exit pupil size, processing latency, and device performance and capabilities. User characteristics can include, for example, pupil size, eye slew rate, and head movement.

[0005] In other implementations, a gaze-based weighted color correction mask can be applied to the display that is based on measurements of the user's entire eye or the orbit of the eye, as opposed to the pupil in foveal color correction as described above. Using these measurements, the color correction mask can be applied to a larger surface area of ​​the display relative to the gaze area that is foveally color corrected. Gaze-based weighting can be operated separately from the foveal color correction mode, or the two modes can be operated simultaneously. Switching between gaze-based weighting mode and foveal color correction mode can be responsive to user input or dynamically. For example, dynamic switching can occur when the power state of the device reaches a low threshold level or the user's eye rotation rate reaches a threshold rate. In both cases, switching to gaze-based weighting may be preferred because it uses lower battery power and does not use real-time pupil tracking when correcting the display.

[0006] The implementation of foveal color-corrected gaze regions and gaze-based weighted color correction enables optimization of device operation and user experience. For example, for foveal color correction, the user will not know that real-time color correction is being implemented because a uniform color correction region of the display FOV can be observed regardless of where the user gazes on the display. Depending on the size of the foveal color-corrected gaze region, the user will perceive a display that appears to be perfectly uniform in color, but in fact only the selected area around the gaze position of the display FOV is well corrected. Compared with manipulating other optical parameters, the current concave color correction of the display can improve device performance. For example, conventional static color correction schemes (such as achieving an overall brighter display) can enhance the perceived color fidelity at the expense of increased power consumption and device resource load. For battery-powered devices, this trade-off may be undesirable. By implementing non-uniform color correction across the entire device display, the current concave color correction can save system resources while enhancing the user's perception of high-fidelity color rendering.

[0007] The implementation of gaze-based weighted color correction complements the features of foveal color correction. While gaze-based weighting provides an optimized display FOV like foveal color correction, doing so can reduce the battery consumption rate. In addition, when the user's eye rotation rate exceeds a threshold rate during foveal color correction, the user can select or the device can dynamically switch to gaze-based weighting, thereby enabling color correction of a larger surface area of ​​the display without using pupil position.

[0008] This "Summary of the Invention" is provided to introduce some concepts in a simplified form, which will be further described in the detailed description below. This "Summary of the Invention" is neither intended to identify the key features or essential features of the claimed subject matter nor to be used to help determine the scope of the claimed subject matter. In addition, the claimed subject matter is not limited to implementations that solve any or all of the shortcomings noted in any part of this disclosure. It should be appreciated that the above-mentioned subject matter can be implemented as a computer-controlled device, a computer process, a computing system, or an article such as one or more computer-readable storage media. These and various other features will become clear by reading the following "Detailed Description of the Invention" and viewing the associated drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of an illustrative head mounted display (HMD) device system is shown;

[0010] Figure 2 shows the propagation of light in a waveguide by total internal reflection;

[0011] Figure 3 A diagram showing illustrative parameters of a waveguide design for correcting color non-uniformity across a display;

[0012] Figure 4 A diagram showing illustrative parameters of software when implementing a color correction mask on a display;

[0013] FIG. 5A to FIG. 5B shows an illustrative user gaze location on a display of an HMD device;

[0014] Figure 6 One or more illustrative sensors that monitor a user's eyes are shown;

[0015] Figures 7 and 8 A schematic diagram showing the xyz coordinates of a user's pupil determined using an eye monitoring sensor;

[0016] Fig. 9 An illustrative display field of view, gaze location, and foveal color-corrected gaze area are shown;

[0017] Fig.10 A taxonomy of factors influencing a dynamically sized foveal color-corrected gaze region is shown;

[0018] Fig.11 Shown based on Fig.10 The factors of the taxonomy shown are the dynamics of the foveal gaze color correction area;

[0019] Fig.12 Shows the use of Fig.10Exemplary effects of exemplary factors on the size of the foveal gaze area are shown;

[0020] Fig.13 A schematic diagram showing eye gyration rate relative to the device is shown;

[0021] Fig.14 A continuous field of view (FOV) of a user tracking a moving object on a display is shown;

[0022] Fig.15 shows a continuous FOV of a user viewing a static object on a display;

[0023] Figures 16 to 18 illustrative foveal color corrected gaze regions of varying sizes across a display FOV are shown;

[0024] Fig.19 An illustrative display FOV with a color-corrected peripheral region is shown;

[0025] Fig. 20 An illustrative eye monitoring sensor for gaze-based weighting is shown;

[0026] Fig.21 shows illustrative adjustable positions of eye monitoring sensors when measuring foveal color correction relative to gaze-based weighting;

[0027] Fig. 22 An illustrative correction region for gaze-based weighting is shown;

[0028] Fig.23 showing an inner region of variable size for a gaze-based weighted correction region;

[0029] Fig.24 An illustrative switch between a gaze-based weighting mode and a concave color correction mode is shown;

[0030] Fig.25 An illustrative dual operation of a gaze-based weighting mode and a concave color correction mode is shown;

[0031] Figure 26 to Figure 28 An illustrative process performed by an HMD device is shown;

[0032] Fig.29 is a diagram of an illustrative example of a virtual reality or mixed reality HMD device;

[0033] Fig.30 A block diagram showing an illustrative example of a virtual reality or mixed reality HMD device; and

[0034] Fig.31is a simplified block diagram of an illustrative computing device that can be used to implement the present concave color correction for improving color uniformity of a head-mounted display.

[0035] Like reference numerals indicate like elements in the drawings. Unless otherwise noted, the elements are not drawn to scale. DETAILED DESCRIPTION

[0036] Figure 1 A block diagram of an illustrative near-eye display system 100 is shown that may incorporate a combination of a diffractive optical element (DOE) that provides incoupling of incident light into a waveguide, exit pupil expansion in two directions, and outcoupling of light out of the waveguide. Near-eye display systems are commonly used in head-mounted display (HMD) devices, such as in industrial, commercial, and consumer applications. Other devices and systems may also use near-eye systems, as described below. Near-eye display system 100 is an example used to provide context and illustrate various features and aspects of the present concave color correction for improving color uniformity of a head-mounted display.

[0037] The system 100 may include one or more imagers (indicated by reference numeral 105) used in conjunction with the optical system 110 to deliver an image as a virtual display to the user's eye 115. The imager 105 may include, for example, RGB (red, green, blue) light emitting diodes (LEDs), LCOS (liquid crystal on silicon) devices, OLED (organic light emitting diode) arrays, MEMS (micro-electromechanical systems) devices, or any other suitable display or micro-display operating in transmission, reflection, or emission. The imager 105 may also include mirrors and other components that enable a virtual display to be constructed and provide one or more input beams to the optical system. The optical system 110 may generally include a magnifying optical device 120, a pupil forming optical device 125, and one or more waveguides 130. In some implementations, the imager 105 may include or incorporate an illumination unit and / or light engine (not shown) that may be configured to provide illumination within a specific wavelength and intensity range.

[0038] In a near-eye display system, the imager does not actually shine an image onto a surface such as a glass lens to create a visual display for the user. This is not feasible because the human eye cannot focus on such close objects. Instead of producing a visible image on a surface, the near-eye display system 100 uses pupil forming optics 125 to form a pupil, and the eye 115 acts as the last element in the optical chain and converts the light from the pupil into an image on the retina of the eye as a virtual display.

[0039] Waveguide 130 facilitates light transmission between the imager and the eye. One or more waveguides may be used in a near-eye display system because they are see-through and because they are generally small and lightweight (which is desirable in applications such as HMD devices, where it is generally desirable to minimize size and weight for performance and user comfort reasons). For example, waveguide 130 may enable imager 105 to be positioned undisturbed, e.g., at the side of a user's head or near the forehead, while leaving only a relatively small, light, and transparent waveguide optical element in front of the eye. In one implementation, waveguide 130 operates using the principle of total internal reflection, such as Figure 2 As shown, light can be coupled between various optical elements in system 100.

[0040] The software color correction mask 140 utilizes an algorithm to correct for color non-uniformity issues on the display that may be caused, for example, by the image propagating along the waveguide. The color correction mask software may use various techniques known in the art to apply a color that blends with the displayed color within the masked area to preserve the desired color and thereby eliminate color non-uniformity.

[0041] The eye monitoring sensor 135 may include hardware configured to monitor the user's eyes by detecting the position of the user's pupils and identifying the corresponding position on the display of the HMD device. A light sensor such as infrared light may radiate at the user's eyes and be picked up by a camera or other optical sensor to determine the relative position of the pupil within the eye. An optical sensor such as a camera may pick up the user's pupil to determine its relative position. The pupil position may be identified according to various axes of xyz coordinates or xy coordinates that may identify the user's pupil. Figure 29 to Figure 30 The discussion of provides additional illustrative discussion of identifying the gaze location of a user's eyes while operating an HMD device.

[0042] Figure 3 and Figure 4 Schematic diagrams of factors that influence parameters when manufacturing color correction hardware and software are shown, respectively. Waveguide design 305 is a method of implementing color correction at the hardware level (as shown by numeral 310). The design of the waveguide can be customized to correct color non-uniformity when the imaging light travels to the display of the HMD device. Alternatively, a color correction mask that can be implemented programmatically in software can be configured at the HMD device and can be executed by one or more processors, such as at a graphics processing unit (GPU) or a central processing unit (CPU). Parameters 315 considered when designing software and hardware include field of view 320, contrast at the display 325, and other parameters 330. The software also considers the impact of color mixing 405 when applying the mask to the colors on the display.

[0043] Figure 5A and Figure 5B An illustrative eye 115 and a display field of view (FOV) 505 associated with the HMD device are shown. In each example, a user's pupil 510 is fixated on a particular point on the display field of view, illustratively indicated as gaze locations 520 and 525 of the user.

[0044] like Figure 6 As shown, eye monitoring sensor 135 is utilized to track and detect characteristics of the user's eyes. For example, the eye monitoring sensor can track pupil position 605 to determine the gaze position on the display FOV ( FIG. 5A to FIG. 5B ). Figure 7 and Figure 8 The eye coordinates of the user's pupil 510 along the xyz axis are illustratively shown. Figure 5A The coordinates can be translated to a specific gaze location on the display as shown in Figure B. The eye monitoring sensor can also track the pupil size of the user's eye 610. The pupil size can be measured when the HMD device is initialized and checked periodically to measure the dilation and contraction of the pupil.

[0045] Fig. 9 An illustrative display FOV 905 is shown in which a detected gaze position 910 is utilized to determine a foveally color-corrected gaze region 915 on the display FOV. The foveally color-corrected gaze region is the portion of the display FOV that is visually optimized with a color correction mask. Applying a mask only to the gaze region can save system resources, such as processing power and battery consumption. For example, a mask can be generated and applied to a concave portion of the screen while the HMD device is in use, as opposed to applying the mask to the entire display FOV (which inherently includes the user's suboptimal peripheral vision).

[0046] The foveal color corrected gaze region may be dynamically sized based on various characteristics specific to the HMD device and specific to the user. Fig.10 An illustrative taxonomy of factors that influence a passively sized foveal color-corrected gaze region 1005 is shown. Factors unique to the HMD device 1010 include displayed content 1015, exit pupil size 1020, latency 1025, and device capabilities (e.g., battery life, processing power, and available memory) 1030. User-specific factors 1035 include pupil size (dilation and constriction) 1040, eye gyration rate 1045, and head motion 1050.

[0047] Using these factors, the HMD device employs a multivariate approach to determine the appropriate size of the foveal color corrected gaze region. Various factors may be measured periodically or continuously while the device is in use using various sensors or information available to the HMD device (including the eye monitoring sensors discussed above). The HMD device may be able to access and use device-local data related to Fig.10 Various information about the characteristics of the display. Such accessible information may include, for example, exit pupil size, remaining battery life, performance capabilities of the processor (e.g., speed in GhZ), extensive use or lack of use of the processor, and latency. The HMD device may employ content recognition software to measure the complexity or resource intensity of the displayed content.

[0048] Information may be collected periodically to provide real-time determination and adjustment of the size of the foveal color corrected gaze region. The information collected or sensed may be performed upon initial use of the HMD device, wherein based on the information, the size of the foveal color corrected gaze region may be determined and set. The size of the foveal color corrected gaze region may then be adjusted during use, or may remain unchanged after initialization.

[0049] Fig.11 Shows the use of Fig.10 Schematic diagram of the dynamics of the size of the foveal color-corrected gaze area for one or more factors shown. Each circle around the gaze position shows an exemplary size of the foveal color-corrected gaze area that defines the limits and boundaries of the programmatically implemented color correction mask. Thus, the eye monitoring sensor and the software-implemented color correction mask interoperate with each other to deliver a customized color correction mask to the user that optimizes system resources and user experience. That is, the HMD device can operate with low latency or lag while still providing the user with an optimized visual experience of the display FOV. The user may not be aware of the real-time application of the color correction mask because the mask tracks the user's gaze position on the display. In addition, the user may not be aware that the color correction mask is only applied to a portion of the display FOV because the color correction area follows wherever the user gazes. Therefore, system resources can be saved without interfering with the user experience.

[0050] Fig.12 Shows Fig.10 The effects of the factors shown on a passively sized foveal color-corrected gaze region, where the region can expand or contract based on the relative measured extent detected for each factor. Fig.12 All factors are shown in FIG. 1 , but in operation, the HMD device may consider one, two, or any multiple of the factors when determining or adjusting the size of the foveal color-corrected gaze region.

[0051] Threshold 1205 may be assigned to one or more factors that, when satisfied, trigger the HMD device to adjust the size of the foveal color corrected gaze area. For example, eye turn rate, latency, system performance, etc. may be assigned to one or more thresholds that trigger size adjustment. If the latency exceeds the threshold time, the size of the foveal color corrected gaze area may be reduced to reduce the latency. Conversely, if the latency is minimal, the HMD device may expand the foveal color corrected gaze area to provide a wider range of color uniformity to the user. Thus, along Fig.12 In the bidirectional line in , there may be one or more thresholds to trigger size adjustment. In another embodiment, the foveal color corrected gaze area can be operated along a sliding scale for each factor.

[0052] Some factors may be weighted more heavily than others in determining the size of the foveal color-corrected gaze zone. For example, while system performance may be optimal, if the displayed content is essential (such as an article or book that the user is reading), the foveal color-corrected gaze zone may be reduced despite the device's expanded capabilities. This judicious consideration may save battery life by not unnecessarily expanding the zone, for example, by applying a mask to the last line of a page when the user is currently reading the first line.

[0053] Continuing with the example where a user reads an article or book, the user's eye turn rate may be relatively high because the user continuously manipulates their eyes from left to right on the display. The displayed content is simple and can make the movement of the user's eyes predictable and narrowed to only one line at a time. The gaze area can span several lines on the page to manage the user's high turn rate. Spanning more than a few lines of the foveal color-corrected gaze area may unnecessarily waste device resources, such as battery power, because the user will predictively read only many words on the page at a time. Device resources can also be preserved by removing the color correction mask from the previously viewed portion. Therefore, the predictability of applying the foveal color-corrected gaze area to an e-book provides proactive display adjustment to provide a more user experience while saving system resources.

[0054] Fig.13 A schematic diagram is shown in which a user's eye gyration rate 1045 may be relative to device 1305. In some cases, the user's eyes may be moving rapidly, but if the user's eyes are tracking an object moving on a display, the user's eye gyration rate may be assigned a relatively slower value. Fig.14The user's pupil 510 is shown sequentially tracking a bird 1410 flying over the display FOV 1405. While the user's pupil can move, the image displayed and observed by the user is similar in each sequential scene, so the transition rate may not be as fast as in other scenes.

[0055] Fig.15 shows that the user's eye rotation rate can be compared Fig.14 A relatively faster example of the example shown. The user's pupil moves from box 1510 to star 1515 on display FOV 1505. The display does not change across the sequence, and the user is not tracking any objects. The user's pupil changes direction while the screen remains stationary. Therefore, the user's eye gyration rate relative to the device is fast.

[0056] Figures 16 to 18 Shown based on Fig.10 Illustrative sizes of the foveal color corrected gaze region on the display FOV 1605 around the gaze location 1610 for the factors shown. Figures 16 to 18 As shown, the size of the corresponding foveal color-corrected gaze areas 1615, 1715, and 1815 is enlarged based on an evaluation of specific factors to provide an optimized user experience while operating the HMD device. The gray area within the foveal color-corrected gaze area graphically represents the color correction mask applied to the area. The remaining area 1620 within the display FOV 1605 may be within the user's peripheral field of view and does not receive a color correction mask, and the color uniformity relative to the foveal color-corrected gaze area may not be optimal. The remaining area 1620 may not be corrected by the software or may be corrected to a lesser extent than the gaze area.

[0057] Fig.19 An illustrative display FOV 1605 is shown in which the HMD device employs a color correction mask within a foveally color corrected gaze region 1915 and a suboptimal mask within a peripheral region 1925 that may be in the user's peripheral field of view. The peripheral region may be color corrected to a lesser extent relative to the gaze region to bridge the gap between the foveally color corrected gaze region and the uncorrected region represented by numeral 1620.

[0058] Fig. 20is a schematic diagram in which an eye monitoring sensor measures characteristics of a user's eye to provide color correction on a display using a gaze-based weighting feature, as opposed to the foveal color correction feature described above. The eye monitoring sensor 135 measures the position and size of the full or entire eye 115 or the orbit of the eye, and uses the corresponding measured sizes as a basis for color correction of the display FOV, as representatively shown by numerals 2005 and 2010. Measuring the full or entire eye includes measuring what is exposed and observable by the eye monitoring sensor, i.e., the surface area of ​​the eye facing outward from the user's head. As shown in FIG. Fig. 20 As shown, the eye may be measured based on the diameter or radius of the user's eye or the eye socket or other distance measurement.

[0059] Fig.21 A schematic diagram is shown in which the eye monitoring sensor 135 is repositioned depending on the color correction mode used (i.e., gaze-based weighting or foveal color correction). In order to enable the eye monitoring sensor to measure a larger surface area of ​​the entire eye or the orbit of the eye, the position of the eye monitoring sensor relative to the sensor is repositioned outward from the eye when measuring the user's pupil for foveal color correction. The eye monitoring sensor can be adjusted by repositioning the lens of the sensor or by sliding on a movable track that is powered to move forward and backward in two directions.

[0060] exist Fig.21 , the solid line representatively shows the measurement of the user's eye for gaze-based weighting, while the dashed line representatively shows the measurement of the user's pupil for foveal color correction. Variations in sensor position enable the sensor to capture a larger or smaller surface area of ​​the corresponding pattern.

[0061] Fig. 22 An illustrative representation of a display FOV 1605 with a gaze-based color correction region 2205 is shown. As illustratively depicted by dashed lines extending from opposite ends of each of the user's eyes, the gaze-based color correction region 2205 is derived from the size of the user's eyes, as described above and as Figures 16 to 19 The pupil position in the foveal color corrected gaze region shown is reversed.In other embodiments, the gaze-based color correction region may be based on the size of the eye's orbit rather than the size of the eye (not shown).

[0062] Fig.23A schematic diagram is shown in which a gaze-based color correction region 2205 has different corrections based on various calculations to accommodate a larger correction on the exit pupil. For example, the gaze-based color correction region has multiple inner regions 2305 that are color corrected to different degrees. In this example, the most finely corrected inner region is the center of the color correction region 2205, and the degree of color correction decreases as other inner regions are placed outward from the center. The inner region located in the center may be the most color corrected because this is typically where the user's eyes are focused. As shown, a medium level of color correction is applied in the middle inner region, while a relatively low level of color correction is applied to the outermost inner region. Changing the color correction region can save system resources by applying the finest color correction to the area where the user typically focuses and reducing color correction resources in the area where the user is less typically focused.

[0063] Gaze-based weighted color correction can also achieve variable gaze size 2310, such as Fig.23 23. The size of the inner region is dynamic and can vary for different uses and different users. The size 2310 can depend on the current measurement of the size of the eye or eye socket ( Fig. 20 ), and eye gyration rate relative to the device, and other dependent factors. For example, a higher eye gyration rate relative to the device may indicate an increase in the centrally located color correction area, thereby finely optimizing more of the display to cater for the higher gyration rate. Larger sized eyes or eye sockets may also contribute to increasing the centrally located inner area to cater for the larger surface area viewed by the user. The color correction area 2205 may also be increased or decreased based on the size of the user's eyes or eye sockets.

[0064] Fig.24 A schematic diagram is shown in which an HMD device operating the present color correction feature can switch modes between gaze-based weighting, concave color correction, or operating in both modes simultaneously 2405. The device can switch modes in response to user input 2410 or dynamically 2415. User input can be accomplished via a user interface on the display, including any of voice, touch, gesture, or other input mechanisms provided by the corresponding device.

[0065] Dynamic mode switching can be performed automatically by the device based on the user's measured power state 2420 or pupil movement 2425. For example, foveal color correction may take more resources to operate relative to gaze-based weighting. When the device's available battery life drops below a threshold level, the device can automatically switch to gaze-based weighting, at least until the device is in a charging state or has been charged above the threshold.

[0066] If the movement of the pupil relative to the device extends above a threshold rate, the device can switch to gaze-based weighting. If the eye monitoring sensors cannot keep up with the user's pupil position and / or the color correction mask cannot be applied quickly enough to keep up with the user's fast-paced pupil movement, foveal color correction may not provide a satisfactory user experience. Switching to a gaze-based weighting mode can increase the surface area to which the color correction mask is applied because it can correct a larger FOV relative to the gaze area of ​​foveal color correction. That is, gaze-based weighting is based on a measurement of the user's eyes or eye sockets, thereby providing a higher corrected FOV relative to pupil utilization.

[0067] Fig.25 2505 is a schematic diagram showing dual operations of gaze-based weighted and concave color correction that an HMD device can perform simultaneously. In such an implementation, a gaze-based weighted color correction mask can be applied to the FOV while the HMD device continues to monitor and color correct a specific location of the user's pupil on the FOV. For example, if the user gazes at a smaller corrected portion of the color correction area (e.g., Fig.23 The foveal color correction mode can correct a particular gaze position to a greater extent if the foveal color correction mode is used for a low level correction area of ​​the foveal color correction region. The two modes complement each other and are configured to take into account the correction applied by the other mode.

[0068] Fig.26 is a flow chart of an illustrative method 2600 in which an HMD device applies color correction to a foveal gaze area on a display. Unless otherwise specified, the methods or steps shown in the flow charts and described in the accompanying text are not limited to a particular order or sequence. In addition, depending on the requirements of such implementation, certain methods or steps thereof may occur or be performed simultaneously, and not all methods or steps must be performed in a given implementation, and some methods or steps may be optionally utilized.

[0069] In step 2605, the user's eyes are monitored while the user operates the HMD device. In step 2610, the size of the eye or the eye socket is measured. In step 2615, the size of the correction region is determined using the measured size, wherein the correction region is an area on the display of the HMD device that is color corrected. In step 2620, the size of an inner region of the correction region is determined, wherein the inner region is color corrected to a greater extent than the rest of the correction region. In step 2625, color correction is applied to the inner region and the correction region.

[0070] Fig. 2727 is a flow chart of an illustrative method 2700 in which a computing device programmatically applies a color correction mask to a foveal gaze region. In step 2705, coordinates for a pupil of a user are received while the computing device is in use. In step 2710, a gaze location on a display is identified based on the received coordinates for the pupil. In step 2715, the foveal gaze region is evaluated based on at least the identified gaze location and a current size of the user's pupil while operating the computing device. In step 2720, a color correction mask is programmatically applied only to the foveal gaze region, wherein the color correction mask optimizes a rendered image in the region.

[0071] Fig.28 28 is a flow chart of an illustrative method 2800 in which an HMD device switches between color correction modes for a display. In step 2805, the position and size of a user's eyes are monitored. In step 2810, the size of a correction area on the display is set using information derived from the eye monitoring. In step 2815, while rendering an image on the display, color correcting a portion of the display, where the portion is the set size of the correction area. In step 2820, dynamically or in response to user input, switching between color correction modes includes color correcting the set size based on the user's pupil, the user's entire eye or eye socket, or both the pupil and the entire eye or eye socket.

[0072] Fig.29 One specific illustrative example of a see-through, mixed reality, or virtual reality HMD device 2900 is shown. Fig.30 A functional block diagram of device 2900 is shown. HMD device 2900 includes one or more lenses 2902 that form part of a see-through display subsystem 2904, such that images can be displayed using lenses 2902 (e.g., using projection onto lenses 2902, one or more waveguide systems (such as near-eye display systems) incorporated into lenses 2902, and / or in any other suitable manner). HMD device 2900 also includes one or more outward-facing image sensors 2906 configured to acquire images of a background scene and / or physical environment being viewed by a user, and may include one or more microphones 2908 configured to detect sounds such as voice commands from a user. Outward-facing image sensors 2906 may include one or more depth sensors and / or one or more two-dimensional image sensors. As described above, in an alternative arrangement, instead of incorporating a see-through display subsystem, a mixed reality or virtual reality display system may display mixed reality or virtual reality images through a viewfinder mode for outward-facing image sensors.

[0073] As described above, the HMD device 2900 may also include a gaze detection subsystem 2910 configured to detect the direction or position of the gaze direction or focus of each eye of the user. The gaze detection subsystem 2910 may be configured to determine the gaze direction of each eye of the user in any suitable manner. For example, in the illustrative example shown, the gaze detection subsystem 2910 includes one or more glint sources 2912 (such as infrared light sources) configured to cause glints of light to be reflected from each eyeball of the user and one or more image sensors 2914 (such as inward-facing sensors) configured to capture images of each eyeball of the user. Glitter changes from the user's eyeballs and / or the user's pupil position determined based on image data collected using (multiple) image sensors 2914 can be used to determine the gaze direction.

[0074] Additionally, the location where gaze rays projected from the user's eyes intersect the external display can be used to determine the object the user is gazing at (e.g., a displayed virtual object and / or a real background object). Gaze detection subsystem 2910 can have any suitable number and arrangement of light sources and image sensors.

[0075] The HMD device 2900 may also include additional sensors. For example, the HMD device 2900 may include a global positioning system (GPS) subsystem 2916 to allow determination of the location of the HMD device 2900. This may be helpful in identifying real objects, such as buildings, etc., that may be located in the user's immediate physical environment.

[0076] The HMD device 2900 may also include one or more motion sensors 2918 (e.g., inertial sensors, multi-axis gyroscopes, or accelerometers) to detect the motion and position / orientation / posture of the user's head when the user wears the system as part of a mixed reality or virtual reality HMD device. The motion data can be potentially used for gaze detection and image stabilization together with the eye tracking flicker data and the outward-facing image data to help correct blur in the image from the outward-facing (multiple) image sensor 2906. The use of motion data can allow tracking of changes in gaze position even if the image data from the outward-facing (multiple) image sensor 2906 cannot be resolved. The motion sensor can also be used to determine the extent of the user's head movement, which the HMD device can use to expand or contract the foveal color-corrected gaze area.

[0077] Additionally, the motion sensor 2918, as well as the microphone(s) 2908 and gaze detection subsystem 2910, may also be used as user input devices, allowing a user to interact with the HMD device 2900 via gestures with the eyes, neck, and / or head, and in some cases, via verbal commands. Fig.29 and Fig.30The sensors shown in and described in the accompanying text are included for purposes of example and are not intended to be limiting in any way, as any other suitable sensors and / or sensor combinations may be utilized to meet the needs of a particular implementation. For example, biometric sensors (e.g., for detecting heart and breathing rates, blood pressure, brain activity, body temperature, etc.) or environmental sensors (e.g., for detecting temperature, humidity, altitude, UV (ultraviolet) light levels, etc.) may be used in certain implementations.

[0078] The HMD device 2900 may also include a controller 2920, such as one or more processors, having a logic subsystem 2922 and a data storage subsystem 2924 that communicate with sensors, gaze detection subsystem 2910, display subsystem 2904, and / or other components via a communication subsystem 2926. The communication subsystem 2926 may also facilitate the display system to operate with remote resources such as processing, storage, power, data, and services. That is, in some implementations, the HMD device may operate as part of a system that may allocate resources and capabilities between different components and subsystems.

[0079] The storage subsystem 2924 may include instructions stored thereon that are executable by the logic subsystem 2922 to, for example, receive and interpret input from sensors, identify the user's position and motion, identify real objects using surface reconstruction and other techniques, and dim / fade the display based on the distance to the object so that the object can be seen by the user, among other tasks.

[0080] The HMD device 2900 is configured with one or more audio transducers 2928 (e.g., speakers, headphones, etc.) so that audio can be used as part of a mixed reality or virtual reality experience. The power management subsystem 2930 may include one or more batteries 2932 and / or a protection circuit module (PCM) and an associated charger interface 2934 and / or a remote power interface for powering components in the HMD device 2900.

[0081] It will be appreciated that the HMD device 2900 is described for purposes of example and not limitation. It will also be appreciated that the display device may include additional and / or alternative sensors, cameras, microphones, input devices, output devices, etc., in addition to those shown, without departing from the scope of the present arrangement. Additionally, the physical configuration of the HMD device and its various sensors and subcomponents may take a variety of different forms without departing from the scope of the present arrangement.

[0082] Fig.31 An illustrative architecture 3100 of a computing device capable of executing the various components described herein to provide concave color correction to improve color uniformity of a head mounted display is shown. Fig.31 The illustrated architecture 3100 illustrates a system architecture that may be suitable for various computing devices such as smartphones, tablet computers, laptop computers, personal computers, etc. For example, a user may hold a smartphone to his face, where the smartphone may provide a virtual or mixed reality experience. A front-facing camera (i.e., facing the user) may detect the user's eye position, while a rear-facing camera (i.e., facing away from the user) may be used to capture real-world images.

[0083] Fig.31 The illustrated architecture 3100 includes one or more processors 3102 (e.g., central processing units, graphics processing units, etc.), system memory 3104 (including RAM (random access memory) 3106 and ROM (read only memory) 3108), and a system bus 3110 that operatively and functionally couples the components in the architecture 3100. A basic input / output system containing basic routines that help to pass information between elements in the architecture 3100, such as during a boot process, is typically stored in the ROM 3108. The architecture 3100 also includes a mass storage device 3112 that is used to store software code for implementing applications, file systems, and operating systems, or other computer-executed code. The mass storage device 3112 is connected to the processor 3102 through a mass storage controller (not shown) connected to the bus 3110. The mass storage device 3112 and its associated computer-readable storage media provide non-volatile storage for the architecture 3100. Although the descriptions of computer-readable storage media contained herein refer to mass storage devices, such as a hard disk or CD-ROM drive, those skilled in the art will appreciate that computer-readable storage media may be any available storage media that architecture 3100 may access.

[0084] The architecture 3100 also supports a sensor package 3130 that includes one or more sensors or components configured to detect parameters describing the environment. For example, the sensor may include a gyroscope to determine the orientation of the device, such as when a user holds a smartphone to his face. The camera may also be used to identify features of the pupil, such as the location and size of the pupil. The sensor may be configured to operate continuously or periodically. The architecture also supports a power supply and / or battery assembly (identified collectively by reference numeral 3115). For example, one or more batteries or power packs may be rechargeable or replaceable to facilitate portability, mobility, and reuse.

[0085] By way of example and not limitation, computer storage media may include volatile and nonvolatile removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer readable media include, but are not limited to, RAM, ROM, EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), flash memory or other solid-state storage technology, CD-ROM, DVD, HD-DVD (high-definition DVD), Blu-ray or other optical storage devices, cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the architecture 3100.

[0086] According to various embodiments, the architecture 3100 can operate in a networked environment using logical connections to remote computers through a network. The architecture 3100 can be connected to the network through a network interface unit 3116 connected to the bus 3110. It should be understood that the network interface unit 3116 can also be used to connect to other types of networks and remote computer systems. The architecture 3100 can also include an input / output controller 3118 for receiving and processing input from many other devices, including a keyboard, a mouse, a touch pad, a touch screen, a control device (such as buttons and switches), or an electronic pen ( Fig.31 Similarly, the input / output controller 3118 may provide output to a display screen, user interface, printer, or other type of output device (not shown in FIG. Fig.31 (not shown in the figure).

[0087] The architecture 3100 may include a speech recognition unit (not shown) to facilitate user interaction via voice commands, a natural language interface, or by interacting with a personal digital assistant (such as provided by Microsoft Corporation). The architecture 3100 may include a gesture recognition unit (not shown) to facilitate user interaction with a device supporting the architecture through sensed gestures, motions, and / or other sensed inputs.

[0088] It should be understood that the software components described herein, when loaded into the processor 3102 and executed, can convert the processor 3102 and the entire architecture 3100 from a general-purpose computing system to a special-purpose computing system customized to facilitate the functions introduced herein. The processor 3102 can be composed of any number of transistors or other discrete circuit elements (which can present any number of states individually or collectively). More specifically, the processor 3102 can operate as a finite state machine in response to executable instructions contained in the software modules disclosed herein. These computer executable instructions can convert the processor 3102 by specifying how the processor 3102 converts between states, thereby converting the transistors or other discrete hardware elements constituting the processor 3102.

[0089] Encoding the software modules proposed herein can also convert the physical structure of the computer storage readable medium proposed herein. In different implementations of this specification, the specific conversion of the physical structure can depend on various factors. Examples of such factors may include, but are not limited to, the technology used to implement the computer storage readable medium, whether the computer storage readable medium is characterized as primary storage or auxiliary storage, etc. For example, if the computer storage readable medium is implemented as a semiconductor-based memory, the software disclosed herein can be encoded on the computer storage readable medium by converting the physical state of the semiconductor memory. For example, the software can convert the state of transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software can also convert the physical state of these components to store data thereon.

[0090] As another example, the computer readable storage medium disclosed herein can be implemented using magnetic or optical technology. In such an implementation, when software is encoded therein, the software proposed herein can convert the physical state of magnetic or optical media. These conversions can include changing the magnetic properties of a specific location in a given magnetic medium. These conversions can also include changing the physical characteristics or properties of a specific location in a given optical medium to change the optical properties of these locations. Without departing from the scope and spirit of this specification, other conversions of physical media are possible, and the aforementioned examples are provided only to facilitate this discussion.

[0091] In view of the above, it should be understood that many types of physical transformations occur in the architecture 3100 in order to store and execute the software components presented herein. It should also be understood that the architecture 3100 may include various types of computing devices, including wearable devices, handheld computers, embedded computer systems, PDAs, and other types of computing devices known to those skilled in the art. It is also contemplated that the machine architecture 3100 may not include Fig.31 All components shown may include Fig.31Other components not explicitly shown in the figure may also be used in conjunction with Fig.31 The architecture shown is a completely different architecture.

[0092] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for calibrating a display of a head mounted display (HMD) device suitable for rendering a virtual image, comprising: monitoring an eye of a user while operating the HMD device, wherein the HMD device comprises a near-eye waveguide-based display, the near-eye waveguide-based display comprising a waveguide, at least one diffractive optical element for in-coupling light for the virtual image into the waveguide, and at least one diffractive optical element for out-coupling the virtual image light from the waveguide to a pupil of the user, wherein each diffractive optical element and the waveguide inherently provide color non-uniformity in a virtual image rendered on the near-eye waveguide-based display; measuring the size of the eye or the orbit of the eye; using the measured size, determining a size of a correction region for color correction on the near-eye waveguide-based display; determining a size of an inner region within the correction region, wherein the inner region is color corrected to a higher level relative to a remainder of the correction region; and Respective levels of color correction are applied to the inner region and the correction region to correct color non-uniformity in the inner region and the correction region of the near-eye waveguide-based display. The method of claim 1 , wherein the size of more than one inner region is determined. The method of claim 2 , wherein each interior region is color corrected differently. 4 . The method according to claim 3 , wherein an inner region located at a center with respect to the correction region is color-corrected more finely than an inner region relatively farther from the center of the correction region.

5. The method of claim 1, wherein the size of the eye or the orbit of the eye is measured as a radius or a distance from one end to an opposite end.

6. One or more hardware-based non-transitory computer-readable memory devices storing instructions that, when executed by one or more processors disposed in a computing device, cause the computing device to: receiving coordinates for a pupil of a user while the computing device is in use, wherein the computing device comprises a near-eye waveguide-based display configured to render a virtual image, the near-eye waveguide-based display comprising a waveguide, at least one diffractive optical element for in-coupling light for the virtual image into the waveguide, and at least one diffractive optical element for out-coupling the virtual image light from the waveguide to a pupil of the user, each diffractive optical element and the waveguide inherently providing color non-uniformity in the rendered virtual image; identifying a gaze location on the near-eye waveguide-based display based on the received coordinates for the pupil; evaluating a foveal gaze region based on at least one of the gaze location or a current size of a pupil of the user while operating the computing device, the foveal gaze region being within an inner boundary of a peripheral visual field of the user; as well as A color correction mask is programmatically applied only to the foveal gaze region, wherein the applied mask corrects color non-uniformity in the virtual image rendered in the foveal gaze region of the near-eye waveguide-based display.

7. One or more hardware-based non-transitory computer-readable memory devices according to claim 6, wherein the coordinates are derived from sensors that track eye movements and pupils of the user, and the coordinates are received at a processor within the computing device.

8. One or more hardware-based non-transitory computer-readable memory devices according to claim 6, wherein a separate color correction mask is applied in a peripheral area outside the foveal gaze area, and the rendered virtual image in the peripheral area is optimized to a lesser extent relative to the color correction mask applied to the foveal gaze area.

9. One or more hardware-based non-transitory computer-readable memory devices according to claim 6, wherein the size of the foveal gaze area shrinks and expands based on measurements of factors associated with the computing device and the user, the measurements being periodically obtained while the computing device is in use.

10. One or more hardware-based non-transitory computer-readable memory devices according to claim 9, wherein the factors that cause the size of the foveal gaze area to shrink include one or more of the following: latency exceeding a threshold when applying the color correction mask, reduced pupil size, reduced system resources, or displayed content.

11. One or more hardware-based non-transitory computer-readable memory devices according to claim 9, wherein the factors that cause the size expansion of the foveal gaze area include one or more of the following: eye rotation rate exceeding a threshold rate, expansion of pupil size, increased availability of system resources, and displayed content.

12. One or more hardware-based non-transitory computer-readable memory devices according to claim 6, wherein various operations in the computing device are performed in real time to facilitate real-time application of the color correction mask to the foveal gaze area, and the real-time operations include periodically tracking pupil position, pupil size, displayed content identification, and latency determination.

13. A head mounted display (HMD) device configured to reduce or eliminate color non-uniformity on a display suitable for rendering a virtual image and reduce resource consumption, comprising: a waveguide-based display comprising a waveguide, at least one diffractive optical element for in-coupling light for the virtual image into the waveguide, and at least one diffractive optical element for out-coupling the virtual image light from the waveguide to a pupil of a user, each diffractive optical element and the waveguide inherently providing color non-uniformity in a virtual image rendered on the waveguide-based display; one or more processors; as well as one or more hardware-based non-transitory memory devices storing computer-readable instructions that, when executed by the one or more processors, cause the HMD device to: Monitor the position and size of the user's eyes; using information derived from the eye monitoring to set a size of a correction region on the waveguide-based display; When rendering an image on the waveguide-based display, color correcting a portion of the waveguide-based display, wherein the portion is the set size of the correction region, and wherein the color-corrected portion is appropriate for at least one of: an entire eye, an eye socket of the eye, or a pupil of the eye; as well as Dynamically or in response to user input, switch between color correction modes, wherein the color correction mode includes color correction of the set size based on the user's pupil, or based on the user's entire eye or eye socket, or based on the pupil and the entire eye or eye socket simultaneously.

14. The HMD device of claim 13, wherein dynamically switching between the color correction modes is responsive to measured power state or pupil movement.

15. The HMD device according to claim 14, wherein: switching from pupil color correction to full eye or orbital color correction when the pupil movement exceeds a threshold movement rate; and When the power state reaches a threshold power level, switching is performed from pupil color correction to full eye or orbital color correction. The HMD device of claim 15 , wherein the pupil movement measured is relative to the HMD device.

17. The HMD device of claim 13, wherein the eye monitoring measures pupil size and pupil movement when the pupil of the eye is color corrected. 18 . The HMD device according to claim 13 , wherein when the whole eye or the eye socket of the eye is color corrected, the eye monitoring measures the size and position of the whole eye or the eye socket of the eye.

19. The HMD device of claim 18, wherein the one or more processors further cause the HMD device to measure the size of the user's entire eye or eye socket using a diameter or a radius.

20. The HMD device of claim 13, further comprising an adjustable sensor for monitoring and measuring the user's eyes, wherein: When measuring the size and position of the pupil, the adjustable sensor is positioned relatively closer to the user's eye; and When measuring the size and position of the entire eye or eye socket, the adjustable sensor is positioned relatively farther away from the user's eye.