Color reference for overlay images

Abstract

An adjusted overlay image is generated in response to at least one color reference object or color reference point. The overlay image is presented to an eye (202) of a user of a head mounted display (HMD), wherein the overlay image overlaps a real world scene. The optical system (200) includes a display (250), a light sensor (147), and a color reference object (133). The display presents a virtual image included in display light (251) to a first eye (202), while the other eye (202) can also view scene light (271) from an external environment of the optical system. All or a portion of the display can be transparent or translucent to allow scene light from the user's external environment to be incident on the eye (202), so that the user can view her external environment in addition to viewing the overlay image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 867,525, entitled “Color Reference for Overlaid Images,” filed on June 27, 2019, and U.S. Non-Provisional Application No. 16 / 710,153, entitled “Color-Reference for Overlaid Images,” filed on December 11, 2019. U.S. Provisional Application No. 62 / 867,525 and U.S. Non-Provisional Application No. 16 / 710,153 are expressly incorporated herein by reference in their entireties.

[0003] Background Information

[0004] In augmented reality (AR), mixed reality (MR), and / or hybrid reality, artificial reality content can include computer-generated content combined with real-world content. In these experiences, it may be desirable for the computer-generated content overlaid with the real-world content to be consistent with the lighting and color environment of the outside world.

[0005] Overview

[0006] According to the present invention, a device is provided, comprising: a display configured to generate an overlaid image for overlapping with a real-world scene; a light sensor configured to capture light measurements of scene light from an environment external to the device; and processing logic configured to: determine a color space adjustment value in response to the light measurements; and adjust an image color of at least a portion of the overlaid image in response to the color space adjustment value.

[0007] Optionally, the device further comprises a color reference object configured to direct scene light to the light sensor, wherein the light sensor is configured to measure spectral reflectance of the color reference object.

[0008] Optionally, determining the color space adjustment value in response to the light measurement comprises comparing a calibrated color value of the color reference object with the light measurement of the color reference object when the color reference object of the device is illuminated by scene light.

[0009] Optionally, the color reference object is sealed in a transparent protective barrier so that the color reference object is protected from the external environment while still receiving scene light.

[0010] Optionally, the light sensor comprises an image sensor oriented to capture an image of the scene as the light measurement.

[0011] Optionally, determining the color space adjustment value in response to the light measurement result includes: identifying a color reference point in the scene image; and determining a difference vector between a pixel value of the color reference point in the scene image and a calibrated color value of the color reference point as the color space adjustment value, wherein the image color of at least a portion of the overlapping image is adjusted based on the difference vector.

[0012] Optionally, identifying the color reference point comprises identifying a whitest object in the scene image, and wherein the calibrated color value is the white reference point.

[0013] Optionally, the processing logic is configured to receive light measurements from a light sensor.

[0014] Optionally, the device is a head mounted display (HMD).

[0015] According to the present invention, there is also provided a computer-implemented method, the method comprising: capturing light measurements with a light sensor of a head-mounted display (HMD), the light sensor configured to receive scene light from an environment external to the HMD; determining a color space adjustment value in response to the light measurements; and adjusting an image color of at least a portion of an overlay image generated by a display of the HMD and overlapping a real-world scene visible to a user of the HMD in response to the color space adjustment value.

[0016] Optionally, the HMD includes a color reference object configured to direct scene light to the light sensor, and wherein determining the color space adjustment value in response to the light measurement includes comparing a calibrated color value of the color reference object to the light measurement of the color reference object when the color reference object of the HMD is illuminated by the scene light.

[0017] Optionally, the light sensor includes an image sensor, and the image sensor is oriented to capture a scene image, the scene image including a color reference object contained in the HMD, the light measurement result includes the scene image, and wherein determining the color space adjustment value in response to the light measurement result includes: identifying a color reference point in the scene image; and determining a difference vector between a pixel value of the color reference point in the scene image and a calibrated color value of the color reference point as the color space adjustment value, wherein image color of at least a portion of the overlapping images is adjusted based on the difference vector.

[0018] Optionally, adjusting the image color comprises adjusting sub-pixel values of the overlapping image based on the color space adjustment value.

[0019] Optionally, the display comprises a first light source, a second light source, and a third light source, and wherein adjusting the image color comprises adjusting an intensity of at least one of the first light source, the second light source, or the third light source based on the color space adjustment value.

[0020] According to the present invention, a computer-implemented method is also provided, the method comprising: receiving a scene image of a real-world scene; identifying at least one color reference point in the scene image of the real-world scene; generating a color space adjustment map for an overlapping image using the at least one color reference point; and applying the color space adjustment map to the overlapping image to generate an adjusted overlapping image by adjusting the image color of at least a portion of the overlapping image, wherein the adjusted overlapping image will overlap the real-world scene visible to a user.

[0021] Optionally, the computer-implemented method further comprises: identifying an overlapping region of the real-world scene that the overlapping images will occupy; and selecting a closest color reference point from at least one color reference point that is closest to the overlapping region, wherein the closest color reference point is used to generate the color space adjustment map.

[0022] Optionally, identifying at least one color reference point includes identifying a first color reference point and a second color reference point, and wherein generating a color space adjustment map for the overlapping image includes: determining a first difference vector between a first pixel value of the first color reference point and a first calibrated color value of the first color reference point; and, identifying a second difference vector between a second pixel value of the second color reference point and a second calibrated color value of the second color reference point, wherein generating the color space adjustment map includes generating a blending vector of the color space adjustment map between the first color reference point and the second color reference point based on the first difference vector and the second difference vector, the blending vector gradually transitioning the image color of a transition portion of the adjusted overlapping image to overlap between the first color reference point and the second color reference point.

[0023] Optionally, identifying at least one color reference point comprises matching an object in the real world scene with known color reference objects included in an array of previous photographs accessible to the user.

[0024] Optionally, the color reference point includes the sclera.

[0025] Optionally, the scene image is captured by a front-facing camera of the head-mounted display. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0028] Figure 1 An example head-mounted display (HMD) including a color reference object and a light sensor is shown according to an embodiment of the present disclosure.

[0029] Figure 2 A top view of an optical system for presenting an overlay image to the eyes of an HMD user, wherein the overlay image overlaps a real-world scene, is shown in accordance with an embodiment of the present disclosure.

[0030] Figure 3 An overlay image of a tiger overlaying a real-world scene is shown in accordance with an embodiment of the present disclosure.

[0031] Figure 4 An example head-mounted display (HMD) is shown that includes a front-facing camera oriented to capture one or more scene images of a real-world scene of the HMD, in accordance with an embodiment of the present disclosure.

[0032] Figure 5 A top view of an optical system for presenting an overlay image to the eyes of an HMD user, wherein the overlay image overlaps a real-world scene, is shown in accordance with an embodiment of the present disclosure.

[0033] Figure 6A and Figure 6B A color space chromaticity diagram and difference vectors according to an embodiment of the present disclosure are shown.

[0034] Figure 7A and Figure 7B Example scene images with different reference areas and example adjusted overlapping images with adjustment areas according to an embodiment of the present disclosure are shown.

[0035] Figure 8 An exemplary flowchart of a process of color adjustment on overlapping images according to an embodiment of the present disclosure is shown.

[0036] Figure 9 An example flow chart of a process for color-adjusting overlapping images relative to a color reference object according to an embodiment of the present disclosure is shown.

[0037] Detailed description

[0038] Embodiments for generating overlay images relative to a color reference are described herein. In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. However, those skilled in the relevant art will recognize that the techniques described herein can be implemented without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.

[0039] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0040] In augmented reality (AR), mixed reality (MR), and / or hybrid reality, a user may encounter different lighting conditions in a real-world scene. For example, a real-world scene may be illuminated by LEDs having different color temperatures, sunlight, cold cathode fluorescent lamps, or sodium lamps. These different illumination sources may illuminate the real-world scene with light having various color temperatures and hues. Adjusting the color of a computer-generated overlay image presented to the user overlying the real-world scene will make the overlay image more consistent and believable in the experience. Thus, when the real-world scene is illuminated with blue CCFL light, the pixel values of the overlay image may be adjusted by a difference vector between a white light reference point and the blue light illuminating the real-world scene. By adjusting the color of objects or animals in the overlay image to be more similar to their color when illuminated by light of a similar color illuminating the real-world scene, the user experience is improved. In combination with the attached Figures 1-9 These and other embodiments are described in further detail.

[0041] Figure 1 An example head-mounted display (HMD) 100 including a color reference object 133 and a light sensor 147 according to an embodiment of the present disclosure is shown. The HMD 100 includes a frame 114 coupled to arms 111A and 111B. Lenses 121A and 121B are mounted to the frame 114. Lenses 121 can be prescription lenses or non-prescription lenses matched to the specific wearer of the HMD. The illustrated HMD 100 is configured to be worn on or around the head of a user of the HMD.

[0042] exist Figure 1In the embodiment, each lens 121 includes a waveguide 150 to direct image light generated by the display 130 to a viewport area for viewing by a wearer of the HMD 100. The display 130 may include an LCD, an organic light-emitting diode (OLED) display, a micro-LED (micro-LED) display, a quantum dot display, a pico-projector, or a liquid crystal on silicon (LCOS) display for directing image light to the wearer of the HMD 100.

[0043] The frame 114 and arms 111 of the HMD 100 may comprise the support hardware for the HMD 100. The HMD 100 may comprise any of the following: processing logic, a wired and / or wireless data interface for transmitting and receiving data, a graphics processor, and one or more memories for storing data and computer-executable instructions. In one embodiment, the HMD 100 may be configured to receive wired power. In one embodiment, the HMD 100 may be configured to be powered by one or more batteries. In one embodiment, the HMD 100 may be configured to receive wired data, including video data, via a wired communication channel. In one embodiment, the HMD 100 may be configured to receive wireless data, including video data, via a wireless communication channel.

[0044] Lens 121 appears transparent to the user to facilitate augmented reality or mixed reality, wherein the user can view scene light from the external environment around them while also receiving image light directed to their eyes by waveguide 150. Thus, lens 121 can be considered (or include) an optical combiner. In some embodiments, when lenses 121 are optical combiners, they may introduce a certain amount of color distortion that modifies the real-world scene, and the compensation techniques described below can address the color distortion introduced by lens 121. In some embodiments, image light is directed only into one eye of the wearer of HMD 100. In embodiments, both displays 130A and 130B are included to direct image light into waveguides 150A and 150B, respectively. Displays 130A and 130B (in combination with waveguides 150A and 150B) can be configured to generate an overlay image for overlaying with the real-world scene.

[0045] Figure 1The example HMD 100 includes a color reference object 133 and a light sensor 147. The color reference object 133 is configured to receive scene light and direct the scene light to the light sensor 147 so that the light sensor 147 can capture light measurements of the scene light from the external environment of the HMD. In some embodiments, the light sensor 147 may include a complementary metal oxide semiconductor (CMOS) image sensor. The pixel values of the image captured by the image sensor can be used to determine the color of the scene light. The light sensor 147 may include a color sensor chip that includes three or more photodiodes disposed under red, green, and blue color filters to measure the color of the scene light. The fourth photodiode may not have a color filter disposed thereon. The color sensor chip can output digital or analog information that can be used to determine the color of the scene light.

[0046] Color reference object 133 can be similar to a "gray card" used in film (video) or photography to provide a middle gray reference point for calibration color values, or similar to a white card to provide a white reference point for calibration color values. Color reference object 133 can be another color with a calibration color value. Color reference object 133 can be sealed in a transparent protective barrier to protect it from the external environment.

[0047] When a color sensor captures light measurements of scene light, the light measurements can be analyzed to determine the color difference, if any, between the light measurements and calibrated color values obtained from light measurements taken in a known and controlled lighting environment. Thus, the color delta of the scene light can be measured by calculating the difference between the light measurements captured by the light sensor in an unknown light environment and the calibrated color values of a color reference object measured by the light sensor in a known light environment. For example, the known light environment can be a white light source with known color coordinates in a color space chromaticity diagram or other perceptually matching color space.

[0048] Figure 2A top view of an optical system 200 for presenting an overlay image to an eye 202 of an HMD user, wherein the overlay image overlaps a real-world scene, according to an embodiment of the present disclosure, is shown. Optical system 200 includes a display 250, a light sensor 147, and a color reference object 133. Display 250 presents a virtual image, included in display light 251, to eye 202, while eye 202 also views scene light 271 from the external environment of optical system 200. Therefore, the virtual images included in display light 251 are referred to as overlay images because they overlay real-world scene light 271. All or a portion of display 250 may be transparent or translucent to allow scene light 271 from the user's external environment to be incident on eye 202, allowing the user to view their external environment in addition to the overlay image.

[0049] In some embodiments, light sensor 147 captures light measurements of color reference object 133 illuminated by scene light 271. Color reference object 133 is configured to receive scene light from light sensor 147, allowing the light sensor to image the color reference object when illuminated by scene light 271. In the illustrated embodiment, light sensor 147 is communicatively coupled to processing logic 240 via communication channel X1. Processing logic 240 can initiate light measurement by sending an initiation signal to light sensor 147 via communication channel X1. Processing logic 240 can receive light measurement data 249 via communication channel X1. Light measurement data 249 can be an image comprising rows and columns of pixel values. In some embodiments, light measurement data 249 can include an analog or digital representation of chromaticity from a color sensor chip. For example, processing logic 240 can be included in arm 111 of an HMD.

[0050] Light measurement data 249 may be provided to an overlay image adjustment engine 245 of processing logic 240. Processing logic 240 and overlay image adjustment engine 245 are configured to receive overlay image 241 and, utilizing overlay image adjustment engine 245, generate an adjusted overlay image 242. Processing logic 240 may be configured to determine a color space adjustment value in response to light measurement data 249 and adjust the image color of at least a portion of received overlay image 241 in response to the color space adjustment value. In an embodiment, determining the color space adjustment value in response to the light measurement results includes comparing a calibrated color value of color reference object 133 with light measurement data 249 for color reference object 133 when color reference object 133 is illuminated by scene light 271. Processing logic 240 is communicatively coupled to display 250, and processing logic 240 may drive display 250 to display adjusted overlay image 242 in display light 251 based at least in part on light measurement data 249.

[0051] For example, the overlay image 241 may be received from onboard memory (not shown) included in the HMD 100. The image 241 may also be transmitted wirelessly to the HMD and received by a wireless interface (not shown) of the HMD.

[0052] Figure 3 An overlay image 379 of a tiger overlaying a real-world scene 300 is shown, in accordance with aspects of the present disclosure. Some real-world scenes viewed by a user of an HMD may have illumination that is generally uniform in color. For example, an outdoor scene may be illuminated by sunlight having a uniform color, or an indoor scene may be illuminated by a similar light source that emits the same color of light throughout the indoor scene. Some real-world scenes viewed by a user of an HMD may include illumination that is non-uniform in color or varies across different areas of the real-world scene. In real-world scene 300, object 307A is illuminated by sunlight from the sun 345, and object 307B is illuminated by light 349 from a second illumination source 347. Second illumination source 347 may be a CCFL that generates blue light 349 that illuminates object 307B, while object 307A may be illuminated by red light from the sun 345. Therefore, even though objects 307A and 307B are wearing the same clothing and have similar skin tones, they may be perceived by the user of the HMD as having slightly different colors.

[0053] Figure 3A first color reference point 333A and a second color reference point 333B are included. First color reference point 333A is the white panel of a soccer ball. Second color reference point is the white outlet cover. Of course, other objects and other colors can also be used as color reference points. Color reference point 333 can be identified in scene 300 by capturing a scene image of scene 300 using the HMD's image sensor and then identifying the whitest object by searching for interconnected groups of white (or nearly white) pixels and assuming that the object is actually white. If the whitest object in the image reflects red light (as captured by the pixel data in the image), it can be assumed that the white object (and nearby objects) are illuminated by red light (e.g., sunlight). In other embodiments of the present disclosure, color reference points are identified by matching a portion of the scene image of scene 300 (captured by the HMD's image sensor) with a known object. For example, outlet cover 333B has a shape and contrast profile that can be identified using an image processing matching algorithm. Some image processing matching algorithms may include matching objects or object shapes in scene 300 with multiple images of known objects. Image processing matching algorithms can also be used to identify the football's panel portion at color reference point 333A. Once an object is identified as a color reference point, the calibrated color value corresponding to the color reference point can be used as a baseline to determine the color of the light currently illuminating the identified object. For example, if outlet cover 333B has a calibrated color value of a specific white and the scene image pixels of outlet cover 333B have chromaticity values that are relatively close to bluer wavelengths in the color space diagram, it can be determined that color reference point 333B is illuminated by blue light. At a high level, if the system can determine or identify the true or intrinsic color of an object in the scene, the colors of other objects in the scene (or similarly illuminated portions of the scene) can be recovered and known. This concept can be referred to as an intrinsic property of an object, and identifying corresponding surfaces under different illuminants can be used to estimate the illuminant.

[0054] Figure 4 An example head-mounted display (HMD) 400 is shown that includes a front-facing camera 447 oriented to capture one or more scene images of a real-world scene of the HMD 400, in accordance with an embodiment of the present disclosure. Other elements of the HMD 400 may be similar to the HMD 100. The front-facing camera 447 is shown disposed in the middle of the HMD 400, but the camera 447 may be disposed on a side of the HMD 400. Furthermore, more than one camera may be used in the HMD 400 to capture scene images.

[0055] exist Figure 5, camera 447 is configured to image scene light 271. Thus, light measurement data 549 may include an image. Camera 447 may include a CMOS image sensor and a lens that focuses scene light 271 onto an imaging plane of the image sensor. Figure 5 A top view of an optical system 500 for presenting an overlay image to an eye 202 of an HMD user, where the overlay image overlaps a real-world scene, according to an embodiment of the present disclosure. The optical system 500 includes a display 250, a camera 447, and processing logic 540. The display 250 presents a virtual image, included in display light 251, to the eye 202, while the eye 202 also views scene light 271 from the external environment of the optical system 200. Therefore, the virtual images included in the display light 251 are referred to as overlay images because they overlay the real-world scene light 271. All or a portion of the display 250 can be transparent or translucent to allow scene light 271 from the user's external environment to be incident on the eye 202, allowing the user to view their external environment in addition to the overlay image.

[0056] The processing logic 540 can be configured to determine a color space adjustment value in response to the light measurement data 549 and adjust the image color of at least a portion of the received overlay image 241 in response to the color space adjustment value. The processing logic 540 is communicatively coupled to the display 250, and the processing logic 540 can drive the display 250 to present the adjusted overlay image 542 in the display light 251 based at least in part on the light measurement data 549.

[0057] In some embodiments, camera 447 captures an image of a scene of the real-world external environment of the HMD. In the illustrated embodiment, camera 447 is communicatively coupled to processing logic 540 via communication channel X1. Processing logic 540 can initiate image capture by sending an activation signal to camera 447 via communication channel X1. Processing logic 540 can receive light measurement data 549 (e.g., an image of the scene) via communication channel X1. Light measurement data 549 can be an image comprising rows and columns of pixel values. As shown, processing logic 540 can be included in arm 111 of the HMD.

[0058] The light measurement data 549 can be provided to a color reference identification engine 543 to identify color reference points for scene images included in the light measurement data 549. For example, the color reference identification engine 543 can identify a white outlet cap (e.g., Figure 3 333B in the figure as a color reference point, or the white panel of the football (e.g., 333A) as a color reference point.

[0059] Figure 6AA color space chromaticity diagram 600 is shown in accordance with an embodiment of the present disclosure. As will be familiar to those skilled in the art, the diagram 600 includes blue wavelengths at the bottom left of the diagram 600 to green wavelengths at the top and continuing to red wavelengths at the bottom right.

[0060] Calibrated color value 681 refers to the chromaticity coordinates within reference graph 600. The calibrated color value can be associated with a color reference point, such as the spectral reflectance of a white outlet cover at color reference point 333B under a known light source. When the color reference point is illuminated with blue light (e.g., from a CCFL light source), the scene image captured by camera 447 can associate the pixel value of the color reference point with value 683. The chromaticity of value 683 has shifted toward blue compared to calibrated color value 681. When the color reference point is illuminated with red light (e.g., from a warm LED source or sunlight), the scene image captured by camera 447 can associate the pixel value of the color reference point with value 682. The chromaticity of value 682 has shifted toward red compared to calibrated color value 681.

[0061] Figure 6B A difference vector 693 is shown representing a color shift from the calibrated color value 681 of the color reference point to the pixel value of the color reference point in a blue illumination environment (e.g., illuminated with blue CCFL light). Difference vector 691 represents a color shift from the calibrated color value 681 of the color reference point to the pixel value of the color reference point in a red illumination environment. Those skilled in the art will appreciate that, in addition to graph 600, color differences may also be calculated using a perceptually matched color space (e.g., CIECAM (International Committee on Illumination Color Appearance Model) or other suitable perceptually matched color space) according to aspects of the present disclosure.

[0062] Reference again Figure 5Overlay image adjustment engine 545 is configured to receive overlay image 241 and generate an adjusted overlay image 542. Overlay image adjustment engine 545 may generate a color space adjustment map to apply to overlay image 541 when generating adjusted overlay image 542, which adjusts the image color of at least a portion of overlay image 541. In some embodiments, overlay image adjustment engine 545 may determine these difference vectors and adjust the image color of overlay image 241 based on the determined difference vectors. For example, green pixel value 690 of overlay image 241 may be adjusted downward to pixel value 696 based on difference vector 693 to simulate illumination of the overlay image by blue-white light of the same color as that illuminating the color reference point. Similarly, green pixel value 690 of overlay image 241 may be adjusted downward to pixel value 697 based on difference vector 692 to simulate illumination of the overlay image by red-white light of the same color as that illuminating the color reference point, for example, when sunlight illuminates the color reference point. In one embodiment, generating the adjusted overlay image 542 includes correcting the white point of the overlay image 241 by chromatic adaptation according to a 3×3 matrix transformation to associate the white point from the overlay image 241 with a different white point to generate the adjusted overlay image 542. Thus, there are a variety of techniques that can be used for a chromatic adaptation process that varies spatially due to input from an external sensor, such as sensor 447.

[0063] Figure 7A An example scene image 700 with different reference regions according to an embodiment of the present disclosure is shown. Scene image 700 can approximate the field of view (FOV) of a user viewing a real-world scene through an HMD. In the illustrated example, scene image 700 is divided into different reference regions 701A-D. More than four regions are possible. There can be six, eight, sixteen, or any other number of regions. Figure 7B Shown with Figure 7A An example adjusted overlapping image corresponding to the reference area 701 in FIG. 7 and having the same number of adjustment areas 711 .

[0064] In one embodiment, a color reference point 733 is identified in each reference region 701 of the scene image 700. The overlapping regions of the overlapping image 770 can then be adjusted region by region based on the color reference points in the reference region 701. An overlapping region 779 of the real-world scene (e.g., the outline of a tiger) can be identified as the region that the overlapping image (e.g., 279) will occupy. The overlapping region 779 overlaps the adjustment regions 711A, 711C, and 711D corresponding to the reference regions 701A, 701C, and 701D, respectively. The portion of the overlapping region 779 in region 711A can therefore be adjusted based on the pixel value of the color reference point 733A in the reference region 701A. Similarly, the portion of the overlapping region 779 in region 711C can therefore be adjusted based on the pixel value of the color reference point 733C in the reference region 701C, and the portion of the overlapping region 779 in region 711D can therefore be adjusted based on the pixel value of the color reference point 733D in the reference region 701D. Figure 3 In the example scene 300, the tail of the overlapping tiger in region 711D may be adjusted to a bluer hue to account for CCFL illumination, and the front and head of the tiger in region 711C may be adjusted to a redder hue to account for sunlight illumination. Figure 6A and Figure 6B The difference vector described in is used to adjust the color of the tiger in the overlapping image according to the pixel values of the color reference points 733C and 733D (in the scene image). It is worth noting that according to some aspects of the present disclosure, the different regions 701 can change over time, so that a spatiotemporal chromatic adaptation technique is used.

[0065] In one embodiment, the color reference point 733 closest to the overlap region 779 is used to generate a color space adjustment map that is applied to the overlapped images for image color adjustment purposes. Figure 7A In FIG. 7 , color reference point 733A may be used to adjust the image color of the entire overlapping image.

[0066] Figure 8 An example flow chart of a process 800 for color adjustment of overlapping images according to aspects of the present disclosure is shown. The order in which some or all of the process blocks appear in process 800 should not be considered limiting. Rather, one of ordinary skill in the art having the benefit of this disclosure will understand that some of the process blocks may be performed in various orders not shown, or even in parallel.

[0067] In process block 805, light measurements are captured using a light sensor of the HMD. For example, the light sensor can be a color sensor chip or an image sensor based on a photodiode and a color filter. The light measurements can be an image of the scene or provide an analog or digital value of chromaticity.

[0068] In process block 810 , a color space adjustment value is determined in response to the light measurement results.

[0069] In process block 815, image color of at least a portion of the overlay image is adjusted in response to the color space adjustment value. The overlay image is generated by a display (e.g., display 250) of an HMD (e.g., HMD 100 or 400) and overlaps a real-world scene visible to the HMD user.

[0070] In some embodiments of process 800, the HMD includes a color reference object (e.g., color reference object 133) configured to direct scene light toward the light sensor, and determining the color space adjustment value in response to the light measurement includes comparing a calibrated color value of the color reference object to the light measurement of the color reference object when the color reference object of the HMD is illuminated by the scene light.

[0071] In some embodiments of process 800, the light sensor includes an image sensor, and the image sensor is oriented to capture a scene image that includes a color reference object included in the HMD. In other words, the color reference object (e.g., 133) may be included in a portion of the scene image. Color reference object 133 may be strategically positioned at the very edge of the field of view of a camera including the image sensor, such that the image captured by the image sensor includes both the color reference object and the real-world scene. Determining a color space adjustment value in response to the light measurement may include identifying a color reference point in the scene image and determining a difference vector between a pixel value of the color reference point in the scene image and a calibrated color value of the color reference point as the color space adjustment value. The image color of (at least a portion of) the overlay image is then adjusted based on the difference vector. In some embodiments of process 800, the light sensor is not an image sensor and may include only three photodiodes with three different color filters disposed thereon. The three color filters may be matched to a human color matching function (CMF) (e.g., a CMF of 10 degrees).

[0072] In some embodiments of process 800 , adjusting the image color includes adjusting sub-pixel values (eg, RGB sub-pixel values) of the overlay image based on the color space adjustment value.

[0073] In an embodiment of process 800, a display of the HMD includes a first light source, a second light source, and a third light source (e.g., an RGB LED or an RGB laser source), and adjusting the image color includes adjusting an intensity of at least one of the first light source, the second light source, or the third light source based on a color space adjustment value.

[0074] Figure 9An example flow chart of a process 900 for color-adjusting overlapping images relative to a color reference point according to aspects of the present disclosure is shown. The order in which some or all of the process blocks appear in process 900 should not be considered limiting. Rather, one of ordinary skill in the art having the benefit of this disclosure will appreciate that some of the process blocks may be performed in various orders not shown, or even in parallel. The process blocks of process 900 may be performed using processing logic included in an HMD, or may be performed on a computing system remote from the HMD but accessible via a network.

[0075] In process block 905 , a scene image of a real-world scene is received.

[0076] At process block 910, at least one color reference point in the scene image is identified. Example color reference points may include color reference points 333A, 333B, 733A, 733B, 733C, or 733D. Example color reference points may include a stop sign, a yield sign, or a road sign. For example, a color reference point may include a sign (e.g., a road sign) or a signage having a consistent color. Example color reference points may include jewelry, an exit door, a tabletop, a soccer ball, or the skin color of a user or other person in the scene image. Example color reference points may include clothing or shoes. In one embodiment, the color reference point includes the white of the human eye (sclera), and the color reference point is identified by matching a portion of the scene image with an existing database of eye images that includes the sclera. Sensors that capture real-world scenes may have to undergo a calibration process during production in order to be used in process 900. The sensor may include three color filters that match the human color matching function (CMF) (e.g., CMF 10 degrees).

[0077] In process block 915, a color space adjustment map is generated for the overlay image using the at least one color reference point. Generating the color space adjustment map may include generating a difference vector or other color adjustment techniques described in this disclosure.

[0078] In process block 920, the color space adjustment map is applied to the overlay image to adjust the image color of at least a portion of the overlay image. The adjusted overlay image will overlap with the real world scene visible to the user of the HMD.

[0079] Some embodiments of process 900 also include identifying an overlapping region of the real-world scene that the overlapping image will occupy (e.g., 779), and selecting a closest color reference point (e.g., 733A) from the plurality of identified color reference points. The closest color reference point (e.g., calculated in Euclidean coordinates with reference to a perceptually matched color space) is then used to generate a color space adjustment map. For example, the pixel values of the closest color reference point can be used to generate a difference vector to adjust the pixel values of the overlapping image.

[0080] In an embodiment of process 900, identifying at least one color reference point includes identifying a first color reference point (e.g., 733C) and a second color reference point (e.g., 733D). Then, generating a color space adjustment map for the overlapping image includes determining a first difference vector between a first pixel value of the first color reference point and a first calibrated color value of the first color reference point, and identifying a second difference vector between a second pixel value of the second color reference point and a second calibrated color value of the second color reference point. Then, generating the color space adjustment map includes generating a blending vector of the color space adjustment map between the first color reference point and the second color reference point based on the first difference vector and the second difference vector. The blending vector gradually transitions the image color of a transition portion of the adjusted overlapping image to overlap between the first color reference point and the second color reference point. Using Figure 7B As an example, blending vector 735 may blend the blue light adjustment determined by color reference point 733D into the red light adjustment determined by color reference point 733C. The blending function may be linear or nonlinear. For example, using blending vector 735 included in the color space adjustment map, the tiger's tail in the overlaid image may be red-compensated, with a gradual transition to blue-compensated light on the tiger's chest and head.

[0081] In one embodiment of process 900, identifying at least one color reference point includes matching a point in the real-world scene with known color reference points included in an array of previous photographs accessible to the HMD user. For example, an image of a family member may have a particular appearance (garment) imaged in a known lighting environment (e.g., the sun in a clear sky). Similarly, the sensor that captures the real-world scene may have to undergo a calibration process during production in order to be used in this embodiment of process 900.

[0082] Embodiments of the present invention may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality, hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated in combination with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or be used in other ways in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0083] The term "processing logic" (e.g., 240 or 540) in the present disclosure may include one or more processors, microprocessors, multi-core processors, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs) to perform the operations disclosed herein. In some embodiments, a memory (not shown) is integrated into the processing logic to store instructions and / or store data for performing the operations. Depending on the embodiment of the present disclosure, the processing logic may also include analog or digital circuits to perform the operations.

[0084] The "memory" or "memories" described in this disclosure may include one or more volatile or non-volatile memory structures. The "memory" or "memories" may be removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Example memory technologies may include RAM, ROM, EEPROM, flash memory, CD-ROM, digital versatile disks (DVDs), high-definition multimedia / data storage disks or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device.

[0085] The network may include any network or network system, such as, but not limited to: a peer-to-peer network; a local area network (LAN); a wide area network (WAN); a public network such as the Internet; a private network; a cellular network; a wireless network; a wired network; a combined wireless and wired network; and a satellite network.

[0086] The communication channel (eg, X1) may include or be routed through one or more wired or wireless communications utilizing IEEE 802.11 protocols, Bluetooth, SPI (Serial Peripheral Interface), I 2 C (interconnect integrated circuit), USB (universal serial port), CAN (controller area network), cellular data protocol (e.g., 3G, 4G, LTE, 5G), optical communication network, Internet service provider (ISP), peer-to-peer network, local area network (LAN), wide area network (WAN), public network (e.g., "Internet"), private network, satellite network, or other.

[0087] The computing device may include a desktop computer, a laptop computer, a tablet computer, a phablet, a smartphone, a feature phone, a server computer, or other computing device. The server computer may be located remotely from a data center or may be stored locally.

[0088] The processes explained above are described from the perspective of computer software and hardware. The described techniques may constitute machine-executable instructions embodied in a tangible or non-transitory machine (e.g., computer) readable storage medium, which, when executed by a machine, causes the machine to perform the described operations. Additionally, these processes may be embodied in hardware, such as an application-specific integrated circuit ("ASIC") or other hardware.

[0089] Tangible, non-transitory machine-readable storage media include any mechanism that provides (i.e., stores) information in a form accessible by a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device having a set of one or more processors, etc.). For example, machine-readable storage media include recordable / non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).

[0090] The above description of the illustrated embodiments of the present invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Although specific embodiments and examples of the invention are described herein for illustrative purposes, those skilled in the relevant art will recognize that various modifications are possible within the scope of the invention.

[0091] These modifications may be made to the present invention in light of the above detailed description. The terms used in the appended claims should not be interpreted as limiting the invention to the specific embodiments disclosed in the specification. Rather, the scope of the present invention is to be determined entirely by the appended claims, which are to be interpreted in accordance with established principles of claim interpretation.

Claims

1. A device comprising: a display configured to generate an overlay image for overlaying with a real-world scene; a light sensor configured to capture light measurements of scene light from an environment external to the device; a color reference object configured to direct the scene light toward the light sensor, wherein the light sensor is configured to measure a spectral reflectance of the color reference object; and Processing logic, which is configured to: determining a color space adjustment value in response to the light measurement; and An image color of at least a portion of the overlapping images is adjusted in response to the color space adjustment value.

2. The device according to claim 1, wherein Determining the color space adjustment value in response to the light measurement result includes: comparing a calibrated color value of a color reference object of the device with the light measurement result of the color reference object when the color reference object is illuminated by the scene light; and / or wherein the color reference object is sealed in a transparent protective barrier so that the color reference object is protected from the influence of the external environment while still receiving the scene light.

3. The device according to claim 1, wherein The light sensor includes an image sensor oriented to capture an image of a scene as the light measurement; Wherein, determining the color space adjustment value in response to the light measurement result includes: identifying color reference points in the scene image; and determining a difference vector between a pixel value of the color reference point in the scene image and a calibrated color value of the color reference point as the color space adjustment value, wherein an image color of at least a portion of the overlapped image is adjusted based on the difference vector; Wherein identifying the color reference point comprises identifying a whitest object in the scene image, and wherein the calibrated color value is a white reference point.

4. The device according to claim 1, wherein The processing logic is configured to receive light measurements from the light sensor.

5. The apparatus according to claim 1, wherein The device is a head mounted display (HMD).

6. A computer-implemented method comprising: capturing light measurements with a light sensor of a head-mounted display (HMD), the light sensor configured to receive scene light from an environment external to the HMD; determining a color space adjustment value in response to the light measurement; as well as adjusting an image color of at least a portion of an overlay image generated by a display of the HMD and overlapping a real-world scene viewable by a user of the HMD in response to the color space adjustment value; Wherein the HMD comprises a color reference object configured to direct the scene light toward the light sensor, and wherein the light sensor is configured to measure a spectral reflectance of the color reference object.

7. The computer-implemented method of claim 6, wherein: Determining the color space adjustment value in response to the light measurement includes comparing a calibrated color value of a color reference object of the HMD with the light measurement of the color reference object when the color reference object is illuminated by the scene light.

8. The computer-implemented method of claim 6, wherein: the light sensor comprises an image sensor, and the image sensor is oriented to capture an image of a scene, the image of the scene including a color reference object contained in the HMD, the light measurements comprising the image of the scene, And wherein determining the color space adjustment value in response to the light measurement result comprises: identifying color reference points in the scene image; and A difference vector between a pixel value of the color reference point in the scene image and a calibrated color value of the color reference point is determined as the color space adjustment value, wherein an image color of at least a portion of the overlapping image is adjusted based on the difference vector.

9. The computer-implemented method of claim 6, wherein: Adjusting the image color comprises adjusting sub-pixel values of the overlaid image based on the color space adjustment value, and / or wherein the display comprises a first light source, a second light source, and a third light source, and wherein adjusting the image color comprises adjusting an intensity of at least one of the first light source, the second light source, or the third light source based on the color space adjustment value.

10. A computer-implemented method comprising: receiving a scene image of a real-world scene from a light sensor; identifying at least one color reference point in a scene image of the real-world scene; generating a color space adjustment map for the overlapping images using the at least one color reference point; and applying the color space adjustment map to the overlay image to generate an adjusted overlay image by adjusting an image color of at least a portion of the overlay image, wherein the adjusted overlay image is to overlap with a real-world scene viewable by a user; Wherein the method further comprises applying a color reference object, the color reference object being configured to direct scene light toward the light sensor, and wherein the light sensor is configured to measure the spectral reflectance of the color reference object.

11. The computer-implemented method of claim 10 , further comprising: identifying an overlapping region in the real-world scene that the overlapping images will occupy; as well as A closest color reference point closest to the overlapping area is selected from the at least one color reference point, wherein the closest color reference point is used to generate the color space adjustment map.

12. The computer-implemented method of claim 10, wherein: Identifying the at least one color reference point includes identifying a first color reference point and a second color reference point, and wherein generating the color space adjustment map for the overlaid image includes: determining a first difference vector between a first pixel value of the first color reference point and a first calibrated color value of the first color reference point; and identifying a second difference vector between a second pixel value of the second color reference point and a second calibrated color value of the second color reference point, Wherein, generating the color space adjustment map includes generating a mixing vector of the color space adjustment map between the first color reference point and the second color reference point based on the first difference vector and the second difference vector, and the mixing vector gradually transitions the image color of the transition part of the adjusted overlapping image to overlap between the first color reference point and the second color reference point.

13. The computer-implemented method of claim 10, wherein: Identifying the at least one color reference point comprises matching an object in the real-world scene with a known color reference object included in an array of previous photographs accessible to the user; wherein the color reference point can include a sclera; and / or The scene image is captured by a front camera of a head-mounted display.

Images