A method, apparatus and storage medium for color correction

Abstract

According to some implementations, a method is performed at an electronic device with one or more processors, non-transitory memory, and a see-through display. The method includes determining a plurality of light superposition characteristic values associated with ambient light from a physical environment. The plurality of light superposition characteristic values quantify the ambient light. The method includes modifying image data based on a function of the plurality of light superposition characteristic values and a reference perceptual color gamut to generate modified image data. The method includes transforming the modified image data into display data based on a function of a portion of the plurality of light superposition characteristic values and a reference physical color gamut associated with the see-through display. The method includes displaying the display data on the see-through display.

Description

Technical Field

[0001] The present disclosure relates to color correction, and in particular, to performing color correction based on light characteristics associated with a physical environment. Background Art

[0002] In some augmented reality (AR) environments, computer-generated content is added to light from the physical environment so that both the computer-generated content and a representation of the physical environment can be displayed. A user can experience AR using an electronic device that includes a see-through display, which in turn allows light from the physical environment to pass to the user's eyes.

[0003] However, in some cases, light from the physical environment can have illuminance or chromaticity that interferes with computer-generated content in a way that degrades the AR experience. For example, light from the physical environment can cause the displayed computer-generated content to have distorted perceived contrast levels or incorrect color distribution. However, previously available color correction methods do not effectively account for light from the physical environment. Summary of the Invention

[0004] According to some implementations, a method is performed at an electronic device having one or more processors, non-transitory memory, and a see-through display. The method includes determining a plurality of light superposition characteristic values ​​associated with ambient light from a physical environment. The plurality of light superposition characteristic values ​​quantifies the ambient light. The method includes modifying image data based on a function of the plurality of light superposition characteristic values ​​and a reference perceptual color gamut to generate modified image data. The method includes transforming the modified image data into display data based on a function of a portion of the plurality of light superposition characteristic values ​​and a reference physical color gamut associated with the see-through display. The method includes displaying the display data on the see-through display.

[0005] According to some embodiments, an electronic device includes one or more processors, non-volatile memory, and a see-through display. One or more programs are stored in the non-volatile memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the performance of the operations of any of the methods described herein. According to some embodiments, a non-volatile computer-readable storage medium has instructions stored therein that, when executed by one or more processors of an electronic device, cause the device to perform or cause the performance of the operations of any of the methods described herein. According to some embodiments, an electronic device includes means for performing or causing the performance of the operations of any of the methods described herein. According to some embodiments, an information processing device for use in an electronic device includes means for performing or causing the performance of the operations of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] For a better understanding of the various described implementations, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0007] Figure 1 is a block diagram of an example of a portable multifunction device according to some implementations.

[0008] Figure 2 is an example of a block diagram of a color correction pipeline according to some specific implementations.

[0009] Figure 3 is another example of a block diagram of a color correction pipeline according to some specific implementations.

[0010] Figure 4 is an example of a block diagram of a semantic-based color correction pipeline according to some specific implementations.

[0011] Figure 5 are examples of graphical representations of color correction according to some specific implementations.

[0012] Figure 6 is an example of a flow chart of a method for modifying image data based on light superposition characteristic values ​​and a reference color gamut according to some implementations.

[0013] Figure 7 is an example of a flow chart of a method for selectively modifying a portion of image data based on a light superposition characteristic value and a reference color gamut, according to some implementations. Summary of the Invention

[0015] A user can experience augmented reality (AR) via an electronic device (e.g., a tablet or smartphone) that includes a see-through display, which in turn allows light from the physical environment to pass to the user's eyes. For example, a see-through display projects computer-generated content that is reflected from the see-through display into the user's eyes. As another example, a see-through display projects computer-generated content directly onto the user's retina, with both light from the physical environment and the projected light from the computer-generated content reaching the retina simultaneously. However, electronic devices cannot effectively perform color correction because they do not account for light from the physical environment. For example, electronic devices do not account for the intensity of light (e.g., illuminance), which can vary over time. The illuminance of light from the physical environment may limit the contrast level between the physical environment and the displayed computer-generated content. As another example, light from the physical environment may have a chromaticity that interferes with the computer-generated content in a way that degrades the AR experience. The chromaticity of the light (such as the presence of predominantly one color) can provide a dominant color tone that is difficult to mask. The dominant color tone associated with light from the physical environment may interfere with the color characteristics of the displayed computer-generated content. Furthermore, certain color correction methods used in pass-through video display systems, such as backlight tinting, are not applicable to electronic devices with see-through displays. Additionally, applying previously available tone mapping is not efficient because it does not take into account luminance and chromaticity characteristics associated with light from the physical environment.

[0016] In contrast, various embodiments disclosed herein provide a color correction pipeline for modifying image data based on light overlay characteristics, a reference perceptual color gamut, and a reference physical color gamut. The reference physical color gamut is associated with a see-through display on which the display data is displayed. Modifying the image data based on characteristics of ambient light enables the see-through display to display computer-generated images that are less adversely affected by ambient light than with other color correction systems. Consequently, the user experience (e.g., an augmented reality experience) is enhanced compared to other color correction systems.

[0017] To this end, in some implementations, an electronic device with a see-through display determines a light overlay characteristic value associated with ambient light from a physical environment. In some implementations, the electronic device includes an environmental sensor (e.g., an ambient light sensor and / or an image sensor) and utilizes environmental data from the environmental sensor to determine the light overlay characteristic value. In some implementations, the light overlay characteristic value includes a combination of an illuminance value and a chromaticity value (e.g., chroma, hue, saturation) associated with the ambient light.

[0018] In addition, the electronic device modifies the image data to generate modified image data based on the light superposition characteristics and the reference perceptual color gamut. The reference perceptual color gamut can be a function of a reference physical color gamut associated with the see-through display. For example, the reference physical color gamut indicates the range of colors that can be displayed by the see-through display, and the reference perceptual color gamut indicates a subset of the range of colors perceptible to a user. In some embodiments, the reference physical color gamut is characterized by ideal or near-ideal ambient conditions, such as a black or near-black physical environment, where only a nominal amount of ambient light enters the see-through display. Thus, the electronic device accounts for variations in the characteristics of the ambient light entering the see-through display, compared to other systems that perform uniform spatial mapping. As a result, the modified image data has enhanced color and luminance characteristics, such as having improved contrast reproduction or improved chromaticity reproduction.

[0019] The electronic device transforms the modified image data into display data based on a function of a portion of the light superposition characteristic value and the reference physical color gamut. Thus, compared to other systems that perform three-dimensional color gamut mapping (e.g., RGB to RGB mapping), the electronic device can perform at least four-dimensional color gamut mapping because it also considers colorimetric characteristics (e.g., three-dimensional), illuminance values ​​associated with ambient light (one-dimensional), and / or AR display colors (e.g., three-dimensional). DETAILED DESCRIPTION

[0020] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. Numerous specific details are provided in the following detailed description to provide a thorough understanding of the various described implementations. However, it will be apparent to one of ordinary skill in the art that the various described implementations can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure various aspects of the implementations.

[0021] It will also be understood that, although in some cases, the terms "first," "second," etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first contact can be named a second contact, and similarly, a second contact can be named a first contact without departing from the scope of the various described embodiments. A first contact and a second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0022] The terms used in the description of various embodiments described herein are for the purpose of describing specific embodiments only and are not intended to be limiting. As used in the description of various embodiments described and in the appended claims, the singular forms "a", "an" and "the" are intended to also include the plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "includes", "including", "comprises" and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their groupings.

[0023] As used herein, the term "if" is optionally interpreted to mean "when" or "at" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining" or "in response to determining" or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]," depending on the context.

[0024] Various examples of electronic systems and techniques for using such systems in connection with various computer-generated reality techniques are described.

[0025] A physical environment refers to the physical world that a person can interact with and / or sense without using an electronic device. A physical environment can include physical features, such as physical objects or physical surfaces. For example, a physical environment can include a physical city, which includes physical buildings, physical streets, physical trees, and physical people. People can interact with and / or sense the physical environment directly, for example, through touch, vision, taste, hearing, and smell. On the other hand, an extended reality (XR) environment refers to a fully or partially simulated environment that a person can interact with and / or sense using an electronic device. For example, an XR environment can include virtual reality (VR) content, augmented reality (AR) content, mixed reality (MR) content, and the like. Using an XR system, a portion of a person's body movement or a representation thereof can be tracked. In response, one or more features of virtual objects simulated in the XR environment can be adjusted so that they adhere to one or more laws of physics. For example, an XR system can detect a user's head movement and, in response, adjust the graphical and auditory content presented to the user in a manner similar to how views and sounds would change in a physical environment. In another example, an XR system may detect movement of an electronic device (e.g., a laptop, mobile phone, tablet, etc.) presenting an XR environment and, in response, adjust the graphical and auditory content presented to the user in a manner similar to how views and sounds would change in a physical environment. In some cases, the XR system may adjust one or more features of the graphical content in the XR environment in response to an indication of physical movement (e.g., a voice command).

[0026] Various electronic systems enable a person to interact with and / or sense an XR environment. For example, projection-based systems, head-mounted systems, heads-up displays (HUDs), windows with integrated displays, vehicle windshields with integrated displays, displays designed to be placed on a user's eyes (e.g., similar to contact lenses), speaker arrays, headphones / earpieces, input systems (e.g., wearable or handheld controllers with or without haptic feedback), tablet computers, smartphones, and desktop / laptop computers may be used. One head-mounted system may include an integrated opaque display and one or more speakers. In other examples, a head-mounted system may accept an external device with an opaque display (e.g., a smartphone). A head-mounted system may include one or more image sensors and / or one or more microphones to capture images or video and / or audio of the physical environment. In other examples, a head-mounted system may include a transparent or translucent display. The medium through which light representing the image is directed may be included within the transparent or translucent display. The display may utilize OLEDs, LEDs, uLEDs, digital light projection, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The medium can be a holographic medium, an optical combiner, an optical waveguide, an optical reflector, or a combination thereof. In some examples, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can use retinal projection technology to project graphic images onto the user's retina. Projection systems can also be configured to project virtual objects into a physical environment, such as on a physical surface or as a hologram.

[0027] Figure 1 1 is a block diagram of an example of a portable multifunction device 100 (sometimes referred to herein as "electronic device 100" for brevity) according to some implementations. Electronic device 100 includes memory 102 (the memory optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an input / output (I / O) subsystem 106, an inertial measurement unit (IMU) 130, an image sensor 143 (e.g., a camera), a depth sensor 150, an eye tracking sensor 164, an ambient light sensor 190, and other input or control devices 116. In some implementations, electronic device 100 corresponds to one of a mobile phone, a tablet computer, a laptop computer, a wearable computing device, a head-mounted device (HMD), a head-mounted housing (e.g., to which electronic device 100 slides or is otherwise attached), and the like. In some implementations, the head-mounted housing is shaped to form a receiver for receiving electronic device 100 having a display.

[0028] In some implementations, peripherals interface 118, one or more CPUs 120, and memory controller 122 are, optionally, implemented on a single chip, such as chip 103. In some other implementations, they are, optionally, implemented on separate chips.

[0029] The I / O subsystem 106 couples the input / output peripherals and other input or control devices 116 on the electronic device 100 to the peripheral device interface 118. The I / O subsystem 106 optionally includes an image sensor controller 158, an eye tracking controller 162, and one or more input controllers 160 for other input or control devices, as well as a privacy subsystem 170. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative implementations, the one or more input controllers 160 are optionally coupled to (or not coupled to) any of the following: a keyboard, an infrared port, a universal serial bus (USB) port, a stylus, and / or a pointer device such as a mouse. The one or more buttons optionally include an increase / decrease button for volume control of a speaker and / or an audio sensor. The one or more buttons optionally include a push button. In some implementations, other input or control devices 116 include a positioning system (eg, GPS) that obtains information about the location and / or orientation of the electronic device 100 relative to the physical environment.

[0030] The I / O subsystem 106 optionally includes a speaker and an audio sensor that provide an audio interface between the user and the electronic device 100. The audio circuit receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker. The speaker converts the electrical signals into sound waves audible to humans. The audio circuit also receives electrical signals converted from sound waves by an audio sensor (e.g., a microphone). The audio circuit converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or RF circuit by the peripheral device interface 118. In some specific implementations, the audio circuit also includes a headset jack. The headset jack provides an interface between the audio circuit and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single-ear headset or a binaural headset) and input (e.g., a microphone).

[0031] The I / O subsystem 106 optionally includes a touch-sensitive display system that provides an input interface and an output interface between the electronic device 100 and the user. The display controller can receive electrical signals from the touch-sensitive display system and / or send electrical signals to the touch-sensitive display system. The touch-sensitive display system displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some specific implementations, some or all of the visual outputs correspond to user interface objects. As used herein, the term "indicator" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to input directed to the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0032] The touch-sensitive display system has a touch-sensitive surface, sensor, or sensor group that accepts input from the user based on tactile and / or haptic contact. The touch-sensitive display system and display controller (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system. In an exemplary implementation, the point of contact between the touch-sensitive display system and the user corresponds to the user's finger or stylus.

[0033] The touch-sensitive display system optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other specific implementations. The touch-sensitive display system and display controller optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system to detect contact and any movement or interruption thereof.

[0034] The user optionally uses any suitable object or appendage, such as a stylus, finger, etc., to contact the touch-sensitive display system. In some implementations, the user interface is designed to work with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some implementations, the electronic device 100 converts the rough finger-based input into a precise pointer / cursor position or command for performing the action desired by the user.

[0035] The I / O subsystem 106 includes an inertial measurement unit (IMU) 130, which may include an accelerometer, a gyroscope, and / or a magnetometer to measure various forces, angular rates, and / or magnetic field information relative to the electronic device 100. Therefore, according to various specific implementations, the IMU 130 detects one or more position change inputs of the electronic device 100, such as the electronic device 100 being shaken, rotated, moved in a specific direction, etc. The IMU 130 may include an accelerometer, a gyroscope, and / or a magnetometer to measure various forces, angular rates, and / or magnetic field information relative to the electronic device 100. Therefore, according to various specific implementations, the IMU 130 detects one or more position change inputs of the electronic device 100, such as the electronic device 100 being shaken, rotated, moved in a specific direction, etc.

[0036] The image sensor 143 captures still images and / or video. In some implementations, the optical sensor 143 is located on the back of the electronic device 100, opposite the touch screen on the front of the electronic device 100, so that the touch screen can be used as a viewfinder for still image and / or video image acquisition. In some implementations, another image sensor 143 is located on the front of the electronic device 100, so that an image of the user is acquired (e.g., for selfies, for conducting a video conference while the user views other video conference participants on the touch screen, etc.). In some implementations, the image sensor 143 corresponds to one or more cameras. In some implementations, the image sensor 143 includes one or more depth sensors. In some implementations, the image sensor 143 includes a monochrome or color camera. In some implementations, the image sensor 143 includes an RGB depth (RGB-D) sensor.

[0037] I / O subsystem 106 optionally includes a contact force sensor that detects the intensity of contact on electronic device 100 (e.g., touch input on a touch-sensitive surface of electronic device 100). The contact force sensor can be coupled to an intensity sensor controller in I / O subsystem 106. The contact force sensor optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact force sensor receives contact intensity information (e.g., pressure information or a surrogate for pressure information) from the physical environment. In some implementations, at least one contact force sensor is juxtaposed with or adjacent to the touch-sensitive surface of electronic device 100. In some implementations, at least one contact force sensor is located on the back of electronic device 100.

[0038] In some implementations, the depth sensor 150 is configured to obtain depth data, such as depth information representing objects within the obtained input image. For example, the depth sensor 150 corresponds to one of a structured light device, a time-of-flight device, and the like.

[0039] Eye tracking sensor 164 detects eye gaze of a user of electronic device 100 and generates eye tracking data indicative of the user's eye gaze. In various implementations, the eye tracking data includes data indicative of a user's fixation point (e.g., gaze point) on a display panel (such as a display panel within an electronic device).

[0040] Ambient light sensor (ALS) 190 detects ambient light from the physical environment. In some implementations, ambient light sensor 190 is a color light sensor. In some implementations, ambient light sensor 190 is a two-dimensional (2D) or three-dimensional (3D) light sensor.

[0041] In various implementations, electronic device 100 includes a privacy subsystem 170 that includes one or more privacy setting filters associated with user information, such as user information included in eye gaze data and / or body position data associated with the user. In some implementations, privacy subsystem 170 selectively prevents and / or restricts electronic device 100 or portions thereof from acquiring and / or transmitting user information. To do so, privacy subsystem 170 receives user preferences and / or selections from a user in response to prompting the user for such preferences and / or selections. In some implementations, privacy subsystem 170 prevents electronic device 100 from acquiring and / or transmitting user information unless and until privacy subsystem 170 obtains informed consent from the user. In some implementations, privacy subsystem 170 anonymizes (e.g., scrambles or obfuscates) certain types of user information. For example, privacy subsystem 170 receives user input specifying which types of user information privacy subsystem 170 anonymizes. As another example, privacy subsystem 170 independently of user specification (e.g., automatically) anonymizes certain types of user information that may include sensitive and / or identifying information.

[0042] Figure 2 is an example of a block diagram of a color correction pipeline 200 according to some implementations. In various implementations, the color correction pipeline 200 or a portion thereof is integrated into an electronic device, such as a Figure 1 In some implementations, the color correction pipeline 200 is integrated into a mobile device such as a smartphone, tablet computer, laptop computer, wearable device, etc.

[0043] In some implementations, color correction pipeline 200 is integrated into an electronic device that includes a see-through display 270. See-through display 270 operates as an additional display by adding computer-generated content (e.g., extended reality (XR) content) from the physical environment to ambient light 202.

[0044] In some implementations, the see-through display 270 corresponds to an additional display that enables optical see-through of a physical environment, such as an optical head-mounted display (HMD). For example, as opposed to using pure synthesis of video streams, the additional display can reflect projected images from the display while enabling the user to see through the display. In some implementations, the see-through display 270 displays at least a nominal amount of light from the physical environment. In some implementations, the see-through display 270 includes a photochromic lens or an electrochromic layer.

[0045] The color correction pipeline 200 includes a sensor subsystem 206 to sense ambient light 202 and output corresponding sensor data. In some implementations, the sensor subsystem 206 includes a combination of environmental sensors, such as an ambient light sensor (ALS) (e.g., a two-dimensional (2D) sensor), an image sensor, and / or an inertial measurement unit (IMU). For example, in some implementations, the sensor subsystem 206 includes a monochrome or color camera with a depth sensor (RGB-D) and determines the camera pose for the viewpoint projection based on data from the RGB-D. As another example, in some implementations, the sensor subsystem 206 captures a lower resolution scene image, such as via a dedicated low-resolution image sensor or a dedicated high-resolution image sensor. In some implementations, the sensor subsystem 206 is implemented as a hardened IP block. In some implementations, the sensor subsystem 206 is implemented using software and hardware accelerators.

[0046] In some implementations, the color correction pipeline 200 includes a light superposition feature value generator 208 to determine (e.g., generate) a plurality of light superposition feature values ​​based on corresponding sensor data. The plurality of light superposition feature values ​​are associated with ambient light 202 from a physical environment. The plurality of light superposition feature values ​​quantify the ambient light 202. For example, in some implementations, the plurality of light superposition feature values ​​include a combination of luminance values ​​(e.g., brightness) and chromaticity values ​​(e.g., saturation, hue, and chroma) that characterize the ambient light 202.

[0047] The color correction pipeline 200 includes an image data modifier 240 that modifies the image data to generate modified image data based on a function of a plurality of light superposition feature values ​​and a reference perceptual color gamut. In some implementations, the image data is stored in an image data database 250. In some implementations, the reference perceptual color gamut (e.g., stored in the reference perceptual color gamut database 230) indicates the range of colors perceptible to a user. For example, the reference perceptual color gamut is affected by factors such as the user's eye accommodation state, size and contour sharpness, and position on the retina.

[0048] In some implementations, the modified image data satisfies a color contrast threshold relative to the plurality of light superposition feature values. As an example, the image data represents a white phantom, and the plurality of light superposition feature values ​​include a green chromaticity value associated with ambient light 202, such as when a see-through display of the electronic device is pointed at trees in a forest. Continuing with the aforementioned example, the image data modifier 240 modifies the white phantom so that, when displayed on the see-through display 270, the white phantom appears white and has substantially no green tint. Thus, the image data modifier 240 changes the color of the white phantom to counteract the green chromaticity associated with ambient light 202. In some implementations, the image data modifier 240 includes a tone mapper 245 that performs a tone mapping operation on the image data. For example, the tone mapper 245 applies the tone mapping operation to a face to achieve a substantially uniform skin tone.

[0049] In some implementations, the modified image data satisfies a luminance contrast threshold relative to a plurality of light superposition characteristic values. For example, the image data may represent a white ghost image, and the plurality of light superposition characteristic values ​​may include a relatively high luminance level, such as when the electronic device is pointed at the sun. Continuing with the aforementioned example, the image data modifier 240 modifies the white ghost image, such as by darkening or coloring a portion of the image data representing the white ghost image to offset the relatively high brightness of the sun.

[0050] In some implementations, the color correction pipeline 200 includes a first gamut mapper 220 that maps (e.g., transforms) a reference physical color gamut (e.g., stored in a reference physical color gamut database 210) to a reference perceptual color gamut. The reference physical color gamut is associated with a see-through display 270. For example, the reference physical color gamut indicates a range of illuminances or colors that can be displayed by the see-through display 270. As another example, when a nominal amount of ambient light 202 enters the see-through display 270, the reference physical color gamut characterizes the see-through display 270. In some implementations, the first gamut mapper 220 determines the reference perceptual color gamut based on a function of the reference physical color gamut and a color appearance model.

[0051] The color correction pipeline 200 includes a second gamut mapper 260. The second gamut mapper 260 receives modified image data from the image data modifier 240. The second gamut mapper 260 transforms the modified image data into display data based on a function of a portion of the plurality of light superposition characteristic values ​​and a reference physical color gamut. For example, in some implementations, the second gamut mapper 260 transforms the modified image data based on a function of a combination of color characteristics of the modified image data (e.g., RGB to RGB mapping) and chromaticity values ​​and luminance values ​​included in the plurality of light superposition characteristic values. Thus, in some implementations, the second gamut mapper 260 performs gamut mapping in four or more dimensions. The second gamut mapper 260 provides display data for display on the perspective display 270.

[0052] Figure 3 is another example of a block diagram of a color correction pipeline 300 according to some implementations. In various implementations, the color correction pipeline 300 or a portion thereof is integrated into an electronic device, such as a Figure 1 In some implementations, the color correction pipeline 300 is similar to Figure 2 The color correction pipeline 200 shown in FIG. 2 is adapted from the color correction pipeline 200 .

[0053] Color correction pipeline 300 includes an image data modifier 320 that modifies image data based on one or more modified portions of a plurality of light overlay feature values. To this end, color correction pipeline 300 includes a light overlay feature value identifier 302 that identifies corresponding portions of a plurality of light overlay feature values ​​across see-through display 270 based on corresponding portions of the image data. Light overlay feature value identifier 302 obtains the plurality of light overlay feature values ​​from light overlay feature value generator 208. For example, in some implementations, light overlay feature value identifier 302 identifies a first portion of the plurality of light overlay feature values ​​associated with a first region of see-through display 270. Continuing with the previous example, the first region of see-through display 270 corresponds to a display location of a first portion of image data 250 representing an object of interest, such as text or a person's face. In some implementations, color correction pipeline 300 includes depth sensor 150, and light overlay feature value identifier 302 uses depth sensor data from depth sensor 150 to identify corresponding portions of the plurality of light overlay feature values. For example, light overlay feature value identifier 302 identifies a second portion of the plurality of light overlay feature values ​​that is associated with the foreground of the physical environment as indicated by the depth sensor data.

[0054] Color correction pipeline 300 includes a light superposition feature value modifier 310 that obtains a portion of a plurality of light superposition feature values ​​from light superposition feature value identifier 302. Light superposition feature value modifier 310 modifies one or more of the corresponding portions of the plurality of light superposition feature values ​​based on a function of a predetermined display characteristic associated with the image data to generate one or more modified portions of the plurality of light superposition feature values; for example, the predetermined display characteristic includes a preferred combination of chrominance and luminance values ​​associated with the image data. In some implementations, light superposition feature value modifier 310 applies a uniform illumination function 312 to one or more of the corresponding portions of the plurality of light superposition feature values. For example, when a portion of the image data represents a face, the corresponding region will be rendered with a substantially uniform skin tone. In some implementations, light superposition feature value modifier 310 applies an illumination smoothing function 314 to one or more of the corresponding portions of the plurality of light superposition feature values. For example, illumination smoothing function 314 implements Gaussian smoothing, uniform moving average smoothing, or the like. The light superposition feature value modifier 310 provides one or more modified portions of the plurality of light superposition feature values ​​to the image data modifier 320 .

[0055] The image data modifier 320 modifies the image data based on a function of one or more modified portions of the plurality of light superposition characteristic values ​​and a reference perceptual color gamut, such as a reference Figure 2 The image data modifier 320 provides the modified image data to the second gamut mapper 260. The second gamut mapper 260 transforms the modified image data into display data for display on the perspective display 270, such as the reference image data modifier 240. Figure 2 described.

[0056] Figure 4 is an example of a block diagram of a semantic-based color correction pipeline 400 according to some implementations. In various implementations, the semantic-based color correction pipeline 400 or a portion thereof is integrated into an electronic device, such as a Figure 1 In some implementations, the semantic-based color correction pipeline 400 is similar to Figure 2 The color correction pipeline 200 shown in or Figure 3 The color correction pipeline 300 shown in FIG. 1 is adapted from the color correction pipeline 200 or the color correction pipeline 300 .

[0057] The semantic-based color correction pipeline 400 selectively modifies image data based on a function of corresponding semantic values. For example, in some implementations, the semantic-based color correction pipeline 400 modifies a first portion of the image data associated with a first semantic value that meets a criterion. For example, the first semantic value is associated with an object of interest (such as a person's "face" or a "painting"). As another example, the first semantic value corresponds to a specific object type, such as a living object (e.g., a person, an animal, a tree, etc.).

[0058] To this end, the semantic-based color correction pipeline 400 includes a semantic value generator 410 that obtains or generates a plurality of semantic values ​​associated with a plurality of portions of the image data. For example, in some implementations, the semantic value generator 410 obtains the plurality of semantic values ​​from another system (such as from the internet). As another example, in some implementations, the semantic value generator 410 generates the plurality of semantic values ​​by performing semantic segmentation on the image data. In some implementations, the semantic value generator 410 generates the plurality of semantic values ​​using a neural network integrated within the semantic-based color correction pipeline 400.

[0059] Furthermore, the semantic-based color correction pipeline 400 includes a semantic value identifier 420 that obtains a plurality of semantic values ​​from the semantic value generator 410. The semantic-based color correction pipeline 400 identifies a first semantic value that meets a criterion from the plurality of semantic values. The semantic value identifier 420 provides the first semantic value to the image data modifier 430.

[0060] The image data modifier 430 modifies the image data to generate modified image data, similar to the one described in reference Figure 2 Image data modifier 240 or Figure 3 . In some implementations, the image data modifier 240 modifies the image data based on a function of a predetermined display characteristic associated with a first portion of the image data associated with the first semantic value. For example, the predetermined display characteristic includes a combination of a chromaticity value and a luminance value. In some implementations, the image data modifier 430 applies a scene object modifier 432 to the first portion of the image data so as to emphasize the identified object of interest. For example, the scene object modifier 432 increases the color contrast or illuminance contrast between the first portion of the image data and a plurality of light overlay feature values. In some implementations, the image data modifier 430 applies a scene background modifier 434 to a portion of the image data outside of the first portion of the image data so as to de-emphasize the scene background. In some implementations, the image data modifier 430 performs both an object emphasis operation and a background de-emphasis operation on the image data.

[0061] The image data modifier 430 provides the modified image data to the second gamut mapper 260. The second gamut mapper 260 transforms the modified image data into display data for display on the see-through display 270, such as the reference image data. Figure 2 described.

[0062] Figure 5 is an example of a graphical representation 500 of color correction according to some specific implementations. The graphical representation 500 shows the relationship between illuminance 504 (on the x-axis) and brightness 502 (on the y-axis). The illuminance 504 is associated with the ambient light from the physical environment, such as the reference Figures 2 to 4 2. The ambient light 202 depicted in FIG. Luminance 502 is associated with a perceptual space associated with the user's perception of light. Graphical representation 500 corresponds to a logarithmic curve 506 illustrating a perceptual color space mapping between physical illuminance values ​​(x-axis) and perceived luminance values ​​(y-axis). In other words, graphical representation 500 represents color correction (e.g., achromatization) as a function of illuminance values ​​associated with the physical environment. However, one of ordinary skill in the art will appreciate that in some implementations, color correction may additionally or alternatively be a function of chromaticity. For example, as described above, color correction may be a function of chromaticity associated with the image data, as well as chromaticity values ​​included in a plurality of light superposition feature values ​​associated with the ambient light.

[0063] The x-axis ranges from zero to L Ref,Max The value of 508 corresponds to the illuminance range associated with the physical space before color correction is performed. For example, L Ref,Max 508 indicates a reference physical color gamut associated with the see-through display (such as Figures 2 to 4 Furthermore, the y-axis values ​​including the first range 510 correspond to the luminance range associated with the perceptual space before color correction is performed.

[0064] To restore the originally intended contrast relationship, various embodiments disclosed herein provide color correction 520. Color correction 520 is based on light superposition characteristics, a reference perceptual color gamut (e.g., Figures 2 to 4 ) and a reference physical color gamut (e.g., Figures 2 to 4 The display data is generated from the image data using the reference physical color gamut 210 in FIG. Figure 2 Color correction pipeline 200, Figure 3 Color correction pipeline in 300 or Figure 44. The color correction pipeline 400 of claim 1 is implemented by one of the semantic-based color correction pipelines 400 in

[0065] . In some implementations, color correction 520 includes performing contrast-restoring tone mapping on the image data based on determined light overlay feature values ​​associated with ambient light from the physical environment. To this end, color correction 520 restores motion loss between a reference perceptual color gamut and a reference physical color gamut. In some implementations, color correction 520 includes a 4+ dimensional color gamut transform. For example, color correction 520 includes a 7-dimensional color gamut mapping, including three dimensions associated with mapping the image data to a see-through display (e.g., RGB to RGB), three dimensions associated with chromaticity values ​​associated with the ambient light (e.g., two-dimensional values), and another dimension associated with luminance values ​​associated with the ambient light.

[0065] Based on the color correction 520, the brightness value is expanded from the first range 510 to the second range 512, as shown in FIG. Figure 5 Therefore, the illuminance values ​​on the x-axis range from zero to L Ref,Max 508 extended to extended range L b 514 to L t,Max 516. For example, L b 514 corresponds to the background illumination as indicated by the illumination value. Figure 2 , the light superposition characteristic value generator 208 determines an illuminance value associated with the ambient light 202 based on the ambient light data from the sensor subsystem 206. t,Max 516 corresponds to the maximum luminance associated with the physical color gamut produced by color correction 520. Therefore, color correction 520 increases the luminance from L Ref,Max 508 to L t,Max The maximum illumination range is 516.

[0066] Figure 6 is an example of a flow chart of a method 600 for modifying image data based on light superposition characteristic values ​​and a reference color gamut according to some implementations. In various implementations, the method 600 or a portion thereof is performed by an electronic device (e.g., a device) including a see-through display. Figure 1 In various implementations, the method 600 or a portion thereof is performed by a color correction pipeline, such as a color correction pipeline. Figure 2 Color correction pipeline 200, Figure 3 Color correction pipeline in 300 or Figure 4 In various implementations, the method 600 or a portion thereof is performed by a user comprising a see-through display (e.g., Figures 2 to 4The method 600 is performed by a head mounted device (HMD) including a see-through display 270 (e.g., a computer readable medium). In some implementations, the method 600 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the method 600 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0067] As represented by block 602, method 600 includes determining a plurality of light superposition feature values ​​associated with ambient light from a physical environment. The plurality of light superposition feature values ​​quantify the ambient light. For example, referring to Figure 2 , the light superposition feature value generator 208 determines a plurality of light superposition feature values ​​associated with the ambient light 202. To this end, in some specific implementations, the electronic device includes an environmental sensor that senses the ambient light and outputs corresponding sensor data. For example, referring to Figure 2 The light superposition feature value generator 208 determines a plurality of light superposition feature values ​​based on the ambient light data from the sensor subsystem 206. As an example, the environmental sensor includes one or more of an ambient light sensor, an image sensor, a visual inertial odometry (VIO), an inertial measurement unit (IMU), etc.

[0068] As represented by block 604, in some implementations, method 600 includes obtaining a reference physical color gamut associated with the see-through display. Figure 2 , the color correction pipeline 200 obtains a reference physical color gamut associated with the see-through display 270 and stores the reference physical color gamut in the reference physical color gamut database 210. In some implementations, the reference physical color gamut characterizes the see-through display when a nominal amount of ambient light from the physical environment enters the see-through display. In some implementations, the reference physical color gamut indicates a first set of colors that can be displayed on the see-through display. For example, the reference physical color gamut indicates a range of colors that can be displayed by the see-through display when a nominal amount of ambient light is present. As another example, the reference physical color gamut corresponds to one of an RGB color gamut or a P3 color gamut.

[0069] As represented by block 606, in some implementations, method 600 transforms the reference physical color gamut into a reference perceptual color gamut based on a function of a color appearance model. Figure 2, the first gamut mapper 220 transforms the reference physical color gamut into a reference perceptual color gamut. As an example, the reference perceptual color gamut indicates a second set of colors that may be different from the first set of colors that can be displayed on the see-through display. The color appearance model provides perceptual aspects of human color vision, such as the degree to which the viewing conditions of the color deviate from the corresponding physical measurements of the stimulus source. For example, the color appearance model is associated with the CIELAB color space. In some embodiments, the reference perceptual color gamut indicates a range of colors that are perceptible to a particular user, which is affected by factors such as the user's eye adaptation state, size and contour sharpness, position on the retina, and the like. In some embodiments, the reference physical color gamut is associated with illuminance, while the reference perceptual color gamut is associated with brightness, such as the reference Figure 5 described.

[0070] As represented by block 608, method 600 includes modifying the image data based on a function of the plurality of light superposition characteristic values ​​and the reference perceptual color gamut to generate modified image data. Figure 2 , the image data modifier 240 modifies the image data 250 based on the reference perceptual color gamut 230 and a function of the plurality of light overlay feature values ​​from the light overlay feature value generator 208. In some implementations, modifying the image data includes mapping the image data to the reference perceptual color gamut within a performance threshold based on the plurality of light overlay feature values. For example, the performance threshold is a function of the level of distortion (e.g., minimum distortion of color or contrast), color reproduction quality, user experience, etc. As represented by box 610, in some implementations, modifying the image data includes applying a tone mapping operation to the image data. For example, the tone mapping operation restores contrast that is lost in a physical environment with non-zero illuminance values. For example, with reference to Figure 2 , the tone mapper 245 performs a tone mapping operation. In some implementations, the tone mapping operation corresponds to a high dynamic range (HDR) tone mapping operation.

[0071] As represented by block 612, method 600 includes transforming the modified image data into display data based on a function of a portion of the plurality of light superposition characteristic values ​​and a reference physical color gamut. The display data is displayed on the see-through display 270. As represented by block 614, in some implementations, transforming the modified image data includes applying a color gamut mapping operation to the modified image data. For example, reference Figure 2, the second gamut mapper 260 transforms the modified image data into display data based on the reference physical gamut 210 and a portion of the plurality of light superposition feature values ​​from the light superposition feature value generator 208. As represented by block 616, in some implementations, the gamut mapping operation represents at least four dimensions. For example, the 6-dimensional gamut mapping operation is a function of the three dimensions associated with mapping the image data to a see-through display (e.g., RGB to RGB) and an additional three dimensions associated with chromaticity values ​​(of the plurality of light superposition feature values) associated with ambient light from the physical environment. As an example, with reference to Figure 2 , the second gamut mapper 260 obtains a portion of the plurality of light superposition feature values ​​from the plurality of light superposition feature values ​​from the light superposition feature value generator 208. Thus, compared to other systems that perform three-dimensional mapping from image data to display data, the color correction techniques disclosed herein can include more than three dimensions for gamut mapping.

[0072] As represented by block 618, method 600 includes displaying on a see-through display (such as on a Figure 2 The display data is displayed on the perspective display 270 in the image processing apparatus.

[0073] Figure 7 is an example of a flow chart of a method 700 for selectively modifying a portion of image data based on a light superposition characteristic value and a reference color gamut according to some implementations. In various implementations, the method 700 or a portion thereof is performed by an electronic device (e.g., a device) including a see-through display. Figure 1 In various implementations, the method 700 or a portion thereof is performed by a color correction pipeline, such as a color correction pipeline. Figure 2 Color correction pipeline 200, Figure 3 Color correction pipeline in 300 or Figure 4 In various implementations, the method 700 or a portion thereof is performed by a user comprising a see-through display (e.g., Figures 2 to 4 The method 700 is performed by a head mounted device (HMD) including a see-through display 270 (e.g., a computer readable medium). In some implementations, the method 700 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the method 700 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0074] As represented by block 702, method 700 includes determining a plurality of light superposition feature values ​​associated with ambient light from a physical environment. The plurality of light superposition feature values ​​quantify the ambient light. For example, referring to Figure 3 , the light superposition characteristic value generator 208 determines a plurality of light superposition characteristic values ​​associated with the ambient light 202 based on the sensor data from the sensor subsystem 206 .

[0075] As represented by block 704, in some implementations, method 700 includes identifying corresponding portions of the plurality of light superposition feature values ​​across the see-through display based on corresponding portions of the image data. Figure 3 , light overlay feature value identifier 302 identifies a corresponding portion of the plurality of light overlay feature values. In some implementations, light overlay feature value identifier 302 identifies the corresponding portion of the plurality of light overlay feature values ​​using depth data from depth sensor 150. For example, based on depth data associated with the physical environment, light overlay feature value identifier 302 identifies a portion of ambient light 202 associated with a foreground object within the physical environment (e.g., an object associated with a relatively low depth value).

[0076] As represented by block 706, in some implementations, method 700 includes modifying (e.g., preprocessing) one or more of the respective portions of the plurality of light superposition characteristic values ​​based on a function of a predetermined display characteristic associated with the image data to generate one or more modified portions of the plurality of light superposition characteristic values. The predetermined display characteristic may include a combination of chromaticity values ​​and luminance values ​​associated with the image data. For example, referring to Figure 3 , the light superposition feature value modifier 310 modifies the identified corresponding portion of the plurality of light superposition feature values. In some implementations, the light superposition feature value modifier 310 applies a uniform illumination function 312 to one or more corresponding portions of the corresponding portions of the plurality of light superposition feature values. For example, if a portion of the image data represents a face, the light superposition feature value modifier 310 applies the uniform illumination function 312 to the corresponding portion of the light superposition feature values ​​to achieve a substantially uniform skin tone. In some implementations, the light superposition feature value modifier 310 applies an illumination smoothing function 314 to one or more corresponding portions of the corresponding portions of the plurality of light superposition feature values. For example, the illumination smoothing function 314 includes one of a Gaussian smoothing, a uniform moving average smoothing, and the like.

[0077] As represented by block 708, in some implementations, method 700 includes generating modified image data from the image data based on one or more modified portions of the plurality of light superposition characteristic values ​​and the reference perceptual color gamut. Figure 3 The image data modifier 320 modifies the image data based on the output from the light superposition feature value modifier 310 and the output from the reference perceptual color gamut database 230 .

[0078] As represented by block 710, in some implementations, generating modified image data includes modifying a semantically identified first portion of the image data. To this end, in some implementations, method 700 includes obtaining a plurality of semantic values ​​associated with a plurality of portions within the image data. The plurality of portions include a first portion of the image data and a second portion of the image data. For example, referring to Figure 4 , the semantic value generator 410 generates a plurality of semantic values, such as "wall", "face", "table", etc. In addition, the method 700 includes identifying a first semantic value among the plurality of semantic values ​​that meets the criteria. The first semantic value among the plurality of semantic values ​​is associated with the first portion of the image data. For example, referring to Figure 4 , the semantic value identifier 420 identifies a first semantic value among a plurality of semantic values ​​that meets the criteria. In some implementations, the first semantic value among the plurality of semantic values ​​corresponds to an object of interest, such as a "face" of a person represented within the image data. In addition, the method 700 includes, for example, via the scene object modifier 432 and / or the reference Figure 4 The depicted scene context modifier 434 modifies a first portion of the image data. Thus, by selectively modifying a portion of the image data without modifying the entire image data, the semantic-based color correction pipeline 400 reduces resource utilization in some situations.

[0079] As represented by block 712, method 700 includes transforming the modified image data into display data based on a function of a portion of the plurality of light superposition characteristic values ​​and a reference physical color gamut. For example, reference Figure 3 and Figure 4 The second gamut mapper 260 transforms the modified image data into display data based on the reference physical gamut 210 and a portion of the plurality of light superposition feature values ​​from the light superposition feature value generator 208 .

[0080] As represented by block 714, method 700 includes providing a perspective display (such as Figure 3 or Figure 4 The display data is displayed on the perspective display 270 in the embodiment.

[0081] The present disclosure describes various features, none of which can achieve the benefits described herein alone. It should be understood that the various features described herein can be combined, modified or omitted, which will be apparent to those of ordinary skill in the art. Other combinations and sub-combinations beyond those specifically described herein will be apparent to those of ordinary skill in the art and are intended to form a part of the present disclosure. Various methods are described herein in conjunction with various flow chart steps and / or stages. It should be understood that in many cases, certain steps and / or stages can be combined so that the multiple steps and / or stages shown in the flow chart can be performed as a single step and / or stage. In addition, certain steps and / or stages can be divided into additional sub-components to be performed independently. In some cases, the order of the steps and / or stages can be rearranged, and certain steps and / or stages can be omitted completely. In addition, the methods described herein should be understood to be broadly interpretable so that additional steps and / or stages other than those shown and described herein can also be performed.

[0082] Some or all of the methods and tasks described herein can be performed and fully automated by a computer system. In some cases, a computer system may include multiple different computers or computing devices (e.g., physical servers, workstations, storage arrays, etc.) that communicate and interoperate over a network to perform the functions described. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in a memory or other non-transient computer-readable storage medium or device. The various functions disclosed herein can be implemented in such program instructions, but alternatively some or all of the disclosed functions can be implemented in a dedicated circuit (e.g., ASIC or FPGA or GP-GPU) of a computer system. In the case where a computer system includes multiple computing devices, these devices may be located in the same location or not. The results of the disclosed methods and tasks can be persistently stored by converting physical storage devices such as solid-state memory chips and / or disks into different states.

[0083] The various processes defined herein contemplate the option of obtaining and utilizing users' personal information. For example, such personal information may be utilized to provide enhanced privacy screens on electronic devices. However, to the extent such personal information is collected, it should be obtained with the user's informed consent. As described herein, users should understand and control the use of their personal information.

[0084] Personal information will be used by appropriate parties only for legitimate and reasonable purposes. Parties utilizing such information will adhere to privacy policies and practices that, at a minimum, comply with applicable laws and regulations. Furthermore, such policies should be comprehensive, accessible, and recognized to meet or exceed government / industry standards. Furthermore, parties may not distribute, sell, or otherwise share such information except for any legitimate and lawful purpose.

[0085] However, users can limit the extent to which parties can access or otherwise obtain personal information. For example, settings or other preferences can be adjusted to allow users to determine whether their personal information is accessible to various entities. Furthermore, while some features defined herein are described in the context of using personal information, aspects of these features can be implemented without the use of such information. For example, if user preferences, account names, and / or location history are collected, this information can be obfuscated or otherwise generalized so that it does not identify the corresponding user.

[0086] The present disclosure is not intended to be limited to the specific implementations shown herein. Various modifications to the specific implementations described in this disclosure may be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the spirit or scope of this disclosure. The teachings of the present invention provided herein may be applied to other methods and systems and are not limited to the above-described methods and systems, and the elements and actions of the various specific implementations described above may be combined to provide more specific implementations. Therefore, the novel methods and systems described herein may be implemented in a variety of other forms; in addition, various omissions, substitutions, and changes may be made to the form of the methods and systems described herein without departing from the spirit of this disclosure. The accompanying claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of this disclosure.

Claims

1. A method for color correction, comprising: At an electronic device comprising one or more processors, non-transitory memory, and a see-through display: determining a plurality of light superposition characteristic values ​​associated with ambient light from a physical environment, wherein the plurality of light superposition characteristic values ​​quantify the ambient light; modifying the image data based on a function of the plurality of light superposition characteristic values ​​and a reference perceptual color gamut to generate modified image data; transforming the modified image data into display data based on a function of a portion of the plurality of light superposition characteristic values ​​and a reference physical color gamut associated with the see-through display; as well as displaying the display data on the see-through display, The reference physical color gamut is transformed into the reference perceptual color gamut based on a function of a color appearance model. 2 . The method of claim 1 , wherein modifying the image data comprises mapping the image data to the reference perceptual color gamut within a performance threshold based on the plurality of light superposition characteristic values. 3 . The method of claim 1 , wherein the reference physical color gamut indicates a first set of colors displayable on the see-through display, and wherein the reference perceptual color gamut indicates a second set of colors. 4 . The method of claim 1 , wherein the reference physical color gamut characterizes the see-through display when a nominal amount of ambient light from the physical environment enters the see-through display.

5. The method of claim 1, wherein modifying the image data comprises applying a tone mapping operation to the image data. The method of claim 5 , wherein the tone mapping operation corresponds to a high dynamic range (HDR) tone mapping operation.

7. The method of claim 1 , wherein transforming the modified image data into the display data comprises applying a gamut mapping operation to the modified image data, wherein the gamut mapping operation is a function of the portion of the plurality of light superposition characteristic values ​​and the reference physical color gamut. The method of claim 7 , wherein the gamut mapping operation represents at least four dimensions.

9. The method according to claim 1, further comprising: obtaining a plurality of semantic values ​​respectively associated with a plurality of portions within the image data, wherein the plurality of portions include a first portion of the image data and a second portion of the image data; as well as identifying a first semantic value of the plurality of semantic values ​​that satisfies a criterion, wherein the first semantic value of the plurality of semantic values ​​is associated with the first portion of the image data; Wherein modifying the image data is another function of a predetermined display characteristic associated with the first portion of the image data.

10. The method according to claim 1, further comprising: identifying, on the see-through display, respective portions of the plurality of light superposition characteristic values ​​based on corresponding portions of the image data; as well as modifying one or more respective ones of the respective portions of the plurality of light superposition characteristic values ​​based on a function of a predetermined display characteristic associated with the image data to generate one or more modified portions of the plurality of light superposition characteristic values; Wherein generating the modified image data is based on the one or more modified portions of the plurality of light superposition characteristic values. 11 . The method of claim 1 , wherein the electronic device comprises an environmental sensor that senses the ambient light and outputs corresponding sensor data, wherein determining the plurality of light superposition feature values ​​is based on a function of the sensor data.

12. A color correction pipeline device comprising: a light superposition feature value generator for determining a plurality of light superposition feature values ​​associated with ambient light from a physical environment, wherein the plurality of light superposition feature values ​​quantify the ambient light; an image data modifier for modifying the image data based on a function of the plurality of light superposition characteristic values ​​and a reference perceptual color gamut to generate modified image data; a first gamut mapper configured to transform a reference physical color gamut into the reference perceptual color gamut based on a function of a color appearance model; a second gamut mapper for transforming the modified image data into display data based on a function of a portion of the plurality of light superposition characteristic values ​​and the reference physical gamut; as well as A see-through display is provided for displaying the display data, wherein the reference physical color gamut is associated with the see-through display.

13. The color correction pipeline apparatus of claim 12 , wherein the second gamut mapper transforms the modified image data by applying a gamut mapping operation to the modified image data, wherein the gamut mapping operation is a function of the portion of the plurality of light superposition feature values ​​and the reference physical gamut.

14. The color correction pipeline of claim 13, wherein the gamut mapping operation represents at least four dimensions.

15. The color correction pipeline apparatus according to claim 12, further comprising: a semantic value generator configured to obtain a plurality of semantic values ​​respectively associated with a plurality of portions within the image data, wherein the plurality of portions include a first portion of the image data and a second portion of the image data; as well as a semantic value identifier for identifying a first semantic value of the plurality of semantic values ​​that satisfies a criterion, wherein the first semantic value of the plurality of semantic values ​​is associated with the first portion of the image data; Wherein the image data modifier modifies the image data based on a function of a predetermined display characteristic associated with the first portion of the image data.

16. The color correction pipeline apparatus according to claim 12, further comprising: a light superposition characteristic value identifier for identifying a corresponding portion of the plurality of light superposition characteristic values ​​on the see-through display based on a corresponding portion of the image data; as well as a light superposition characteristic value modifier for modifying one or more respective ones of the respective portions of the plurality of light superposition characteristic values ​​based on a function of a predetermined display characteristic associated with the image data to generate one or more modified portions of the plurality of light superposition characteristic values; Wherein the image data modifier modifies the image data based on the one or more modified portions of the plurality of light superposition characteristic values. 17 . The color correction pipeline apparatus of claim 16 , further comprising a depth sensor that outputs depth data, wherein the light superposition feature value identifier identifies the corresponding portion of the plurality of light superposition feature values ​​based on the depth data.

18. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic device having one or more processors and a see-through display, cause the electronic device to: determining a plurality of light superposition characteristic values ​​associated with ambient light from a physical environment, wherein the plurality of light superposition characteristic values ​​quantify the ambient light; modifying the image data based on a function of the plurality of light superposition characteristic values ​​and a reference perceptual color gamut to generate modified image data; transforming the modified image data into display data based on a function of a portion of the plurality of light superposition characteristic values ​​and a reference physical color gamut associated with the see-through display; as well as displaying the display data on the see-through display, The reference physical color gamut is transformed into the reference perceptual color gamut based on a function of a color appearance model.

Images