Color correction for electronic devices with immersive viewing

By using image sensors and chroma adaptation matrix technology in electronic devices, color correction of video feeds is solved, and the problem of color perception differences in head-mounted devices under immersive and non-immersive viewing conditions is achieved, achieving consistent visual experience sex.

CN119071457APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410672749.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-03-21
Filing Date
2024-05-28
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

There is a difference in the color perception of videos displayed by head-mounted electronic devices under immersive and non-immersive viewing conditions, resulting in inconsistent colors perceived by users when viewing the same scene under different conditions.

Method used

Color correction of video feeds ensures both immersive and non-immersive by capturing video feeds using image sensors in electronic devices and generating a chromaticity adaptation matrix adapted to immersive viewing based on the calculated brightness level and ambient light color temperature. The colors displayed under viewing conditions are consistent.

Benefits of technology

The color adaptation to video feed under different viewing conditions is achieved, reducing the color perception difference between users when viewing under different conditions, and improving the consistency of visual experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071457A_ABST
    Figure CN119071457A_ABST
Patent Text Reader

Abstract

The invention relates to color correction for electronic devices with immersive viewing. There is provided an electronic device comprising: one or more image sensors configured to capture a video feed; an image signal processor configured to perform color correction on the captured video feed based on a brightness level calculated from the captured video feed and / or based on a color temperature of illumination or a luminophor type in the captured video feed, to generate a corresponding color-corrected video feed using a first chroma adaptation model adapted to the immersive viewing condition; and one or more displays configured to output the color corrected video feed. The electronic device may also include a recording pipeline configured to record a version of the captured video feed and may share or transmit the recorded content to an external device. The external device may display the recorded content using a second chroma adaptation model adapted to the non-immersive viewing condition.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 613,004, filed on March 21, 2024, which claims the benefit of U.S. Provisional Patent Application No. 63 / 505,143, filed on May 31, 2023, which are hereby incorporated by reference in their entirety. Technical Field

[0002] The present disclosure relates generally to electronic devices, and more particularly to electronic devices such as head mounted devices. Background Art

[0003] An electronic device, such as a head mounted device, may have a camera for capturing a video feed of an external scene, and one or more displays for presenting the captured video feed of the scene to a user. The head mounted device may include a hardware or software subsystem for processing the video feed, such as a hardware / software subsystem for performing color correction on the captured video feed.

[0004] Designing a head mounted device that presents such a pass-through video feed of a scene to a user can be challenging. When a user wears the head mounted device, the user views the pass-through video feed under specific immersive viewing conditions. When the user removes the head mounted device, the user may view the video content on a display under non-immersive viewing conditions that are potentially significantly different from the immersive viewing conditions viewed through the head mounted device. The user's eyes will color adapt differently to the immersive viewing conditions and non-immersive viewing conditions. If care is not taken, the user may perceive different colors when viewing the same scene under immersive viewing conditions and non-immersive viewing conditions. Summary of the invention

[0005] An electronic device, such as a head mounted device, may include one or more cameras for capturing a video feed of a real world environment and one or more displays for presenting a pass-through video feed to a user. The electronic device may include processing circuitry for performing one or more processing functions on the captured video feed to generate a pass-through video feed.

[0006] One aspect of the present disclosure provides a method of operating an electronic device having one or more image sensors and one or more displays. The method may include: capturing a video feed using the one or more image sensors; performing color correction on the captured video feed based on a brightness level calculated from the captured video feed to generate a corresponding color-corrected video feed adapted for immersive viewing conditions; and outputting the color-corrected video feed using the one or more displays of the electronic device. The method may include: calculating the brightness level using a calibrated linear camera signal in the electronic device, and converting the calibrated linear camera signal into the brightness level using a linear regression model. The method may include: performing color correction, including generating a color-corrected video feed using a first chromaticity adaptation model adapted for immersive viewing conditions. The first chromaticity adaptation model may be a function of the peak brightness level of the one or more displays.

[0007] The method may also include recording a version of the captured video feed, and sending the recorded version of the captured video feed to an external electronic device, wherein the external electronic device is configured to display the recorded version of the captured video feed that has been color corrected using a second chromatic adaptation model adapted to non-immersive viewing conditions. The method may also include receiving content from the external electronic device, and using the one or more displays to output the received content that has been color corrected using the first chromatic adaptation model adapted to immersive viewing conditions.

[0008] One aspect of the present disclosure provides a method, comprising: capturing a video feed using one or more image sensors; performing color correction on the captured video feed based on the color temperature or illuminant type of illumination in the captured video feed to generate a corresponding color-corrected video feed adapted to immersive viewing conditions; and outputting the color-corrected video feed using one or more displays. The color correction may be performed using a chromatic adaptation model adapted to the immersive viewing conditions to generate the color-corrected video feed.

[0009] One aspect of the present disclosure provides a method comprising: capturing a video feed; rendering the video feed using a chromatic adaptation matrix for immersive viewing; modifying the chromatic adaptation matrix; and rendering the video feed using the modified chromatic adaptation matrix for non-immersive viewing. The method may include: generating the chromatic adaptation matrix based on a brightness level calculated from the captured video feed, based on an ambient light color temperature in the captured video feed, and / or based on a detected illuminant type in the captured video feed. The chromatic adaptation matrix may be a function of a peak brightness level of one or more displays in the electronic device.

[0010] One aspect of the present disclosure provides a method comprising: capturing a video feed; rendering the captured video feed using a chromaticity adaptation matrix for non-immersive viewing; modifying the chromaticity adaptation matrix; and rendering the captured video feed using the modified chromaticity adaptation matrix for immersive viewing. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a top view of an illustrative head mounted device according to some embodiments.

[0012] Figure 2 is a schematic diagram of an illustrative electronic device according to some embodiments.

[0013] Figure 3 is a diagram illustrating an electronic device operating in a bright daylight environment according to some embodiments.

[0014] Figure 4 is a diagram illustrating an electronic device operating in a dim warm light environment according to some embodiments.

[0015] Figure 5 is a graph showing different degrees of chromatic adaptation as a function of ambient brightness level under immersive and non-immersive viewing conditions according to some embodiments.

[0016] Figure 6 is a graph showing different degrees of chromatic adaptation as a function of light emitter color temperature according to some embodiments.

[0017] Figure 7 is a diagram showing an illustrative display and recording pipeline within an electronic device according to some embodiments.

[0018] Figure 8 According to some embodiments, Figures 1 to 7 A flow chart of illustrative steps for an electronic device of the type shown. DETAILED DESCRIPTION

[0019] Figure 1 A top view of an exemplary head mounted device is shown in FIG. Figure 1 As shown in , a head-mounted device such as electronic device 10 may have a head-mounted support structure such as housing 12. Housing 12 may include a portion (e.g., head-mounted support structure 12T) for allowing device 10 to be worn on the user's head. Support structure 12T may be formed of fabric, polymer, metal, and / or other materials. Support structure 12T may form a strap or other head-mounted support structure to help support device 10 on the user's head. The main support structure of housing 12 (e.g., a head-mounted housing such as main housing portion 12M) may support electronic components such as display 14.

[0020] The main housing portion 12M may include a housing structure formed of metal, polymer, glass, ceramic, and / or other materials. For example, the housing portion 12M may have a housing wall on the front side F and housing walls on adjacent top, bottom, left, and right sides formed of a rigid polymer or other rigid support structure, and these rigid walls may be optionally covered with electronic components, fabric, leather, or other soft materials, etc. The housing portion 12M may also have an internal support structure, such as a frame (chassis) and / or a structure that performs multiple functions (such as controlling airflow and heat dissipation while providing structural support).

[0021] The walls of housing portion 1254 may enclose internal components 38 in interior region 34 of device 10, and may separate interior region 34 from the environment (exterior region 36) surrounding device 10. Internal components 38 may include integrated circuits, actuators, batteries, sensors, and / or other circuits and structures for device 10. Housing 12 may be configured to be worn on a user's head, and may form glasses, spectacles, a hat, a mask, a helmet, goggles, and / or other head-mounted devices. Configurations in which housing 12 forms goggles are sometimes described herein as an example.

[0022] The front side F of the housing 12 may face outwardly, away from the user's head and face. The opposite rear side R of the housing 12 may face the user. The portion of the housing 12 located on the rear side R (e.g., the portion of the main housing 12M) may form a cover, such as cover 12C (sometimes referred to as a blind). The presence of cover 12C on the rear side R may help hide the internal housing structure, the interior components 38, and other structures in the interior area 34 from being seen by the user.

[0023] Device 10 may have one or more cameras, such as Figure 1 Camera 46. Camera 46 mounted on front face F and facing outward (toward the front of device 10 and away from the user) may sometimes be referred to herein as a forward-pointing or front-facing camera. Camera 46 may capture visual odometry information, image information processed to locate objects in the user's field of view (e.g., so that virtual content can be properly registered relative to real-world objects), image content displayed in real time to the user of device 10, and / or other suitable image data. For example, a forward-pointing (positive) camera may allow device 10 to monitor movement of device 10 relative to the environment surrounding device 10 (e.g., the camera may be used to form part of a visual odometry system or a visual-inertial odometry system). The forward-pointing camera may also be used to capture images of the environment displayed to the user of device 10. If desired, images from multiple forward-pointing cameras may be merged with each other and / or the forward-pointing camera content may be merged with computer-generated content for the user.

[0024] Device 10 may have any suitable number of cameras 46. For example, device 10 may have K cameras, where the value of K is at least one, at least two, at least four, at least six, at least eight, at least ten, at least 12, less than 20, less than 14, less than 12, less than 10, 4-10, or other suitable value. Cameras 46 may be sensitive at infrared wavelengths (e.g., camera 46 may be an infrared camera), may be sensitive at visible wavelengths (e.g., camera 46 may be a visible camera), and / or cameras 46 may be sensitive at other wavelengths. If desired, cameras 46 may be sensitive at both visible and infrared wavelengths.

[0025] The device 10 may have a left optical module and a right optical module 40. The optical module 40 supports electronic components and optical components such as light-emitting components and lenses, and therefore may sometimes be referred to as an optical assembly, an optical system, an optical component support structure, a lens and display support structure, an electronic component support structure, or a housing structure. Each optical module may include a corresponding display 14, a lens 30, and a support structure such as a support structure 32. The support structure 32, which may sometimes be referred to as a lens support structure, an optical component support structure, an optical module support structure, or an optical module portion, or a lens barrel, may include a hollow cylindrical structure with an open end or other support structures for accommodating the display 14 and the lens 30. The support structure 32 may, for example, include a left lens barrel that supports the left display 14 and the left lens 30 and a right lens barrel that supports the right display 14 and the right lens 30.

[0026] Display 14 may include an array of pixels or other display devices to produce images. Display 14 may include, for example, organic light emitting diode pixels formed on a substrate having thin film circuits and / or formed on a semiconductor substrate, pixels formed from a crystalline semiconductor die, liquid crystal display pixels, scanning display devices, and / or other display devices for producing images.

[0027] Lens 30 may include one or more lens elements for providing image light from display 14 to corresponding eye zone 13. The lens may be implemented using a refractive glass lens element, using a mirror lens structure (a catadioptric lens), using a Fresnel lens, using a holographic lens, and / or other lens systems.

[0028] When the user's eyes are located in the eye zone 13, the display (display panel) 14 operates together to form the display of the device 10 (e.g., the user's eyes can view the images provided by the corresponding left and right optical modules 40 in the eye zone 13, so that a stereoscopic image is created for the user). When the user views the display, the left image from the left optical module is fused with the right image from the right optical module.

[0029] It may be desirable to monitor the user's eyes when the user's eyes are located in the eye-adaptive zone 13. For example, it may be desirable to use a camera to capture an image of the user's iris (or other parts of the user's eyes) for user authentication. It may also be desirable to monitor the direction of the user's gaze. Eye tracking information can be used as a form of user input and / or can be used to determine where the image content resolution should be locally enhanced within the image in the fovea imaging system. In order to ensure that the device 10 can capture a satisfactory eye image when the user's eyes are located in the eye-adaptive zone 13, each optical module 40 may be provided with a camera (such as camera 42) and one or more light sources (such as light emitting diodes 44) or other light emitting devices (such as lasers, lamps, etc.). The camera 42 and the light emitting diode 44 can operate at any suitable wavelength (visible light, infrared light, and / or ultraviolet light). For example, the diode 44 can emit infrared light that is invisible (or almost invisible) to the user. This allows the eye monitoring operation to be performed continuously without interfering with the user's ability to view the image on the display 14.

[0030] Figure 2 A schematic diagram of an illustrative electronic device (such as a head-mounted device or other wearable device) is shown in FIG. Figure 2 Device 10 may operate as a standalone device and / or resources of device 10 may be used to communicate with external electronic devices. For example, communication circuitry in device 10 may be used to transmit user input information, sensor information, and / or other information to an external electronic device (e.g., wirelessly or via a wired connection). Each of these external devices may include Figure 2 Components of the type shown in device 10.

[0031] like Figure 2As shown in , a head mounted device such as device 10 may include a control circuit 20. The control circuit 20 may include storage and processing circuits for supporting the operation of the device 10. The storage and processing circuits may include storage devices such as non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured to form a solid state drive), volatile memory (e.g., static or dynamic random access memory), etc. One or more processors in the control circuit 20 may be used to collect inputs from sensors and other input devices, and may be used to control output devices. The processing circuit may be based on one or more processors (such as microprocessors, microcontrollers, digital signal processors, baseband processors and other wireless communication circuits, power management units, audio chips, application specific integrated circuits, etc.). During operation, the control circuit 20 may use the display 14 and other output devices to provide visual output and other outputs to the user. The control circuit 20 may be configured to perform operations in the device 10 using hardware (e.g., dedicated hardware or circuits), firmware, and / or software. Software code for performing operations in the device 10 may be stored on a storage circuit (e.g., a non-transitory (tangible) computer-readable storage medium storing software code). The software codes may sometimes be referred to as program instructions, software, data, instructions or code. The stored software codes may be executed by processing circuitry within circuitry 20.

[0032] To support communication between device 10 and external equipment, control circuit 20 may communicate using communication circuit 22. Circuit 22 may include an antenna, radio frequency transceiver circuits, and other wireless communication circuits and / or wired communication circuits. Circuit 22 (which may sometimes be referred to as control circuits and / or control and communication circuits) may support two-way wireless communication between device 10 and external equipment (e.g., companion devices such as computers, cellular phones, or other electronic devices, accessories such as pointing devices, controllers, computer styluses or other input devices, speakers or other output devices, etc.) via a wireless link.

[0033] For example, circuitry 22 may include radio frequency transceiver circuitry, such as wireless local area network transceiver circuitry configured to support communication via a wireless local area network link, near field communication transceiver circuitry configured to support communication via a near field communication link, cellular telephone transceiver circuitry configured to support communication via a cellular telephone link, or transceiver circuitry configured to support communication via any other suitable wired or wireless communication link. link, Wireless communications may be supported by a wireless link, a wireless link operating at frequencies between 10 GHz and 400 GHz, a 60 GHz link or other millimeter wave link, a cellular telephone link, or other wireless communications link. Device 10 may include power circuitry for transmitting and / or receiving wired and / or wireless power, if desired, and may include a battery or other energy storage device. For example, device 10 may include a coil and a rectifier to receive wireless power provided to circuits in device 10.

[0034] Device 10 may include input-output devices such as device 24. Input-output devices 24 may be used to collect user input, to collect information about the user's surroundings, and / or to provide output to the user. Device 24 may include one or more displays such as display 14. Display 14 may include one or more display devices such as an organic light emitting diode display panel (a panel having organic light emitting diode pixels formed on a polymer substrate or silicon substrate containing pixel control circuitry), a liquid crystal display panel, a micro-electromechanical system display (e.g., a two-dimensional mirror array or scanning mirror display device), a display panel having an array of pixels formed from crystalline semiconductor light emitting diode dies (sometimes referred to as micro-LEDs), and / or other display devices.

[0035] Sensors 16 in input-output devices 24 may include force sensors (e.g., strain gauges, capacitive force sensors, resistive force sensors, etc.), audio sensors (such as microphones), touch and / or proximity sensors (such as capacitive sensors, such as touch sensors forming buttons, trackpads, or other input devices), and other sensors. If desired, sensors 16 may include optical sensors (such as optical sensors that emit and detect light), ultrasonic sensors, optical touch sensors, optical proximity sensors, and / or other touch sensors and / or proximity sensors, monochrome and color ambient light sensors, image sensors (e.g., cameras), fingerprint sensors, iris scan sensors, retina scan sensors, and other biometric sensors, temperature sensors, sensors for measuring three-dimensional contactless gestures (“air gestures”), pressure sensors, sensors for detecting the position, orientation, and / or movement of device 10 and / or information about the user's head posture (e.g., accelerometers, magnetic sensors such as compass sensors, etc.), and / or proximity sensors. Sensors 16 and / or other input-output devices may be used to collect user input. For example, a button may be used to collect button press input, a touch sensor overlapped with a display may be used to collect user touch screen input, a touch pad may be used to collect touch input, a microphone may be used to collect audio input (e.g., voice commands), an accelerometer may be used to monitor when a finger contacts an input surface and may therefore be used to collect finger press input, etc.

[0036] If desired, the electronic device 10 may include additional components (see, for example, other devices 18 in input-output devices 24). Additional components may include tactile output devices, actuators for moving the movable housing structure, audio output devices such as speakers, light emitting diodes for status indicators, light sources such as light emitting diodes that illuminate portions of the housing and / or display structure, other optical output devices, and / or other circuits for collecting input and / or providing output. The device 10 may also include a battery or other energy storage device, a connector port for supporting wired communication with auxiliary equipment and for receiving wired power, and other circuits.

[0037] Display 14 may be used to present various content to the user's eyes. When viewed through eye zone 13, left and right displays 14 used to present a synthetic stereoscopic image to the user's eyes may sometimes be collectively referred to as display 14. For example, display 14 may present virtual reality (VR) content. Virtual reality content may refer to content that includes only virtual objects in a virtual reality (computer-generated) environment. As another example, display 14 may present mixed reality (MR) content. Mixed reality content may refer to content that includes virtual objects and real objects from the real-world physical environment in which device 10 is running. As another example, display 14 may only present real-world content. Real-world content may refer to content captured by one or more front-facing cameras (see, e.g., Figure 1 The real-world content captured by the front-facing camera is thus sometimes referred to as a camera pass-through feed, a (live) video pass-through feed, or a pass-through video feed (stream).

[0038] A physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one law of physics.

[0039] Wearable electronic device 10 (such as Figure 1 A user of a head mounted device 10 (such as a head mounted device 10) can view the display 14 of the device 10 under "immersive" viewing conditions, which may be defined herein as viewing conditions or viewing environments in which the user sees transparent content output from the display 14 when the device 10 is mounted on the user's head. Because the device 10 is mounted in front of and covers the user's eyes, the housing 12 of the device 10 (see FIG. 1 ) is typically made of an opaque material. Figure 1 ) will block ambient light from the surrounding environment (scene). Therefore, the immersive viewing condition of device 10 will be determined by the peak / maximum brightness level (in nits) of display 14. Immersive viewing conditions may therefore also refer to a predominantly dark viewing environment, such as in a dimly lit movie theater or a dark room.

[0040] In contrast, a user who is not wearing the head mounted device 10 can view the surrounding environment (scene) under a "non-immersive" viewing condition, which can be defined herein as a viewing condition or viewing environment in which the user directly sees the surrounding scene when the device 10 is not worn on the user's head. Since the device 10 no longer blocks the user's eyes, the user's eyes are exposed to the surrounding scene. Therefore, the non-immersive viewing condition can vary according to the ambient light brightness within the scene (hereinafter referred to as the "ambient lux level"). When the ambient lux level of the scene is high, the non-immersive viewing condition will be bright. When the ambient lux level of the scene is low, the non-immersive viewing condition will be dark. Generally speaking, non-head mounted devices (such as mobile phones, laptops, desktop computers, tablet computers, watches, voice-activated speakers, smart home devices, monitors, televisions, and other types of electronic devices that are not worn above the user's eyes) operate in non-immersive viewing conditions or environments.

[0041] The human eye perceives color differently depending on the current viewing conditions (e.g., the chromaticity or color adaptation behavior of the human visual system may vary based on whether the current viewing state is an immersive viewing condition or other non-immersive viewing condition). According to an embodiment, device 10 may operate using a chromaticity (color) adaptation model that is configured to simulate the behavior of the human visual system (i.e., the human eye) such that the perceived colors of a pass-through video feed of the same given scene output by display 14 match the perceived colors of the same scene if the user were viewing a given captured scene without wearing device 10. In other words, device 10 may be provided with a color adaptation model that corrects / adjusts the pass-through video feed in a manner such that the resulting corrected colors of the pass-through video feed perceived by the user under immersive viewing conditions match the colors perceived by the user if the user were viewing the same scene or content under non-immersive viewing conditions (e.g., if the user were viewing the same scene without wearing a head mounted device, or if the user were viewing the same captured content on a non-head mounted device).

[0042] Figure 3 is a diagram illustrating operation of device 10 in a bright daylight environment according to some embodiments. Figure 3As shown, the device 10 can be worn on a user's head 100, and one or more forward-facing cameras 46 can be used to capture a video feed of a surrounding scene 104. The surrounding scene 104 may, for example, include a sheet of paper 106 illuminated by daylight 108 emitted from the sun 110. The scene 104 may therefore represent an outdoor environment with a relatively bright ambient light level. Daylight 108 (or natural light) may be considered "warm" when projecting orange light (such as during sunset), or may be considered "cold" when projecting a blue hue (such as during a cloudy or overcast day). Warm light may refer to and be defined herein as light with a color temperature ranging from 2200 Kelvin (K) to 3500K, less than 3500K, less than 3300K, less than 3000K, or less than 2700K. Cold light may refer to and be defined herein as light with a color temperature ranging from 4000K to 7000K, greater than 3500K, greater than 4000K, or greater than 5000K.

[0043] Camera 46 of device 10 may be used to capture a pass-through video feed of scene 104. Device 10 may process the captured video feed and output the processed video feed on display 14. Because device 10 is worn on a user's head 100, the user's eyes (such as eye 101) may see the processed video feed output from display 14 under immersive viewing conditions. The immersive viewing conditions may be determined by internal viewing environment 102, which is determined by the peak brightness level of the display, since the housing of device 10 will block ambient light in the surrounding scene 104. Figure 3 In this particular example of , consider that the surrounding scene 104 is illuminated by relatively cool daylight 108 and exhibits a relatively high ambient lux (brightness) level. If the user is not wearing or using the device 10, the user's eyes 101 will naturally react and perceive the scene 104 using a first amount (or degree) of chromatic adaptation under this non-immersive viewing condition. However, if the user is wearing the device 10, the user's eyes 101 will now adapt to the darker viewing environment 102 and will naturally react and perceive the displayed content using a second amount of chromatic adaptation that is different from the first amount of chromatic adaptation under immersive viewing conditions. If care is not taken, the color of the paper 106 will appear different to the user under immersive viewing conditions and non-immersive viewing conditions. The device 10 may be provided with a camera image signal processing pipeline and / or a display processing pipeline that implements a color correction scheme that takes into account this difference in the amount of chromatic (chromatic) adaptation.

[0044] Figure 4 is a diagram showing the operation of the device 10 in a dim warm light environment according to some embodiments. Figure 3As shown, device 10 may be worn on a user's head 100, and one or more forward-facing cameras 46 may be used to capture a video feed of a surrounding scene 105. The surrounding scene 105 may, for example, include a sheet of paper 106 illuminated by warm light 112 emitted from a candle 114. Scene 105 may represent an indoor environment with a relatively low ambient lux level. Camera 46 of device 10 may be used to capture a pass-through video feed of scene 105. Device 10 may process the captured video feed and output the processed video feed on display 14. Since device 10 is worn on the user's head 100, the user's eyes (such as eye 101) may see the processed video feed output from display 14 under immersive viewing conditions. The immersive viewing conditions may be determined by the internal viewing environment 102, which is determined by the peak brightness of display 14, because the housing of device 10 will block ambient light in the surrounding scene 105.

[0045] In Figure 4 In this particular example of , the surrounding scene 105 is being illuminated by relatively warm candlelight 112 and exhibits a relatively low (dark) ambient lux level. If the user is not wearing or using the device 10, the user's eyes 101 will naturally react and perceive the scene 104 using the third amount of chromatic adaptation under this non-immersive viewing condition. However, if the user is wearing the device 10, the user's eyes 101 will now be adapted to the dark viewing environment 102, and will naturally react and perceive the displayed content using a fourth amount of chromatic adaptation that is potentially different from the third amount of chromatic adaptation under immersive viewing conditions. If care is not taken, the color of the paper 106 illuminated by the candlelight 112 will appear different to the user under immersive viewing conditions and non-immersive viewing conditions. The device 10 may be provided with a display processing pipeline that implements a chromaticity correction scheme that takes into account this difference in chromatic (chromatic) adaptation amounts.

[0046] Figure 5 is a graph showing different degrees of chromatic adaptation as a function of ambient lux level under immersive and non-immersive viewing conditions according to some embodiments. The terms "chromatic adaptation" and "color adaptation" are used interchangeably herein. Curve 150 represents a chromatic (color) adaptation level curve or model that can be implemented by device 10 when display 14 exhibits a first peak brightness level, depending on the ambient lux level corresponding to non-immersive viewing conditions. On the other hand, curve 151 represents a chromatic (color) adaptation level curve or model that depends on the ambient lux level corresponding to immersive viewing conditions when display 14 exhibits a first peak brightness level. Therefore, curve 150 is sometimes referred to as a non-immersive viewing condition color adaptation level curve / model, while curve 151 is sometimes referred to as an immersive viewing condition color adaptation level curve / model.

[0047] At all ambient lux levels, the non-immersive viewing condition chromatic adaptation curve 150 exhibits a higher amount or degree of chromatic adaptation than the immersive viewing condition chromatic adaptation level curve 151. At high ambient lux levels (e.g., at very bright levels, such as when the scene being captured is illuminated by sunlight or other bright lighting), the non-immersive viewing condition may require full chromatic adaptation or more chromatic adaptation relative to the immersive viewing situation. At low ambient lux levels (e.g., at very dark levels, such as when the scene being captured is illuminated by moonlight or other dim lighting), the immersive viewing condition may require much less chromatic adaptation relative to the non-immersive viewing situation. A higher amount / degree of chromatic adaptation may indicate a greater degree of color adjustment to a neutral gray or white point, while a lower amount of chromatic adaptation may indicate a lesser degree of color adjustment to a neutral gray or white point.

[0048] The degree of color adaptation for non-immersive and immersive viewing conditions may vary depending on the peak / maximum brightness level of display 14. Figure 5 As shown, curve 150' represents a chromaticity (color) adaptation level curve or model that can be implemented by device 10 when display 14 exhibits a second peak brightness level greater than the first peak brightness level, depending on the ambient lux level corresponding to the non-immersive viewing condition. On the other hand, curve 151' represents a chromaticity (color) adaptation level curve or model that depends on the ambient lux level corresponding to the immersive viewing condition when display 14 exhibits a second peak brightness level. As an illustrative example, the first peak brightness level can be equal to 100 nits, and the second peak brightness level can be equal to 300 nits. As shown in FIG. Figure 5 As shown, non-immersive viewing condition color adaptation curves 150 and 150' tend to show smaller differences or changes as the peak brightness level changes, while immersive viewing condition color adaptation curves 151 and 151' tend to show larger differences or changes as the peak brightness level changes.

[0049] Figure 6 is a graph showing different degrees of chromatic adaptation as a function of illuminant color temperature under immersive and non-immersive viewing conditions according to some embodiments. Light sources within a scene or environment are sometimes referred to herein as illuminants. Typically, an environment may include one or more types of illuminants (e.g., a scene may include only one type of light source, two types of light sources, three to five types of light sources, five to ten types of light sources, or more than ten different types of light sources). Figure 6 The illuminant color temperature on the x-axis of the graph of can represent the color temperature of the dominant illuminant in the scene, or can also represent the type of illuminant in the scene (e.g., whether the detected light source is a blackbody light source, a non-blackbody light source, or other light source). The color temperature is sometimes referred to as the "correlated" color temperature (CCT), which is mapped to the color temperature of a blackbody light source.

[0050] Curve 152 represents a chromaticity (color) adaptation level curve or model that can be implemented by device 10 when display 14 exhibits a first peak brightness level, depending on the color temperature of ambient light corresponding to non-immersive viewing conditions. On the other hand, curve 153 represents a chromaticity (color) adaptation level curve or model that can be implemented by device 10 when display 14 exhibits a first peak brightness level, depending on the color temperature of ambient light corresponding to immersive viewing conditions. Therefore, curve 152 is sometimes referred to as a non-immersive viewing condition color adaptation level curve / model, while curve 153 is sometimes referred to as an immersive viewing condition color adaptation level curve / model.

[0051] At all color temperatures, the non-immersive viewing condition color adaptation level curve 152 exhibits a higher amount or degree of color adaptation than the immersive viewing condition color adaptation level curve 153. At (higher) or (cooler) color temperatures, the non-immersive viewing condition may require full color adaptation or higher color adaptation relative to the immersive viewing situation. At (lower) or (warmer) color temperatures, the immersive viewing condition color adaptation level curve 153 exhibits a lower amount or degree of color adaptation than the non-immersive viewing condition color adaptation level curve 152 (e.g., the immersive viewing condition may require much less color adaptation or minimal color adaptation relative to the non-immersive viewing situation at low or warm color temperatures). A higher amount / degree of color adaptation may indicate a greater degree of color adjustment to a neutral gray or white point, while a lower amount of color adaptation may indicate a smaller degree of color adjustment to a neutral gray or white point.

[0052] The degree of color adaptation for non-immersive and immersive viewing conditions as a function of color temperature may also vary depending on the peak / maximum brightness level of display 14. Figure 6 As shown, curve 152' represents a chromaticity (color) adaptation level curve or model that can be implemented by device 10 when display 14 exhibits a second peak brightness level greater than the first peak brightness level, depending on the color temperature corresponding to the non-immersive viewing condition. On the other hand, curve 153' represents a chromaticity (color) adaptation level curve or model that depends on the color temperature corresponding to the immersive viewing condition when display 14 exhibits a second peak brightness level. As an illustrative example, the first peak brightness level can be equal to 100 nits, and the second peak brightness level can be equal to 300 nits or other values ​​greater than the first peak brightness level. As shown in FIG. Figure 6As shown, non-immersive viewing condition color adaptation curves 152 and 152' tend to show smaller differences or changes as peak brightness levels vary, while immersive viewing condition color adaptation curves 153 and 153' tend to show larger differences or changes as peak brightness levels vary. At very high color temperatures, the degree of color adaptation for immersive viewing conditions may actually decrease (see, for example, curves 153 and 153 sag in region 160) because human responses in immersive viewing situations may not fully adapt under very cold lighting conditions.

[0053] Scene 104 uses cold light sources to show high ambient lux levels. Figure 3 An example of a scene 105 in which a warm luminaire is used to present a low ambient lux level Figure 4 The examples are illustrative and are not intended to limit the scope of embodiments of the present invention. For another example, device 10 may operate in a scene with a high ambient lux level illuminated by a warm or warmer light source. For another example, device 10 may operate in a scene with a low ambient lux level illuminated by a cool or cooler light source. In general, device 10 may operate in scenes with high, low, or medium ambient lux levels and illuminated by cool, neutral, or warm light sources.

[0054] According to some embodiments, the amount of desired chromatic adaptation level employed by device 10 may be controlled based primarily on the illuminant color temperature (e.g., Figure 6 According to other embodiments, the desired amount of chromatic adaptation level applied by device 10 (e.g., Figure 5 According to other embodiments, the desired chromatic adaptation level amount may be a function of a weighted sum of the ambient lux level and the ambient light color temperature. The amount of chromatic adaptation used by device 10 to perform color correction operations on the captured pass-through feed may be represented by a chromatic (color) adaptation model, which may be encoded in some data structure such as a chromatic (color) adaptation matrix.

[0055] In some applications, it may be desirable to record a version of the real-world content or augmented reality content presented on the display 14 of the head mounted electronic device 10. Such a recording may be stored for later review, may be used for debugging or troubleshooting, or may be shared with an external device or remote server. Figure 7 1 is a diagram illustrating various hardware and software subsystems that may be included in device 10 for displaying content and for separately recording a version of the displayed content. The content may be displayed at a first frame rate and may optionally be recorded at a second frame rate that is different from the first frame rate. Figure 7As shown, the device 10 may include a graphics rendering subsystem such as a graphics rendering pipeline 56, a user tracking subsystem including one or more tracking sensors 54, an imaging subsystem including one or more image sensors 50, an image signal processing subsystem such as an image signal processor (ISP) 52, a virtual content synthesis subsystem such as a virtual content synthesizer 58, and a media merging subsystem such as a media merging synthesizer 60.

[0056] The graphics rendering pipeline 56, sometimes referred to as a graphics rendering engine or graphics renderer, may be configured to render or generate virtual content (e.g., virtual reality content, augmented reality content, mixed reality content, or extended reality content) or may be used to perform other graphics processing functions. Virtual content output from the graphics rendering pipeline may optionally be foveated (e.g., the subsystem 56 may render foveated virtual content). The graphics rendering pipeline 56 may synthesize a photorealistic or non-photorealistic image from one or more 2- or 3-dimensional models defined in a scene file that contains information about how to simulate various features, such as shading (e.g., how the color and brightness of a surface varies with lighting), shadowing (e.g., how shadows are cast on objects), texture mapping (e.g., how details are applied to surfaces), reflections, transparency or opacity (e.g., how light is transmitted through solid objects), translucency (e.g., how light is scattered through solid objects), refraction and diffraction, depth of field (e.g., how certain objects may appear out of focus when outside the depth of view), motion blur (e.g., how certain objects may appear blurry due to rapid motion), and / or other visible features related to lighting or physical properties of objects in the scene. The graphics renderer 56 may apply a rendering algorithm, such as rasterization, ray casting, ray tracing, radiosonde, or other graphics processing algorithms.

[0057] The tracking sensor 54 may include a gaze tracking subsystem, sometimes referred to as a gaze tracker, which is configured to collect gaze information or gaze point information. The gaze tracker may employ one or more inwardly facing cameras and / or other gaze tracking components (e.g., components facing the eyes and / or other light sources emitting light beams so that reflections of light beams from the user's eyes can be detected) to monitor the user's eyes. One or more gaze tracking sensors may face the user's eyes and may track the user's gaze. The camera in the gaze tracking subsystem may determine the position of the user's eyes (e.g., the center of the user's pupil), may determine the direction in which the user's eyes are oriented (the direction of the user's gaze), may determine the user's pupil size (e.g., so that light modulation and / or other optical parameters are adjusted based on pupil size, and / or a sequential amount used to spatially adjust one or more of these parameters, and / or an area in which one or more of these optical parameters are adjusted), may be used to monitor the current focal length of the lens in the user's eyes (e.g., whether the user is focused on the near field or the far field, which may be used to assess whether the user is daydreaming or thinking strategically or tactically), and / or other gaze information. The camera in the gaze tracker may sometimes be referred to as an inward-looking camera, a gaze detection camera, an eye tracking camera, a gaze tracking camera, or an eye monitoring camera. If desired, other types of image sensors (e.g., infrared and / or visible light emitting diodes and photodetectors, etc.) may also be used to monitor the user's gaze.

[0058] The tracking sensors 54 may also include a face and body tracking subsystem that is configured to perform face tracking (e.g., to capture images of the user's jaw, mouth, etc. when the device is worn on the user's head) and body tracking (e.g., by capturing images of the user's torso, arms, hands, legs, etc. when the device is worn on the user's head). If desired, the face and body tracking subsystem may also track the user's head pose by directly determining any movement, yaw, pitch, roll, etc., of the head mounted device 10. The yaw, roll, and pitch of the user's head may collectively define the user's head pose.

[0059] Tracking sensors 54 may also include a hand tracking subsystem, sometimes referred to as a hand tracker, that is configured to monitor the user's hand movements / gestures to obtain hand gesture data. For example, a hand tracker may include a camera and / or other gesture tracking components (e.g., an outward-facing component and / or a light source that emits a light beam so that reflections of the light beam from the user's hand can be detected) to monitor the user's hand. One or more hand tracking sensors 54 may be directed toward the user's hand and may track motion associated with the user's hand, may determine whether the user is performing a swiping motion with his / her hand, may determine whether the user is performing a non-contact button press or object selection operation with his / her hand, may determine whether the user is performing a grasping or gripping motion with his / her hand, may determine whether the user is using his / her hand or finger to point to a given object presented on display 14, may determine whether the user is performing a waving or striking motion with his / her hand, or may generally measure / monitor three-dimensional non-contact gestures ("air gestures") associated with the user's hand. Tracking sensors 54 that are operable to obtain gaze, posture, hand gestures, and other information related to the movements of a user of device 10 are sometimes referred to as user tracking sensors.

[0060] The virtual content generated by the graphics rendering pipeline 56 and the user tracking information output from the user tracking sensor 54 (e.g., gaze point information, head posture information, hand gesture information, and information associated with other user body parts) can be delivered to the virtual content synthesizer 58. Based on the content and information from multiple data sources, the virtual content synthesizer 58 can generate a corresponding synthesized virtual frame. The virtual content synthesizer 58 can perform various synthesizer functions to adjust the virtual content based on the user tracking information to help improve the image quality of the final content to be displayed to the user. The adjustment of the virtual content can be performed by the virtual content synthesizer 58 and / or the media merging synthesizer 60.

[0061] For example, the virtual content synthesizer 58 may perform the following operations: image warping operations for re-projecting virtual content from one user perspective to another user perspective, dynamic foveation operations for adjusting the detail or quality of a video feed based on a user's gaze (e.g., by increasing the image detail or resolution of a video feed in an area of ​​the user's gaze and / or reducing the image detail or resolution of a video feed in an area that is not aligned with the user's gaze), lens distortion compensation operations for fixing problems associated with distortion that may be caused by the lens 30 in front of the display 14, brightness adjustment, color shifting, chromatic aberration correction, optical crosstalk mitigation operations, and / or other optical correction processes for enhancing the apparent quality of the synthesized virtual frames.

[0062] The decisions made by the virtual content compositor 58 or other display control functions to generate each synthesized virtual frame may be listed in a compositor descriptor such as descriptor 64. Descriptor 64 may include a list of image correction / adjustment parameters used in generating the corresponding synthesized virtual frame, including, but not limited to, which input frame(s) from the virtual content were used, specific gaze parameters from a gaze tracker, specific head pose parameters from a body tracker that may be used to determine how image warping operations are to be performed, specific hand gesture parameters from a hand tracker, specific fovea parameters used in performing dynamic foveation, specific lens distortion compensation parameters, specific brightness adjustments, specific amounts or types of color shifts, and / or a specific amount or degree of chromatic aberration correction currently being used to generate the synthesized virtual frame.

[0063] Image correction or adjustment may be applied at the virtual content compositor 58 or some other component such as the media merge compositor 60. In embodiments where image correction / adjustment is performed at the media merge compositor 60, the virtual content compositor 58 may send a mesh including corrections based on gaze parameters, head pose parameters, hand gesture parameters, image warping parameters, fovea parameters, brightness adjustment parameters, color adjustment parameters, chromatic aberration correction parameters, viewpoint correction parameters, and / or other parameters to the media merge compositor 60. If desired, these correction parameters may be sent from the virtual content compositor 58 to the media merge compositor 60 in the form of one or more compositor descriptors 64.

[0064] Operating in this manner, the virtual content compositor 58 may forward its image correction decisions to the media merge compositor 60, and the media merge compositor 60 may then execute those decisions on the virtual frames and / or pass-through feeds, and subsequently perform the desired merge or blending on the corrected video frames. Thus, the merged video frames being delivered to the display 14 may be output according to a first frame rate and / or a first set of image correction parameters provided from the virtual content compositor 58 to the media merge compositor 60, while the video frames being recorded in the recording pipeline 68 may be output according to a second frame rate and / or a second set of image correction parameters provided from the virtual content compositor 58 to the recording pipeline 68 that is different from the first frame rate.

[0065] A separate synthesizer descriptor 64 may be generated for each synthesized virtual frame. Multiple synthesizer descriptors 64 corresponding to a series of synthesized virtual frames may be stored in a shared memory 62. The shared memory 62 may be implemented as a volatile memory 38, such as a random access memory (e.g., a dynamic RAM or a static RAM). Alternatively, the shared memory 62 may also be implemented as a non-volatile memory. Examples of non-volatile memories include flash memory, a solid state drive (SSD), a read-only memory (ROM), an electrically programmable ROM, a disk storage, a hard disk drive, an optical disk, etc. Each descriptor 64 may be a relatively small amount of data. Each descriptor 64 may, for example, be only hundreds or thousands of bytes of data (e.g., less than 5 kB of data, less than 10 kB of data, less than 100 kB of data, or less than 1 megabyte of data).

[0066] The synthesized virtual frame may be combined with a real-time video feed captured by one or more image sensors 50 and then output at the display 14. The sensor 50 may include one or more front-facing cameras and / or other cameras for capturing images of the external real-world environment surrounding the device 10. The video feed output from the camera 50 may sometimes be referred to as a raw video feed or a real-time transparent video stream. The transparent feed output from the camera 50 may be processed by an image signal processor (ISP) 52 configured to perform image signal processing functions that rely only on the input of the real-time camera feed itself. For example, the ISP block 52 may be configured to perform automatic exposure for controlling the exposure setting of the transparent video feed, automatic color correction for controlling white balance (sometimes referred to as automatic white balance), tone curve mapping, gamma correction, shading correction, noise reduction, black level adjustment, demosaicing, image sharpening, high dynamic range (HDR) correction, color space conversion and / or other image signal processing functions (to name a few) to output the corresponding processed video frame.

[0067] Media merging compositor 60 may receive processed video frames output from image signal processor 52, may receive synthesized virtual frames output from virtual content compositor 58, and may overlay or otherwise combine one or more portions of these synthesized virtual frames with the processed video frames to obtain corresponding merged video frames. The merged video frames output from media merging compositor 60 may then be presented on display 14 for viewing by a user of device 10. Figure 7 In the example of FIG. 5 , the combined video frames may be displayed at a frame rate of X frames per second (fps). To achieve this, the virtual content synthesizer 58 may generate synthesized virtual frames at Xfps. For example, X may be 90fps. This is exemplary only. In general, the content presented on the display 14 may be output at less than 90fps, 60fps or less, 50fps or less, 30fps or less, greater than 90fps, 120fps, 150fps, or other display frame rates.

[0068] To provide recording capabilities to device 10, device 10 may include a separate recording subsystem, such as recording pipeline 68. In contrast to the display pipeline that outputs merged video frames at X fps, recording pipeline 68 may render or generate video frames for recording at Y fps, where Y is different from X. The recording frame rate Y may be less than the display frame rate X. For example, when X is 90 fps, Y may be 30 fps. This is merely exemplary. The recording frame rate Y may optionally be greater than the display frame rate X. In general, the content recorded at recording pipeline 68 may be rendered at less than 30 fps, 20 fps or less, 10 fps or less, greater than 30 fps, 60 fps, 90 fps, 120 fps, or other recording frame rates.

[0069] like Figure 7 As shown, the recording pipeline 68 may include a recording synthesizer 70, a recorder processing block 72, and a recorder memory 74. The recording synthesizer 70 may receive the virtual content output from the graphics rendering pipeline 56, and may retrieve one or more synthesizer descriptors 64 from the shared memory 62. Operating in this manner, the recording synthesizer 70 may use the image correction parameters or decisions listed in the retrieved descriptors 64 to generate equivalent synthesized virtual frames for the purpose of recording at different frame rates. The virtual frames output from the recording synthesizer 70 are sometimes referred to or defined herein as "resynthesized" virtual frames (e.g., virtual frames resynthesized at a different frame rate than the frames synthesized and displayed at the display 14 or otherwise with different parameters). The synthesized virtual frames output from the recording synthesizer 70 may optionally be merged with the processed real-time pass-through feed output from the image signal processor 52 using the recording processor 72. The merged video frames output from the recording processor 72 may then be stored or recorded in the memory 74. This example in which the merged content is being recorded is merely illustrative. If desired, the synthesized virtual frames and processed pass-through feeds output from the recording synthesizer 70 may be stored separately in the memory 74 without being merged together. In other suitable embodiments, the recording synthesizer 70 may output different layers (or files), each layer (file) based on different image correction parameters in the retrieved descriptor, and each layer (file) may be stored or recorded separately on the memory 74 without being merged with any other layer or pass-through frame. For example, the recording synthesizer 70 may output a pass-through video feed, a resynthesized virtual content frame generated based on the descriptor 64, a hand matting file obtained using the hand tracking subsystem, and / or other independent output streams for later processing.

[0070] In addition to the merging operation, the recording processor 72 may optionally perform additional operations, such as selecting a subset of the received frames for recording (e.g., selecting alternating frames to be recorded, selecting a frame from every three frames to be recorded, selecting a frame from every four frames to be recorded, selecting a frame from every five to ten frames to be recorded, etc.), limiting the rendered frames to a smaller field of view (e.g., limiting the X or horizontal dimension of the rendered content, limiting the Y or vertical dimension of the rendered content, or otherwise constraining the size or range of the frames to be recorded), undistorting the rendered content (because the recorded content may not be viewed through a lens during later playback), and video matting (to name a few). The video matting operation may determine whether each portion of the recorded content shows synthesized virtual content or real-time pass-through content. In some scenarios, the video matting operation may decide to display more real-time pass-through content when displaying more real-time pass-through content will enhance the user's safety (e.g., such as when the user may be moving towards an obstacle). In other scenarios, the video cutout operation may intentionally obscure a portion of the virtual content or blur a portion of the virtual content (e.g., such as when a user enters a password or other sensitive information on a display screen, and the sensitive information may be obfuscated in the recording). These examples are merely illustrative. In another embodiment, a hand cutout operation may detect the user's hand in a pass-through feed and extract only that portion from the pass-through video feed. The extracted portion may later be mixed with virtual content that targets only the user's hand. In general, the recording processor 72 may create or output recorded content that is at least partially different from the content being output from the display 14.

[0071] To help protect the privacy of users, best practices may be used to handle any personal user information collected by sensors. These best practices include meeting or exceeding any applicable privacy regulations. Opt-in and opt-out options and / or other options may be provided that allow users to control the use of their personal data.

[0072] exist Figure 7 In an example of , the ISP 52 may include an automatic exposure (AE) subsystem, such as an AE block 78 for determining the ambient lux (brightness) level of the scene. For example, the brightness level may be calculated using a calibrated linear camera signal. The calibrated linear camera signal may then be converted to a brightness level based on a linear regression model. The ISP 52 may also include a color correction subsystem, such as a color correction block 80 configured to perform a color correction (auto white balance or AWB) operation based on a chromaticity adaptation model 82. The chromaticity adaptation model 82 may refer to Figure 5 Chromatic adaptation curve 150 or Figure 6Immersive viewing condition chromatic adaptation curve 152 of the type shown. The chromatic adaptation model 82 for simulating the human eye (visual) response to immersive viewing conditions may be represented using any suitable data structure, such as a chromatic adaptation matrix.

[0073] like Figure 7 As shown, the chromatic adaptation matrix may be delivered to the graphics rendering pipeline 56 via the dashed path 84 so that the pipeline 56 can render the virtual content using color settings that match the processed pass-through video feed output from the ISP 52. The chromatic adaptation matrix may also optionally be delivered to the recording pipeline 68 via the path 84 and stored in the recorder memory 74. Operating in this manner, the recording pipeline 68 may know the amount of color correction applied by the display pipeline and may selectively apply the same amount of color adaptation specified by the chromatic adaptation matrix to the recorded content if the recorded content will later be viewed under immersive viewing conditions, or may selectively apply a different amount of color adaptation to the recorded content if the recorded content will later be viewed under non-immersive viewing conditions. This example is illustrative in that the chromatic adaptation matrix is ​​used to match virtual content to pass-through content. In other embodiments, the inverse of the chromatic adaptation matrix may be applied to the pass-through content (e.g., to remove any ambient colors from the pass-through feed) to match the pass-through content to the virtual content.

[0074] Figure 8 It is used to operate the combination Figures 1 to 7 Flowchart of illustrative steps for an electronic device 10 of the type described. During operation of block 200, one or more sensors in device 10 may be used to detect ambient lux (brightness) levels in a scene to be captured. For example, device 10 may include one or more front-facing cameras for capturing a live pass-through feed (e.g., the ambient lux or brightness level may be calculated from the captured video feed). Information detected using the front-facing camera may be processed by auto-exposure block 78 to determine a corresponding ambient lux level.

[0075] During the operations of block 202, one or more sensors in device 10 may be used to detect or estimate the color temperature and chromaticity values ​​of light emitters of ambient lighting in a scene to be captured. For example, device 10 may include a flicker detection sensor or other color sensor configured to gather flicker information and detect whether a light emitter within a captured scene is a cold light source, a warm light source, a neutral light source (e.g., a light emitter that is considered neither cold nor warm), a black body light source, a non-black body light source, or other light source. In some embodiments, the flicker detection sensor may be included as a Figure 2The flicker detection sensor may be a portion of sensor 16 in FIG. 1 and may include one or more light measurement channels. For example, a flicker detection sensor may include channels configured to measure infrared (IR) and visible light, channels configured to measure only infrared light, and / or multiple channels configured to measure the spectrum at many visible wavelengths. In some embodiments, a color sensor may be included as Figure 2 4. For example, an RGB camera may be used to estimate the light source color temperature and chromaticity values. Although block 202 is shown as occurring after block 200, block 202 may be performed before or in parallel (simultaneously) with block 200.

[0076] During operation at block 204, color correction block 80 in ISP 52 may determine a degree or amount of chromatic adaptation corresponding to immersive viewing conditions based on the ambient lux level detected during block 200 and / or based on the estimated color temperature or illuminant type detected during block 202. In some embodiments, the appropriate amount of chromatic adaptation may be determined primarily as determined by Figure 5 The chromatic adaptation curve 151 or 151' is modeled based on the ambient lux level for immersive viewing conditions. In other embodiments, the appropriate amount of chromatic adaptation may be based primarily on the ambient lux level as determined by Figure 6 The chromatic adaptation curve 153 or 153' for immersive viewing conditions is the type of illuminant or color temperature of the ambient light modeled by the chromatic adaptation curve 153 or 153'. In some other embodiments, the appropriate amount of chromatic adaptation can be a function of the ambient lux level, the type of illuminant, the color temperature, flicker information, the amount of IR component of the detected illumination in the captured scene, time information about the AC or DC component of the detected illumination in the captured scene, information about whether the device 10 may be located indoors or outdoors as determined using a flicker detection sensor, some combination of these factors, and / or other parameters. The amount of chromatic adaptation to be applied to the transparent video feed can be encoded in the form of a chromatic adaptation matrix. For example, the degree of adaptation calculated from line 151 will generate a corresponding chromatic adaptation matrix suitable for immersive viewing at a specific ambient lux level and display brightness combination. For another example, another chromatic adaptation matrix suitable for immersive viewing can be generated from line 151 at different ambient lux levels and / or display brightness combinations. For another example, a chromatic adaptation matrix suitable for non-immersive viewing can be generated based on line 150 for each ambient lux level and display brightness combination.

[0077] During operations at block 206, ISP 52 may render the pass-through content using the chroma adaptation matrix generated from block 204. For example, ISP 52 may perform a color correction operation on the pass-through video feed using the chroma adaptation matrix to output a corresponding color-corrected video feed, sometimes referred to as a chroma-adapted pass-through video feed.

[0078] During the operation of block 208, the graphics rendering pipeline 56 may render virtual content using the same chromaticity adaptation matrix generated from block 204. In other words, the graphics rendering block may generate virtual content using the chromaticity adaptation matrix to output corresponding color-corrected virtual content, sometimes referred to as chromaticity-adapted virtual content. Operating in this manner, the chromaticity-adapted virtual content may be color-matched with the chromaticity-adapted pass-through video feed so that the two may later be merged by the media merge compositor 60 or the recording processor 72 to obtain a consistent merged video frame, at least in terms of overall color. Although block 208 is shown as occurring after block 206, the operation of block 208 may occur before or in parallel (simultaneously) with block 206.

[0079] During the operations of block 210, one or more displays 14 on device 10 may present the merged video frame under immersive viewing conditions. The content of the merged video frame, which includes at least the pass-through video feed captured using camera 50 and the virtual content generated by graphics renderer 56, has been color corrected or chromatically adapted using the chromatically adapted matrix generated during block 204.

[0080] During the operation of block 212, the recording pipeline 68 on the device 10 may record a version of the displayed content. The chromatic adaptation model applied to the displayed content during blocks 206 and 208 may be saved as metadata (sometimes referred to herein as color metadata) in the recorder memory 74. During the operation of block 214, the recorded content and the color metadata, optionally including the chromatic adaptation matrix, may optionally be shared or sent (exported) to one or more other electronic devices. The other devices receiving the shared information may present the recorded content under non-immersive viewing conditions or immersive viewing conditions.

[0081] During the operation of block 216, other devices (e.g., mobile phones, laptops, desktop computers, tablet computers, watches, automatic voice control assistants, smart home devices, monitors, televisions, and other types of electronic devices that are not worn above the user's eyes) may render the recorded content for non-immersive viewing conditions. Such types of devices may sometimes be referred to as non-immersive viewing electronic devices. In such scenarios, other non-immersive viewing devices will analyze the shared color metadata including the chromatic adaptation matrix corresponding to the immersive viewing conditions and apply different amounts of chromatic adaptation that are more suitable for the non-immersive viewing conditions. For example, other devices may selectively adjust or modify the values ​​in the shared chromatic adaptation matrix, may convert the chromatic adaptation matrix to a different color space, or may increase the degree of chromatic adaptation. The non-immersive viewing device may then use a modified version of the chromatic adaptation matrix to render the shared recorded content (e.g., the non-immersive viewing device may use another version of the chromatic adaptation matrix adapted to the non-immersive viewing conditions to render the shared recorded content in its display pipeline). This example of modifying the recorded content at the non-immersive viewing device when the content is to be displayed is illustrative. In other embodiments, the recorded content may be modified at device 10 before the content is shared when exported (e.g., during block 214), or at a receiving non-immersive viewing device when the shared content is received. In other embodiments, converting a video stream generated for immersive viewing to non-immersive viewing may involve applying the inverse of an immersive viewing chromatic adaptation matrix to the video stream, and then applying the non-immersive viewing chromatic adaptation matrix to the resulting video stream.

[0082] The above examples in which the other device receiving the shared recorded content is a non-immersive viewing electronic device are illustrative. In other embodiments, the other device receiving the shared recorded content may be a device (e.g., another head-mounted device) that will also present the recorded content under immersive (dark) viewing conditions. This type of device may sometimes be referred to as an immersive viewing electronic device. In such a scenario, the other immersive viewing device will analyze the shared color metadata including the chromatic adaptation matrix corresponding to the immersive viewing conditions, and apply the same or similar chromatic adaptation amount specified in the chromatic adaptation matrix to the recorded content, which will be suitable for the immersive viewing conditions. For example, the other device may maintain at least some of the values ​​in the shared chromatic adaptation matrix (e.g., to maintain the degree or amount of chromatic adaptation), may convert the chromatic adaptation values ​​to a different color space, and the like.

[0083] The above examples in which device 10 (e.g., an immersive viewing device) shares content recorded for immersive viewing with other external devices that may be non-immersive viewing devices or immersive viewing devices are illustrative. In other scenarios, device 10 may also receive (import) content recorded at an external non-immersive viewing device. For example, during the operation of block 218, content recorded at an external non-immersive viewing device and color metadata, optionally including a chromatic adaptation matrix, may optionally be shared with device 10 or received (imported) at the device.

[0084] During the operation of block 220, the device 10 may render the imported content for immersive viewing conditions. For example, the device 10 will analyze the shared color metadata including the chromatic adaptation matrix corresponding to the non-immersive viewing conditions, and apply a different amount of chromatic adaptation that is more suitable for the immersive viewing conditions. The device 10 may selectively adjust or modify the values ​​in the shared chromatic adaptation matrix, may convert the chromatic adaptation values ​​to a different color space, or may reduce the degree of chromatic adaptation. The device 10 may then use the modified version of the chromatic adaptation matrix to render the shared recorded content (e.g., the device 10 may render the shared recorded content in its display pipeline using a version of the chromatic adaptation matrix adapted to the immersive viewing conditions). This example of modifying the recorded content at the device 10 when the content is to be displayed is illustrative. In other embodiments, the recorded content may be modified at the device 10 before the content is imported (e.g., during block 218) at an external non-immersive viewing device or when the shared content is received. In other embodiments, converting a video stream generated for non-immersive viewing to immersive viewing may involve applying the inverse of a non-immersive viewing chromatic adaptation matrix to the video stream, and then applying an immersive viewing chromatic adaptation matrix to the resulting video stream.

[0085] Combination of the above Figures 1 to 8 The described methods and operations may be performed by the components of device 10 using software, firmware, and / or hardware (e.g., dedicated circuits or hardware). The software code for performing these operations may be stored on a non-transitory computer-readable storage medium (e.g., a tangible computer-readable storage medium) stored on one or more of the components of device 10 (e.g., a computer-readable storage medium). Figure 1The software code may sometimes be referred to as software, data, instructions, program instructions, or code. Non-transitory computer-readable storage media may include a drive, non-volatile memory such as non-volatile random access memory (NVRAM), a removable flash drive or other removable media, other types of random access memory, etc. The software stored on the non-transitory computer-readable storage medium may be executed by processing circuitry on one or more of the components of the device 10 (e.g., one or more processors in the control circuitry 20). The processing circuitry may include a microprocessor, an application processor, a digital signal processor, a central processing unit (CPU), an application-specific integrated circuit having a processing circuit, or other processing circuitry.

[0086] There are many different types of electronic systems that can enable a person to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers.

[0087] According to one embodiment, a method of operating an electronic device having one or more image sensors and one or more displays is provided, the method comprising: capturing a video feed using the one or more image sensors; performing color correction on the captured video feed based on a brightness level calculated from the captured video feed to generate a corresponding color-corrected video feed adapted for immersive viewing conditions; and outputting the color-corrected video feed using the one or more displays of the electronic device.

[0088] According to another embodiment, the method includes calculating a brightness level using a calibrated linear camera signal in an electronic device.

[0089] According to another embodiment, the method includes converting the calibrated linear camera signal into a brightness level using a linear regression model.

[0090] According to another embodiment, performing color correction includes generating a color-corrected video feed using a first chromatic adaptation model adapted to immersive viewing conditions.

[0091] According to another embodiment, the first chromatic adaptation model is a function of the peak brightness level of the one or more displays.

[0092] According to another embodiment, the method includes recording a version of the captured video feed, and sending the recorded version of the captured video feed to an external electronic device, the external electronic device being configured to display the recorded version of the captured video feed that has been color corrected using a second chromatic adaptation model adapted for non-immersive viewing conditions.

[0093] According to another embodiment, a first chromatic adaptation model adapted to immersive viewing conditions exhibits a first degree of color adaptation when the calculated brightness level is at a first brightness level, and exhibits a second degree of color adaptation when the calculated brightness level is at a second brightness level greater than the first brightness level, and a second chromatic adaptation model adapted to non-immersive viewing conditions exhibits a third degree of color adaptation greater than the first degree of color adaptation when the calculated brightness level is at the first brightness level, and exhibits a fourth degree of color adaptation greater than the second degree of color adaptation when the calculated brightness level of the captured video feed is at a second brightness level.

[0094] According to another embodiment, the method includes receiving content from an external electronic device and using the one or more displays to output the received content that has been color corrected using a first chromatic adaptation model adapted to immersive viewing conditions.

[0095] According to another embodiment, the method includes: rendering virtual content; performing color correction on the virtual content using a first chromatic adaptation model adapted to immersive viewing conditions to generate corresponding color-corrected virtual content; merging the color-corrected video feed with the color-corrected virtual content to create a merged video feed; and outputting the merged video feed for immersive viewing using the one or more displays.

[0096] According to another embodiment, the method includes: rendering virtual content; modifying a captured video feed using an inverted version of a first chromatic adaptation model to generate a corresponding modified video feed; merging the modified video feed with the virtual content to create a merged video feed; and outputting the merged video feed for immersive viewing using the one or more displays.

[0097] According to another embodiment, the method includes performing color correction on the captured video feed based on the color temperature or illuminant type of the lighting in the captured video feed.

[0098] According to one embodiment, a method of operating an electronic device having one or more image sensors and one or more displays is provided, the method comprising: capturing a video feed using the one or more image sensors; performing color correction on the captured video feed based on the color temperature or light source type of the illumination in the captured video feed to generate a corresponding color-corrected video feed adapted for immersive viewing conditions; and outputting the color-corrected video feed using the one or more displays of the electronic device.

[0099] According to another embodiment, the method includes performing color correction on the captured video feed based on a brightness level calculated from the captured video feed.

[0100] According to another embodiment, performing color correction includes generating a color-corrected video feed using a first chromatic adaptation model adapted for immersive viewing conditions, the method comprising: recording a captured version of the video feed, and transmitting the recorded version of the captured video feed to an external electronic device, the external electronic device being configured to display the recorded version of the captured video feed that has been color corrected using a second chromatic adaptation model adapted for non-immersive viewing conditions.

[0101] According to another embodiment, a first chromatic adaptation model adapted to immersive viewing conditions exhibits a first degree of chromatic adaptation when the color temperature of the captured video feed is at a first value, and exhibits a second degree of chromatic adaptation when the color temperature of the captured video feed is at a second value greater than the first value, and a second chromatic adaptation model adapted to non-immersive viewing conditions exhibits a third degree of chromatic adaptation greater than the first degree of chromatic adaptation when the color temperature of the captured video feed is at the first value, and exhibits a fourth degree of chromatic adaptation greater than the second degree of chromatic adaptation when the color temperature of the captured video feed is at the second value.

[0102] According to another embodiment, the method includes receiving content from an external electronic device and using the one or more displays to output the received content that has been color corrected using a chromatic adaptation model adapted to immersive viewing conditions.

[0103] According to one embodiment, a method of operating an electronic device is provided, the method comprising: capturing a video feed; rendering the video feed using a chromaticity adaptation matrix for immersive viewing; modifying the chromaticity adaptation matrix; and rendering the video feed using the modified chromaticity adaptation matrix for non-immersive viewing.

[0104] According to another embodiment, the method includes generating a chromatic adaptation matrix based on luminance levels calculated from the captured video feed.

[0105] According to another embodiment, the method includes generating a chromatic adaptation matrix based on the color temperature of ambient light in the captured video feed.

[0106] According to another embodiment, the method includes detecting a type of illuminant in the captured video feed, and generating a chromatic adaptation matrix based on the detected type of illuminant in the captured video feed.

[0107] According to another embodiment, the chromaticity adaptation matrix is ​​a function of the peak brightness level of one or more displays in the electronic device.

[0108] According to one embodiment, a method is provided, the method comprising: capturing a video feed; rendering the captured video feed using a chromatic adaptation matrix for non-immersive viewing; modifying the chromatic adaptation matrix; and rendering the captured video feed using the modified chromatic adaptation matrix for immersive viewing.

[0109] The foregoing is merely illustrative and various modifications may be made to the embodiments described. The foregoing embodiments may be implemented independently or in any combination.

Claims

1. A method of operating an electronic device having one or more image sensors and one or more displays, the method include: capturing a video feed using the one or more image sensors; performing color correction on the captured video feed based on a brightness level calculated from the captured video feed to generate a corresponding color-corrected video feed adapted for immersive viewing conditions; as well as The color corrected video feed is output using the one or more displays of the electronic device.

2. The method according to claim 1, further comprising: include: The brightness level is calculated in the electronic device using a calibrated linear camera signal.

3. The method according to claim 2, further comprising: include: The calibrated linear camera signal is converted to the brightness level using a linear regression model. 4 . The method of claim 1 , wherein performing the color correction comprises generating the color-corrected video feed using a first chromatic adaptation model adapted to the immersive viewing condition.

5. The method of claim 4, wherein the first chromatic adaptation model is a function of a peak brightness level of the one or more displays.

6. The method according to claim 4, further comprising: include: recording a version of the captured video feed; as well as The recorded version of the captured video feed is sent to an external electronic device, wherein the external electronic device is configured to display the recorded version of the captured video feed that has been color corrected using a second chromatic adaptation model adapted for non-immersive viewing conditions.

7. The method according to claim 6, in: the first chromatic adaptation model adapted to the immersive viewing condition exhibits a first degree of chromatic adaptation when the calculated luminance level is at a first luminance level and exhibits a second degree of chromatic adaptation when the calculated luminance level is at a second luminance level greater than the first luminance level; and The second chromatic adaptation model adapted to the non-immersive viewing condition exhibits a third degree of chromatic adaptation greater than the first degree of chromatic adaptation when the calculated luminance level of the captured video feed is at the first luminance level, and exhibits a fourth degree of chromatic adaptation greater than the second degree of chromatic adaptation when the calculated luminance level is at the second luminance level.

8. The method according to claim 4, further comprising: include: receiving content from an external electronic device; as well as The received content is output using the one or more displays that has been color corrected using the first chromatic adaptation model adapted to the immersive viewing condition.

9. The method according to claim 1, further comprising: include: Render virtual content; performing color correction on the virtual content using the first chromatic adaptation model adapted to the immersive viewing condition to generate corresponding color-corrected virtual content; merging the color-corrected video feed with the color-corrected virtual content to create a merged video feed; as well as The merged video feed is output using the one or more displays for immersive viewing.

10. The method according to claim 1, further comprising: include: Render virtual content; modifying a captured video feed using an inverted version of the first chromatic adaptation model to generate a corresponding modified video feed; merging the modified video feed with the virtual content to create a merged video feed; as well as The merged video feed is output using the one or more displays for immersive viewing.

11. The method according to claim 1, further comprising: include: Color correction is performed on the captured video feed based on the color temperature or illuminant type of the lighting in the captured video feed.

12. A method of operating an electronic device having one or more image sensors and one or more displays, the method include: capturing a video feed using the one or more image sensors; performing color correction on the captured video feed based on the color temperature or illuminant type of the illumination in the captured video feed to generate a corresponding color-corrected video feed adapted for immersive viewing conditions; as well as The color corrected video feed is output using the one or more displays of the electronic device.

13. The method according to claim 12, further comprising: include: Color correction is performed on the captured video feed based on a brightness level calculated from the captured video feed.

14. The method of claim 12, wherein performing the color correction comprises generating the color-corrected video feed using a first chromatic adaptation model adapted to the immersive viewing condition, the method further comprising: include: recording a version of the captured video feed; as well as The recorded version of the captured video feed is delivered to an external electronic device, wherein the external electronic device is configured to display the recorded version of the captured video feed that has been color corrected using a second chromatic adaptation model adapted for non-immersive viewing conditions.

15. The method according to claim 12, further comprising: include: receiving content from an external electronic device; as well as The received content is output using the one or more displays that has been color corrected using a chromatic adaptation model adapted to the immersive viewing condition.

16. A method of operating an electronic device, include: Capture video feed; rendering the video feed using a chromatic adaptation matrix for immersive viewing; modifying the chromaticity adaptation matrix; as well as The video feed is rendered for non-immersive viewing using the modified chromatic adaptation matrix.

17. The method according to claim 16, further comprising: include: The chromatic adaptation matrix is ​​generated based on luminance levels calculated from the captured video feed.

18. The method according to claim 16, further comprising: include: The chromatic adaptation matrix is ​​generated based on the color temperature of ambient light in the captured video feed.

19. The method according to claim 16, further comprising: include: detecting the type of light source in the captured video feed; as well as The chromatic adaptation matrix is ​​generated based on the detected illuminant type in the captured video feed.

20. A method, the method comprising: include: Capture video feed; rendering the captured video feed using a chromatic adaptation matrix for non-immersive viewing; modifying the chromaticity adaptation matrix; as well as The captured video feed is rendered for immersive viewing using the modified chromatic adaptation matrix.