Changing display rendering mode based on multiple regions

Abstract

A method includes obtaining computer-generated content associated with a first color characteristic. The computer-generated content is associated with a first region of the physical environment. The method includes detecting a second color characteristic associated with a second region of the physical environment, the second region being different from the first region of the physical environment. The method includes, when the electronic device is in a first display rendering mode including the first color characteristic and the second color characteristic, determining that a display rendering mode change condition is satisfied, and correspondingly changing the electronic device from the first display rendering mode to a second display rendering mode. Changing to the second display rendering mode includes changing the first region from the first color characteristic to a third color characteristic, or changing the second region from the second color characteristic to a fourth color characteristic different from the third color characteristic.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 342,279, filed on May 16, 2022, which is incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates to display rendering modes, and in particular to changing display rendering modes. Background Art

[0004] In various cases, a device displays an environment that includes physical (e.g., real-world) elements and computer-generated elements. Each of the physical elements and the computer-generated elements is associated with a corresponding color characteristic, such as chromaticity or brightness. The corresponding color characteristics are often different from each other, thereby preventing an immersive user experience or otherwise degrading the user experience. However, previously available display technologies do not take into account the corresponding color characteristics. Summary of the invention

[0005] According to some specific implementations, a method is performed at an electronic device including one or more processors, a non-volatile memory, and a display. The method includes obtaining computer-generated content associated with a first color characteristic. The computer-generated content is associated with a first area of ​​a physical environment. The method includes detecting a second color characteristic associated with a second area of ​​the physical environment, the second area being different from the first area of ​​the physical environment. The method includes: when the electronic device is in a first display rendering mode including a first color characteristic and a second color characteristic, determining that a display rendering mode change condition is satisfied. In response to determining that the display rendering mode change condition is satisfied, the method includes changing the electronic device from the first display rendering mode to the second display rendering mode. Changing to the second display rendering mode includes changing the first area from the first color characteristic to the third color characteristic on the display, or changing the second area from the second color characteristic to a fourth color characteristic different from the third color characteristic on the display.

[0006] According to some specific implementations, a method is performed at an electronic device including one or more processors, a non-volatile memory, and a display. One or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the operation of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of an electronic device, cause the device to perform or cause the operation of any of the methods described herein. According to some specific implementations, an electronic device includes a device for performing or causing the operation of any of the methods described herein. According to some specific implementations, an information processing device for use in an electronic device includes a device for performing or causing the operation of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] For a better understanding of the various described implementations, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the drawings.

[0008] Figures 1A to 1E is a first example of changing the display rendering mode according to some specific implementations.

[0009] FIG. 2A to FIG. 2D is a second example of changing the display rendering mode according to some specific implementations.

[0010] FIG. 3A to FIG. 3C is a third example of changing a display rendering mode according to some specific implementations.

[0011] Figure 4 is a block diagram of an example of a portable multifunction device according to some implementations.

[0012] Figure 5 is an example of a flowchart of a method of changing a display rendering mode according to some specific implementations. DETAILED DESCRIPTION

[0013] In various cases, the device may concurrently display physical (e.g., real-world) elements and computer-generated elements. For example, in augmented reality (AR), the device displays a computer-generated lamp so that it appears to be placed on a physical table. Each of the physical elements and the computer-generated elements is associated with a corresponding color characteristic such as chromaticity, brightness, or white point. The corresponding color characteristics of the physical elements and the computer-generated elements are usually perceptually conflicting with each other, resulting in a reduced user experience. For example, when the ambient light from the physical environment is relatively bright (e.g., high brightness), the viewing quality of the video stream decreases due to the brightness of the ambient light. For another example, the ambient light from the physical environment has a color characteristic that reduces the user experience. However, previously available technologies do not effectively adjust the corresponding color characteristics to take into account the brightness or color characteristics of the ambient light. For another example, the movement of the device within the physical environment causes the color characteristics of the ambient light and / or the visibility of the computer-generated elements to change. However, previously available display technologies do not effectively adjust the corresponding color characteristics, and therefore cannot provide an immersive user experience.

[0014] In contrast, various embodiments disclosed herein include methods, systems, and devices for changing a display rendering mode based on multiple regions of a physical environment. To this end, a method includes obtaining computer-generated content associated with a first color characteristic. The computer-generated content is associated with a first region of the physical environment. The method includes detecting a second color characteristic associated with a second region of the physical environment, the second region being different from the first region. For example, the second region surrounds (e.g., surrounds) the first region.

[0015] In addition, the method includes: when the electronic device is in a first display rendering mode including a first color characteristic and a second color characteristic, determining that a display rendering mode change condition is satisfied. For example, the display rendering mode change condition is satisfied based on a user input requesting display of computer-generated content. For another example, the display rendering mode change condition is satisfied based on a change in the positioning of the electronic device that causes the field of view of the electronic device to include the first area and the second area. In response to determining that the display rendering mode change condition is satisfied, the method includes changing the electronic device from the first display rendering mode to the second display rendering mode.

[0016] For example, in a case where the computer-generated content should be integrated in a physical environment, changing to the second display rendering mode includes changing the first area from a first color characteristic to a third color characteristic, wherein the third color characteristic is at least partially based on the second color characteristic. For example, the computer-generated content is color adjusted to blend with the color characteristics of the surrounding physical environment.

[0017] For another example, in a case where the first color characteristic of the computer-generated content should be maintained, changing to the second display rendering mode includes changing the second region from the second color characteristic to a fourth color characteristic. The fourth color characteristic is different from the third color characteristic. For example, the area surrounding the first region is color-adjusted to have a neutral tone, thereby reducing the user's distraction from the computer-generated content (displayed at the first region).

[0018] Reference will now be made in detail to implementations, embodiments of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described implementations. However, it will be apparent to one of ordinary skill in the art that the various described implementations can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure various aspects of the implementation.

[0019] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described specific implementations. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0020] The terms used in the description of various embodiments described herein are only for the purpose of describing specific embodiments, and are not intended to be limiting. As used in the description of various embodiments described and in the appended claims, the singular forms "one", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "including" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their groupings.

[0021] As used herein, the term "if" is optionally interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0022] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, the extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system can detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, the XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting the XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), the XR system may adjust characteristics of the graphical content in the XR environment in response to representations of physical movement (e.g., voice commands).

[0023] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some specific implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0024] Figures 1A to 1E is a first example of changing the display rendering mode according to some specific implementations.

[0025] According to various specific implementations disclosed herein, the electronic device 120 obtains computer-generated content associated with a first color characteristic. For example, the specific color characteristic indicates chromaticity, chroma, brightness, white point (e.g., D65), etc. Figure 1A , computer-generated content 100 includes a text content list within a user interface (e.g., an application window of a note-taking application). The text content list includes a first text 101 ("- Reminder to buy eggs and milk"), a second text 102 ("- Reminder to drink more water"), and a third text 103 ("- Reminder to exercise more"). The user interface has a white background to make the text content easily visible. Therefore, in this example, the first color characteristic 104 is white.

[0026] The computer-generated content 100 is associated with a first area of ​​the physical environment. For example, the first area corresponds to a display location of the computer-generated content within the physical environment. Figure 1B As shown, the computer-generated content 100 is associated with a first area 106. The first area 106 corresponds to a rectangular portion of a physical wall 112 of the physical environment 110. The first area 106 can be specified in various ways, such as via user input or independently of user input, for example, a default location of the first area 106 is near the middle of the display of the electronic device 120.

[0027] like Figure 1B As further shown, the physical environment 110 also includes a physical side cabinet 114 disposed along the physical wall 112. The physical side cabinet 114 is associated with a second color characteristic 116 corresponding to the color red. Figures 1B to 1E As shown, the red color is indicated by the surface of the physical sideboard 114 having a first shaded pattern. Thus, in this example, the first color characteristic 104 (white) is different from the second color characteristic 116 (red).

[0028] like Figure 1C As shown, user 50 holds electronic device 120 in physical environment 110. For example, electronic device 120 corresponds to a mobile device such as a smart phone, tablet computer, or the like.

[0029] In some implementations, the electronic device 120 corresponds to a head-mounted device (HMD) that includes an integrated display (e.g., a built-in display). In some implementations, the electronic device 120 includes a head-mounted housing. In various implementations, the head-mounted housing includes an attachment area to which another device with a display can be attached. In various implementations, the head-mounted housing is shaped to form a container for receiving another device (e.g., the electronic device 120) that includes a display. For example, in some implementations, the electronic device 120 slides / snaps into the head-mounted housing or is otherwise attached to the head-mounted housing.

[0030] The electronic device 120 includes a display 122. The display 122 is associated with a viewable area 124 that includes a portion of the physical environment 110. The portion of the physical environment 110 includes a physical sideboard 114 and a physical wall 112 that includes the first area 106.

[0031] like Figure 1C As further shown, in some implementations, the electronic device 120 determines a second region 132 of the physical environment 110. The second region 132 is different from (eg, outside) the first region 106. Figure 1CAs shown, the second area 132 may correspond to the surface of the physical side cabinet 114. Determining the second area 132 may include identifying a physical object within the physical environment 110 via computer vision, and setting the second area 132 to the location of the physical object. To this end, in some embodiments, the electronic device determines the second area 132 based on image data of the physical environment 110. For example, the electronic device 120 includes an image sensor that captures image data (e.g., one or more images) of the physical environment 110. The image sensor may be associated with a field of view (FOV) that approximates the visible area 124. In some embodiments, the electronic device 120 identifies the surface of the physical side cabinet 114 because the surface of the physical side cabinet 114 meets a proximity threshold relative to the first area 106 (e.g., less than a threshold distance from the first area). In some embodiments, the electronic device 120 identifies a subset of pixels of the image data corresponding to the surface of the physical side cabinet 114. In some embodiments, the electronic device 120 synthesizes the image data with various computer-generated content to generate XR data representing the XR environment.

[0032] Additionally, the electronic device 120 detects the second color characteristic 116 associated with the second region 132. Continuing with the previous example, the electronic device 120 determines the second color characteristic 116 based on corresponding pixel values ​​associated with the subset of pixels, wherein the corresponding pixel values ​​are indicative of the second color characteristic 116.

[0033] like Figure 1D As shown, when the electronic device 120 is in a first display rendering mode, the electronic device 120 (e.g., concurrently) displays computer-generated content 100 having a first color characteristic 104 at a portion of the display 122 corresponding to a first area 106 and displays a second color characteristic 116 at a portion of the display 122 corresponding to a second area 132.

[0034] According to various specific implementations, when the electronic device 120 is in a first display rendering mode including a first color characteristic and a second color characteristic, the electronic device 120 determines that a display rendering mode change condition is satisfied. For example, in some specific implementations, the electronic device 120 determines that a display rendering mode change condition is satisfied by detecting (e.g., via computer vision) that the first area 106 and the second area 132 are within the visible area 124. For another example, in some specific implementations, the electronic device 120 determines that a display rendering mode change condition is satisfied by detecting a display request to display computer-generated content 100. In some specific implementations, the electronic device 120 receives a display request from the user 50 via user input. For example, the electronic device 120 performs eye tracking or limb tracking of the user 50, and accordingly determines that the user's gaze or limbs point to a display request indication (e.g., a displayed menu button).

[0035] In response to determining that the display rendering mode change condition is satisfied, the electronic device 120 changes from the first display rendering mode to the second display rendering mode. Changing to the second display rendering mode includes: changing the first area 106 on the display 122 from the first color characteristic 104 to the third color characteristic, and / or changing the second area 132 on the display 122 from the second color characteristic 116 to the fourth color characteristic. The fourth color characteristic is different from the third color characteristic. For example, Figure 1E As shown, the electronic device 120 changes a portion of the first region 106 from the first color characteristic 104 to the third color characteristic 142. That is, in some implementations, the electronic device 140 modifies the computer-generated content 100 so that at least a portion of the modified computer-generated content 140 has the third color characteristic 142. The third color characteristic 142 is Figure 1E . The third color characteristic 142 may be based on a color mix of the first color characteristic 104 (white) and the second color characteristic 116 (red). For example, the third color characteristic 142 is pink. As another example, changing the second region 132 from the second color characteristic 116 to the fourth color characteristic includes increasing the brightness of the second region 132 based on the white color (first color characteristic 104) of the computer-generated content 100. For example, the electronic device 120 renders image data of the second region 132 so as to increase the brightness of the second region 132. In some specific implementations, the increase in brightness of the second region 132 is based on a gradient function. For example, the amount of increase in brightness of the second region 132 is based on a gradient function, wherein the gradient function is based on a distance from the first region 106 (e.g., a portion of the second region 132 that is closer to the first region 106 has a greater increase in brightness).

[0036] In some implementations, a portion of the computer-generated content 100 is modified. Figure 1EAs shown, the bottom portion of the user interface is colored with the third color characteristic 142, while the upper portion of the user interface maintains the third color characteristic 104 (white). In this example, the bottom portion of the user interface is adjusted from white to pink because the bottom portion is close enough to the surface of the physical side cabinet 114. In addition, the upper portion of the user interface is maintained at the first color characteristic 104 (white) because the upper portion is not close enough to the surface of the physical side cabinet 114. In some specific implementations, the electronic device 120 may perform weighted (e.g., gradient-based) color adjustment based on the proximity to the second area 132 to determine the third color characteristic 142. For example, the gradient-based color adjustment causes the modified computer-generated content 140 to include dark pink (closest to the physical side cabinet 114), medium pink, and light pink (farthest from the physical side cabinet 114). In some specific implementations, color adjustment of the first area 106 includes changing the hue or brightness of the first area 106. Those of ordinary skill in the art will appreciate that in some specific implementations, the entire first area 106 is color adjusted.

[0037] Thus, in some implementations, the electronic device 120 performs color adjustments on the computer-generated content while maintaining the color of a portion of the physical environment 110 (e.g., the portion surrounding the first area 106). Incorporating the color characteristics of the physical environment into the displayed computer-generated content provides a more immersive user experience to the user, while maintaining the color characteristics of the physical environment avoids an undesirable intrusive user experience.

[0038] FIG. 2A to FIG. 2D is a second example of changing the display rendering mode according to some specific implementations. FIG. 2A to FIG. 2D The second example described also operates within a physical environment 110 that includes a physical wall 112 and a red physical sideboard 114 .

[0039] The electronic device 120 obtains computer-generated content corresponding to the drawing application user interface (UI) 204. Figure 2A As shown, the drawing application UI 204 is associated with the third area 200 of the physical environment 110 .

[0040] like Figure 2B As shown, when the electronic device 120 is in the first display rendering mode, the electronic device 120 displays a drawing application UI 204 in the third area 200. The drawing application UI 204 is associated with a fifth color characteristic 206 including a white drawing canvas (the area above the horizontal dashed line).

[0041] According to various implementations, when the electronic device 120 is in the first display rendering mode, the electronic device 120 determines a user engagement score with respect to the drawing application UI 204. The user engagement score characterizes a level of user engagement with respect to the drawing application UI 204. The user engagement can be based on whether the user 50 is focused on the drawing application UI 204, and optionally, how long the focus is directed to the drawing application UI 204.

[0042] In some implementations, the user engagement score is based on the gaze of one or more eyes of the user 50. To this end, in some implementations, the electronic device 120 includes an eye tracker that tracks the gaze of the eyes. For example, the user engagement score increases as the gaze moves closer to the drawing application UI 204. For another example, the longer the gaze is directed into the drawing application UI 204, the higher the user engagement score.

[0043] In some implementations, the user engagement score is based on the limbs (e.g., hands or fingers) of the user 50. To this end, in some implementations, the electronic device 120 includes a limb tracker that tracks the limbs of the user 50. For example, when the limbs of the user 50 interact with the drawing application UI 204, such as when the limbs select a particular drawing tool affordance (e.g., a pencil) or when the limbs move within the drawing canvas to perform a drawing operation, the user engagement score increases. For another example, the longer the limbs are pointed within the drawing application UI 204, the higher the user engagement score.

[0044] As an example, Figure 2C As shown, the graticule 210 indicates that the user's focus (e.g., gaze or limbs) is directed to the drawing application UI 204. Therefore, the electronic device 120 increases the high user engagement score. In some specific implementations, the electronic device 120 determines that the display rendering mode change condition is met by determining that the user engagement score meets (e.g., exceeds) the engagement threshold. Continuing with the previous example, because the relatively high user engagement score exceeds the engagement threshold, the electronic device 120 determines that the display rendering mode change condition is met.

[0045] Therefore, based on determining that the display rendering mode change condition is satisfied, the electronic device 120 changes from the first display rendering mode to the second display rendering mode. To this end, the electronic device 120 determines the sixth color characteristic 222 based on the fifth color characteristic 206 and the second color characteristic 116 (associated with the surface of the physical cabinet 114). For example, due to the color mixing of the red color 116 of the physical cabinet 114 and the white color of the drawing canvas, the sixth color characteristic 222 corresponds to pink. In addition, based on determining that the display rendering mode change condition is satisfied, the electronic device 120 generates a fourth computer-generated content based on the computer-generated content (drawing application UI 204) and the sixth color characteristic 222. The fourth computer-generated content corresponds to a modified drawing application UI 220, which corresponds to a color-adjusted version of the drawing application UI 204. That is, the modified drawing application UI 220 adjusts the color from the fifth color characteristic 206 (white) to the sixth color characteristic 222 (pink). As Figure 2D As shown, the electronic device 120 displays a modified drawing application UI 220 having a sixth color characteristic 222 in the third area 200 .

[0046] FIG. 3A to FIG. 3C is a third example of changing the display rendering mode according to some specific implementations. FIG. 3A to FIG. 3C The depicted example also operates within a physical environment 110 that includes a physical wall 112 and a red physical sideboard 114 .

[0047] The electronic device 120 obtains fifth computer-generated content associated with the fourth area 300 of the physical environment 110. The fourth area 300 is Figure 3A Example in. like Figure 3A As shown, the fourth area 300 is surrounded by the fifth area 302. In other words, the entire visible area 124 is composed of the fourth area 300 and the surrounding fifth area 302. The fifth computer-generated content corresponds to a video stream 310 having a seventh color characteristic 312.

[0048] In addition, the electronic device 120 detects an eighth color characteristic 314 associated with the fifth area 302. Because the fifth area 302 includes multiple physical surfaces and objects, the eighth color characteristic 314 may include multiple chromaticity values, brightness values, or white point values ​​associated with the physical environment perceived in the fifth area 302.

[0049] like Figure 3B As shown, when the electronic device 120 is in the first display rendering mode, the electronic device 120 displays a video stream 310 having a seventh color characteristic 312 at a portion of the display 122 corresponding to the fourth area 300, and displays an eighth color characteristic 314 at a portion of the display 122 corresponding to the fifth area 302.

[0050] According to various implementations disclosed herein, the electronic device 120 determines that a display rendering mode change condition is satisfied. In some implementations, in response to determining that the display rendering mode change condition is satisfied, the electronic device 120 changes from the first display rendering mode to the second display rendering mode. In some implementations and as Figure 3C As illustrated, changing from the first display rendering mode to the second display rendering mode includes changing the fifth region 302 from the eighth color characteristic 314 to the ninth color characteristic 316, while maintaining the fourth region 300 with the seventh color characteristic 312. For example, changing the fifth region 302 to the ninth color characteristic 316 includes modifying (e.g., rendering) a portion of image data of the physical environment 110 (e.g., captured via a camera), where the portion corresponds to the fifth region 302. For another example, changing the fifth region 302 to the ninth color characteristic 316 includes displaying an overlay at a portion of the display 122 corresponding to the fifth region 302.

[0051] Changing the fifth region 302 from the eighth color characteristic 314 to the ninth color characteristic 316 helps resolve the simultaneous contrast problem, where the perception of the seventh color characteristic 312 is affected by the eighth color characteristic 314, and vice versa. Thus, changing the fifth region 302 to the ninth color characteristic 316 can help create ideal conditions for the perception of the video stream 310 while reducing distractions when the user perceives the video stream 310.

[0052] In some implementations, the ninth color characteristic 316 is based on a neutral tone standard. For example, the ninth color characteristic 316 is monotonous or dull, such as grayscale. As an example, the ninth color characteristic 316 has a neutral tone relative to the seventh color characteristic 312. To this end, in some implementations, determining the ninth color characteristic 316 includes applying a neutral tone mapping to the image data of the physical environment 110. Applying the neutral tone mapping causes the ninth color characteristic 316 to have a sufficiently high contrast relative to the seventh color characteristic 312, while having a sufficiently high opacity level to enable the user 50 to view the physical environment 110. Changing the fifth area 302 to a neutral tone maintains the creator's intention (of the video stream 310) to minimize distractions, thereby helping the user 50 focus on the video stream 310. In addition, the neutral tone changes the physical environment 110 to a natural color space while maintaining the intended appearance of the video stream 310.

[0053] In some specific implementations, determining the ninth color characteristic 316 is also based on metadata associated with the fifth computer-generated content. For example, the metadata indicates a "video" application type associated with the fifth computer-generated content. Because the "video" application type is typically associated with a high level of user immersion, the electronic device 120 determines that the ninth color characteristic 316 is a non-distracting color (e.g., a neutral tone). As another example, the metadata indicates an immersion level associated with computer-generated content. Continuing with this example, the metadata indicates a relatively high immersion level associated with the fifth computer-generated content because the appropriate experience includes the user 50 being highly immersed in watching the video stream 312.

[0054] Thus, as described in the previous examples, various embodiments disclosed herein include changing a display rendering mode (e.g., a color mode). For example, some embodiments include color-adjusting computer-generated content based on color characteristics associated with a physical environment in order to integrate the computer-generated content with the physical environment. Figures 1A to 1E and FIG. 2A to FIG. 2D These embodiments may be characterized as an integrated color mode, as further described below. As another example, some embodiments include color-adjusting at least a portion of the physical environment so as to appear neutral in color relative to the displayed computer-generated content. FIG. 3A to FIG. 3C Examples of neutral color adjustments to portions of a physical environment are described. These implementations can be characterized as computer-generated color-preserving models.

[0055] As further described herein, changing the display rendering mode can be triggered by user input (e.g., a user request to play a video stream), or can be independent of user input. For example, when the computer-generated content is world-locked (e.g., anchored) to a first area of ​​the physical environment, upon detecting the first area, the electronic device performs a color adjustment on the computer-generated content. For another example, based on detecting an increase in the brightness of ambient light from the physical environment (e.g., a user turns on a light), the electronic device adjusts a portion of the physical environment so as to present a neutral color relative to the computer-generated content.

[0056] Figure 4 is a block diagram of an example of a portable multifunction device 400 (sometimes referred to herein as "electronic device 400" for brevity) according to some implementations. Figures 1A to 1E , FIG. 2A to FIG. 2D or FIG. 3A to FIG. 3C The electronic device 120 includes some or all of the components of the electronic device 400 .

[0057] The electronic device 400 includes a memory 402 (e.g., one or more non-transitory computer-readable storage media), a memory controller 422, one or more processing units (CPUs) 420, a peripheral device interface 418, an input / output (I / O) subsystem 406, a display system 412, an inertial measurement unit (IMU) 430, an image sensor 443 (e.g., a camera), a contact intensity sensor 465, and other input or control devices 416. In some implementations, the electronic device 400 corresponds to one of a mobile phone, a tablet computer, a laptop computer, a wearable computing device, a head-mountable device (HMD), a head-mountable housing (e.g., the electronic device 400 slides into or is otherwise attached to the head-mountable housing), etc. In some implementations, the head-mountable housing is shaped to form a container for receiving the electronic device 400 with a display.

[0058] In some implementations, peripherals interface 418, one or more processing units 420, and memory controller 422 are, optionally, implemented on a single chip, such as chip 403. In some other implementations, they are, optionally, implemented on separate chips.

[0059] The I / O subsystem 406 couples input / output peripherals such as a display system 412 and other input or control devices 416 on the electronic device 400 to a peripheral device interface 418. The I / O subsystem 406 optionally includes a display controller 456, an image sensor controller 458, an intensity sensor controller 459, one or more input controllers 452 for other input or control devices, and an IMU controller 432. One or more input controllers 452 receive electrical signals from / transmit electrical signals to other input or control devices 416. An example of other input or control devices 416 is an eye tracker that tracks the user's eye gaze. Another example of other input or control devices 416 is a limb tracker that tracks the user's limbs (e.g., fingers). In some specific implementations, one or more input controllers 452 are optionally coupled to (or not coupled to) any of the following: a keyboard, an infrared port, a universal serial bus (USB) port, a stylus, a finger wearable device, and / or a pointer device (such as a mouse). One or more buttons optionally include a push button. In some implementations, the other input or control device 416 includes a positioning system (e.g., GPS) that obtains information about the position and / or orientation of the electronic device 400 relative to a particular object. In some implementations, the other input or control device 416 includes a depth sensor and / or a time-of-flight sensor that obtains depth information characterizing physical objects within the physical environment. In some implementations, the other input or control device 416 includes an ambient light sensor that senses ambient light from the physical environment and outputs corresponding ambient light data.

[0060] Display system 412 provides an input interface and an output interface between electronic device 400 and a user. Display controller 456 receives electrical signals from display system 412 and / or transmits electrical signals to the display system. Display system 412 displays visual output to the user. Visual output optionally includes graphics, text, icons, videos, and any combination thereof (sometimes referred to herein as "computer-generated content"). In some specific implementations, some visual outputs or all visual outputs correspond to user interface objects. As used herein, the term "affordance" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0061] The display system 412 may have a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. The display system 412 and display controller 456 (together with any associated modules and / or instruction sets in memory 402) detect contact (and any movement or interruption of the contact) on the display system 412 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the display system 412.

[0062] The display system 412 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other specific implementations. The display system 412 and display controller 456 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the display system 412 to detect contact and any movement or interruption thereof. The display system 412 can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies.

[0063] The user optionally uses any suitable object or appendage such as a stylus, a finger wearable device, a finger, etc. to contact the display system 412. In some implementations, the user interface is designed to work with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some implementations, the electronic device 400 converts the rough finger-based input into a precise pointer / cursor position or command for performing the action desired by the user.

[0064] The inertial measurement unit (IMU) 430 includes an accelerometer, a gyroscope, and / or a magnetometer to measure various forces, angular rates, and / or magnetic field information relative to the electronic device 400. Therefore, according to various specific implementations, the IMU 430 detects one or more position change inputs of the electronic device 400, such as the electronic device 400 being shaken, rotated, moved in a specific direction, etc.

[0065] The image sensor 443 captures still images and / or video. In some implementations, the image sensor 443 is located on the back of the electronic device 400, opposite the touch screen on the front of the electronic device 400, so that the touch screen can be used as a viewfinder for still image and / or video image acquisition. In some implementations, another image sensor 443 is located on the front of the electronic device 400, so that an image of the user is obtained (e.g., for selfies, for video conferencing when the user views other video conference participants on the touch screen, etc.). In some implementations, the image sensor is integrated into the HMD. For example, the image sensor 443 outputs image data representing a physical object (e.g., a physical agent) within the physical environment.

[0066] The contact force sensor 465 detects the intensity of a contact on the electronic device 400 (e.g., a touch input on a touch-sensitive surface of the electronic device 400). The contact force sensor 465 is coupled to the force sensor controller 459 in the I / O subsystem 406. The contact force sensor 465 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). The contact force sensor 465 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the physical environment. In some implementations, at least one contact force sensor 465 is juxtaposed or adjacent to a touch-sensitive surface of the electronic device 400. In some implementations, at least one contact force sensor 465 is located on a side of the electronic device 400.

[0067] Figure 5 is an example of a flowchart of a method 500 for changing a display rendering mode according to some specific implementations. In various specific implementations, the method 500 or a portion thereof is performed by an electronic device (e.g., Figures 1A to 1E , FIG. 2A to FIG. 2D and / or FIG. 3A to FIG. 3C In various implementations, method 500 or a portion thereof is performed by a mobile device (such as a smartphone, a tablet computer, or a wearable device). In various implementations, method 500 or a portion thereof is performed by a head-mounted device (HMD) including a display. In some implementations, method 500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 500 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0068] As represented by box 502, method 500 includes obtaining computer-generated content associated with a first color characteristic. For example, the first color characteristic indicates chromaticity, saturation, brightness, white point, etc. The computer-generated content is associated with a first area of ​​the physical environment. For example, the computer-generated content is displayed at a portion of the display corresponding to the first area. The computer-generated content can be world-locked to the first area, so that a change in the positioning of the electronic device causes a corresponding change in the display position of the computer-generated content. The computer-generated content can correspond to any type of content, such as a user interface (UI), text content, a web page, a static image, a video stream, a two-dimensional (2D) object, a three-dimensional (3D) object, etc. As an example, refer to Figure 1A and Figure 1D , the computer-generated content 100 is associated with the first area 106 and is therefore displayed within the first area. Figure 2B , the computer-generated content corresponds to the drawing application UI 204 displayed in the third area 200 of the physical environment 110. In some implementations, the method 500 includes obtaining the computer-generated content from a non-transitory memory (e.g., cache or random access memory (RAM)) of an electronic device executing the method 500. In some implementations, the method 500 includes obtaining the computer-generated content from an external device / system via a communication interface (e.g., Wi-Fi or Bluetooth). In some implementations, the electronic device executing the method 500 renders (e.g., via a graphics processing unit (GPU)) an object to generate the computer-generated content.

[0069] As represented by block 504, method 500 includes detecting a second color characteristic associated with a second region of the physical environment. For example, the second color characteristic indicates chroma, chroma, brightness, white point, etc. The first region is different from the second region. For example, in some implementations, detecting the second color characteristic includes identifying a pixel value associated with a pixel of the image of the physical environment, where the pixel corresponds to the second region. In the previous example, the pixel value indicates the second color characteristic.

[0070] In some implementations, method 500 includes setting the second region to be the location of the physical object within the physical environment. To this end, in some implementations, the electronic device executing method 500 includes an image sensor that captures a first image of the physical environment, and method 500 includes determining the second region by identifying the physical object represented within the first image via computer vision. For example, referring to Figure 1B, the electronic device 120 captures an image of the physical environment 110, and identifies the physical cabinet 114 (and / or the surface of the physical cabinet 114) within the image. In some implementations, identifying the physical object includes determining that the physical object satisfies a proximity threshold relative to the first region. In some implementations, detecting the second color characteristic includes: identifying a subset of pixels corresponding to the physical object within the image, and identifying pixel values ​​respectively associated with the subset of pixels, wherein the pixel values ​​indicate the second color characteristic.

[0071] As represented by block 506, in some implementations, the first region is positioned in a non-overlapping arrangement relative to the second region. Figure 3A , the fourth area 300 does not overlap with the fifth area 302.

[0072] As represented by block 508, in some implementations, the first region is adjacent to the second region. Figure 2A , the third area 200 (associated with the drawing application UI 204) is adjacent to the second area 132 (associated with the surface of the physical cabinet 114). In some implementations, the user's field of view is formed by the first area and the second area, wherein the second area is the remaining area of ​​the user's field of view outside the first area. In some implementations, the second area surrounds (e.g., surrounds) the first area.

[0073] As represented by box 510, when the electronic device is in a first display rendering mode including a first color characteristic and a second color characteristic, method 500 includes determining that a display rendering mode change condition is satisfied. In some specific implementations, when the electronic device is in the first display rendering mode, method 500 includes rendering computer-generated content using the first color characteristic. In some specific implementations, when the electronic device is in the first display rendering mode, method 500 includes rendering image data of a physical environment using the second color characteristic, wherein the image data represents the second area. In addition, in some specific implementations, when in the first display rendering mode, method 500 includes: displaying computer-generated content having the first color characteristic at a portion of the display corresponding to the first area, and / or displaying the second color characteristic at a portion of the display corresponding to the second area.

[0074] As represented by block 512, in some implementations, determining that a display rendering mode change condition is satisfied includes detecting a first region and / or a second region within a visible region associated with a display of the electronic device. Figure 1C Based on detecting the first area 106 and the second area 132, the electronic device 120 determines that the display rendering mode change condition is met. In some specific implementations, detecting the first area and / or the second area includes performing computer vision, such as instance segmentation or semantic segmentation, with respect to image data of the physical environment.

[0075] In some implementations, determining that the display rendering mode change condition is satisfied is based on metadata associated with the computer-generated content. For example, the metadata indicates an immersion level that indicates an appropriate user immersion level associated with the computer-generated content. As an example, referring to Figure 3B , the electronic device 120 obtains metadata associated with the video stream 310, wherein the metadata indicates a relatively high immersion level. Continuing with this example, because the relatively high immersion level is above the threshold level, the electronic device 120 determines that the display rendering mode change condition is satisfied, and thus initiates a display rendering mode change, such as Figure 3C The metadata may also indicate the content type of the computer-generated content. Therefore, in some implementations, because the computer-generated content is a movie, a work of art, a picture, etc., the electronic device 120 may determine that the display rendering mode change condition is met. For example, for computer-generated content of the "movie" content type, the electronic device 210 changes to a dark display rendering mode in which portions of the physical environment surrounding the computer-generated content are darkened to improve user immersion relative to the computer-generated content.

[0076] In some implementations, the computer-generated content is generated by an application, and the application transmits a request to change the display rendering mode (e.g., via an application programming interface (API)). Receipt of the request can cause the display rendering mode change condition to be satisfied. In some implementations, the application transmits an indication of a content type of the computer-generated content (e.g., movie content), and determining that the display rendering mode change condition is satisfied includes determining that the content type is a predetermined content type.

[0077] As represented by block 514 , in various implementations, determining that a display rendering mode change condition is satisfied is based on user input.

[0078] For example, as represented by block 516, determining that the display rendering mode change condition is satisfied includes: detecting a display request to display computer-generated content. Figure 3A and Figure 3B , the electronic device 120 detects a user input requesting to play the video stream 310, such as the user 50 selecting a thumbnail of the video stream 310. For another example, detecting the display request includes: receiving a request to launch an application associated with the computer-generated content. For example, referring to Figure 2A and Figure 2B, electronic device 120 receives a request from user 50 to open a drawing application associated with drawing application UI 204. In some implementations, detecting the display request includes: receiving user input that selects to initiate display of an affordance. Examples of user input include pointing to a limb (e.g., a finger) of a user that initiates display of an affordance, or pointing to an eye gaze of a user that initiates display of an affordance. In some implementations, detecting the display request is independent of receiving user input, such as when the electronic device executing method 500 receives a display request from another device or from a system via an integrated communication interface (e.g., a Wi-Fi interface of a Bluetooth interface).

[0079] As another example, as represented by box 518, determining that the display rendering mode change condition is satisfied is based on a change in the positioning of the electronic device executing method 500. For example, the electronic device includes an input device corresponding to a positioning sensor (e.g., an inertial measurement unit (IMU)) that detects a change in the positioning of the electronic device. As an example, the positioning change corresponds to a user input that rotates the electronic device. Therefore, based on detecting the positioning change, the electronic device detects a first area and / or a second area within a visible area associated with a display of the electronic device, such as described with reference to box 512. Therefore, the electronic device determines that the display rendering mode change condition is satisfied.

[0080] As yet another example, as represented by block 520, the user input corresponds to a color change request. To this end, method 500 includes detecting a user input requesting a color change. Determining that a display rendering mode change condition is satisfied includes detecting a user input requesting a color change. An example of a color change request is a gaze input or a limb input directed to a color change affordance, such as an increase color contrast affordance, an increase user immersion affordance, and the like.

[0081] As yet another example, as represented by block 522, determining that the display rendering mode change condition is satisfied includes determining that the engagement score satisfies the engagement threshold. The engagement score characterizes a user engagement level with respect to the computer-generated content. To this end, method 500 includes: when displaying computer-generated content having a first color characteristic at a location corresponding to the first area on the display, determining a user engagement score. For example, referring to Figure 2C , the electronic device 120 determines a relatively high engagement score based on the user 50 focusing on the drawing application UI 204, as indicated by the graticule 210. The engagement score may be based on the user's eye gaze relative to the computer-generated content, the position of the user's limbs relative to the computer-generated content, and / or the duration of gaze / limb pointing toward the computer-generated content.

[0082] As yet another example, in some implementations, determining whether the display rendering mode change condition is satisfied is based on a ratio of a size of the first area to a size of a visible area of ​​the display. For example, when the ratio is above a threshold, the display rendering mode change condition is satisfied. Figure 3A , the size of the visible area 124 corresponds to the combined size of the fourth area 300 and the fifth area 302. Continuing with this example, the electronic device 120 may determine that the size of the fourth area 300 is a relatively large percentage of the size of the visible area 124, and accordingly determine that the display rendering mode change condition is met.

[0083] As represented by block 524, in response to determining that the display rendering mode change condition is satisfied, method 500 includes changing the electronic device from the first display rendering mode to a second display rendering mode. In some implementations, the second display rendering mode corresponds to an integrated color mode, such as reference color mode. Figures 1A to 1E and FIG. 2A to FIG. 2D In some implementations, the second display rendering mode corresponds to a color-preserving mode for computer-generated content, such as FIG. 3A to FIG. 3C In some implementations, when the electronic device is in the second display rendering mode, method 500 includes rendering the computer-generated content using the third color characteristic (as described in reference box 526). In some implementations, when the electronic device is in the second display rendering mode, method 500 includes rendering image data of the physical environment using the fourth color characteristic (as described in reference box 532), wherein the image data represents the second area. In addition, in some implementations, when in the second display rendering mode, method 500 includes: displaying the computer-generated content having the third color characteristic at a portion of the display corresponding to the first area, and / or displaying the fourth color characteristic at a portion of the display corresponding to the second area.

[0084] In some implementations, changing the electronic device from the first display rendering mode to the second display rendering mode includes changing a display characteristic (e.g., a hardware characteristic) associated with the display. For example, when the electronic device displays a video on a first set of display pixels, method 500 includes dimming a second set of display pixels surrounding the video (e.g., setting the second set of display pixels to a neutral tone) to reduce distraction when a user is viewing the video on the display.

[0085] As represented by block 526, in some implementations, changing the electronic device from the first display rendering mode to the second display rendering mode includes: changing the first area on the display of the electronic device from the first color characteristic to a third color characteristic. For example, when in the second display rendering mode, method 500 includes: displaying computer-generated content having the third color characteristic at a portion of the display corresponding to the first area. As represented by block 528, in some implementations, the third color characteristic is based on the first color characteristic and the second color characteristic. For example, referring to Figure 1D and Figure 1E , the electronic device 120 changes the bottom portion of the computer-generated content 100 from the first color characteristic 104 (white) to the third color characteristic 142 (pink) based on color mixing the first color characteristic 104 (white) with the second color characteristic 116 (red). In some implementations, the third color characteristic is coordinated with respect to the first color characteristic and the second color characteristic. For example, the third color characteristic is determined by applying a color appearance model to the first color characteristic and the second color characteristic. Examples of color coordination include sufficient chromaticity contrast, sufficient luminance contrast, or a combination thereof.

[0086] As represented by block 530, in some implementations, the third color characteristic is also based on metadata associated with the computer-generated content. For example, the metadata indicates a type of application associated with the computer-generated content, or the metadata indicates an immersion level associated with the computer-generated content. For example, referring to FIG. 2A to FIG. 2D , the electronic device 120 obtains metadata indicating a drawing application type associated with the drawing application UI 204. The drawing application type typically provides a drawing canvas and a tool area including a collection of drawing affordances (e.g., pencils, pens, etc.). The drawing canvas is typically white so that the user can understand the drawing operations performed on the drawing canvas. Continuing with this example, the electronic device 120 may select the sixth color characteristic 222 to blend more into the tool area than the drawing canvas so as to maintain the whiteness of the drawing canvas. That is, the modified drawing application UI 220 may include a pink tool area and a substantially white drawing canvas. For another example, the method 500 includes: when the immersion level is high, adjusting the color of the computer-generated content to a higher degree. For example, the third color characteristic is based on the first color characteristic and the second color characteristic (e.g., weighted color mixing), wherein the weight of the second color characteristic is proportional to the immersion level.

[0087] As represented by block 532, in some implementations, changing the electronic device from the first display rendering mode to the second display rendering mode includes: changing the second area on the display from the second color characteristic to a fourth color characteristic that is different from the third color characteristic. For example, when the electronic device is in the second display rendering mode, method 500 includes: displaying the fourth color characteristic at a portion of the display corresponding to the second area. As an example, referring to FIG. 3A to FIG. 3C , the electronic device 120 changes the fifth region 302 from the eighth color characteristic 314 to the ninth color characteristic 316 , while maintaining the fourth region 300 with the seventh color characteristic 312 .

[0088] As represented by block 534, in some implementations, the fourth color characteristic is based on a neutral tone standard. To this end, in some implementations, method 500 includes assigning the fourth color characteristic as a neutral tone (e.g., grayscale or monotone) so as to reduce user distraction from the computer-generated content. In some implementations, determining the fourth color characteristic includes applying a neutral tone mapping to image data of the physical environment. Thus, in some implementations, the fourth color characteristic is independent of the first color characteristic and the second color characteristic. As represented by block 536, in some implementations, the fourth color characteristic is also based on metadata. For example, reference FIG. 3A to FIG. 3C , when the metadata indicates “video stream,” the fourth color characteristic is selected to be grayscale in order to reduce the likelihood that the user 50 will be distracted from viewing the video stream 310 .

[0089] In some implementations, changing the second area includes modifying image data of the physical environment. To this end, in some implementations, the electronic device performing method 500 includes an image sensor (e.g., a camera) that captures the image data. In addition, method 500 includes modifying a portion of the image data corresponding to the second area to generate modified image data, and displaying the modified image data on a display.

[0090] In some implementations, changing the second area includes modifying a capture characteristic associated with the capture of the image data, rather than modifying a portion of the image data. For example, when in the first display rendering mode, the image sensor captures first image data of the physical environment according to a first exposure level. The first image data has a second color characteristic. Continuing with this example, changing the second area includes changing the capture characteristic (e.g., dimming the image sensor) to capture second image data of the physical environment according to a second exposure level lower than the first exposure level. The second image data has a fourth color characteristic. Changing the capture characteristic of the image can be more efficient than modifying the image data later in the graphics pipeline.

[0091] In some implementations, changing the second region includes displaying an overlay having a fourth color characteristic. For example, changing the second region from the second color characteristic to the fourth color characteristic includes displaying an overlay at a portion of the display corresponding to the second region.

[0092] In some specific implementations, method 500 includes determining whether to change the first area or the second area. To this end, method 500 includes: determining to change the first area based on determining that the computer-generated content meets the integrated color standard, and determining to change the second area based on determining that the computer-generated content meets the computer-generated content color preservation standard. For example, determining whether the computer-generated content meets the integrated color standard or the computer-generated content color preservation standard is based on metadata associated with the computer-generated content. As an example, metadata indicating that the computer-generated content is suitable for presentation in a "mixed reality (MR) environment" satisfies the integrated color standard. For another example, metadata indicating that the computer-generated content is associated with a high user engagement level (e.g., a video stream) satisfies the computer-generated content color preservation standard.

[0093] In some implementations, changing the electronic device from the first display rendering mode to the second display rendering mode includes: changing the first region from the first color characteristic to the third color characteristic, and changing the second region from the second color characteristic to the fourth color characteristic. In some implementations, changing the first region and changing the second region occur substantially concurrently.

[0094] In some implementations, changing the electronic device from the first display rendering mode to the second display rendering mode includes changing the first region from the first color characteristic to the third color characteristic, and maintaining the second region with the second color characteristic. In some implementations, changing the electronic device from the first display rendering mode to the second display rendering mode includes changing the second region from the second color characteristic to the fourth color characteristic, and maintaining the first region with the first color characteristic.

[0095] The present disclosure describes various features, none of which can achieve the benefits described herein alone. It should be understood that the various features described herein can be combined, modified or omitted, which is obvious to those of ordinary skill in the art. Other combinations and sub-combinations other than those specifically described herein will be obvious to those of ordinary skill and are intended to form a part of the present disclosure. Various methods are described herein in conjunction with various flow chart steps and / or stages. It should be understood that in many cases, certain steps and / or stages can be combined together so that the multiple steps and / or stages shown in the flow chart can be performed as a single step and / or stage. In addition, certain steps and / or stages can be divided into additional sub-components to be performed independently. In some cases, the order of steps and / or stages can be rearranged, and certain steps and / or stages can be completely omitted. In addition, the methods described herein should be understood to be broadly interpretable so that additional steps and / or stages other than those shown and described herein can also be performed.

[0096] Some or all of the methods and tasks described herein may be performed and fully automated by a computer system. In some cases, a computer system may include multiple different computers or computing devices (e.g., physical servers, workstations, storage arrays, etc.) that communicate and interoperate over a network to perform the functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in a memory or other non-transient computer-readable storage medium or device. The various functions disclosed herein may be implemented in such program instructions, but alternatively may be implemented in a dedicated circuit (e.g., ASIC or FPGA or GP-GPU) of a computer system Some or all of the disclosed functions. In the case where a computer system includes multiple computing devices, these devices may be located in the same location or not in the same location. The results of the disclosed methods and tasks may be persistently stored by converting physical storage devices such as solid-state memory chips and / or disks into different states.

[0097] The present disclosure is not intended to be limited to the specific implementations shown herein. Various modifications to the specific implementations described in the present disclosure may be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the essence or scope of the present disclosure. The teachings of the present invention provided herein may be applied to other methods and systems, and are not limited to the above-described methods and systems, and the elements and actions of the various specific implementations described above may be combined to provide more specific implementations. Therefore, the novel methods and systems described herein may be implemented in a variety of other forms; in addition, various omissions, substitutions, and changes may be made to the forms of the methods and systems described herein without departing from the essence of the present disclosure. The attached claims and their equivalents are intended to cover such forms or modified forms that fall within the scope and essence of the present disclosure.

Claims

1. A method, comprising: At an electronic device having one or more processors, a non-transitory memory, and a display: obtaining computer-generated content associated with a first color characteristic, wherein the computer-generated content is associated with a first area of ​​a physical environment; detecting a second color characteristic associated with a second area of ​​the physical environment, the second area being different than the first area of ​​the physical environment; When the electronic device is in a first display rendering mode including the first color characteristic and the second color characteristic, determining that a display rendering mode change condition is satisfied; as well as In response to determining that the display rendering mode change condition is satisfied, changing the electronic device from the first display rendering mode to a second display rendering mode, wherein the changing to the second display rendering mode comprises: changing the first region on the display from the first color characteristic to a third color characteristic; or The second region is changed on the display from the second color characteristic to a fourth color characteristic different from the third color characteristic.

2. The method according to claim 1, wherein when the electronic device is in the first display rendering mode, the method comprises: displaying the computer-generated content having the first color characteristic at a portion of the display corresponding to the first area; as well as The second color characteristic is displayed at a portion of the display corresponding to the second area.

3. The method according to claim 2, wherein when the electronic device is in the second display rendering mode, the method comprises: displaying the computer-generated content having the third color characteristic at the portion of the display corresponding to the first area; or The fourth color characteristic is displayed at the portion of the display corresponding to the second area.

4. The method according to any one of claims 1 to 3, wherein changing to the second display rendering mode comprises: changing the first region from the first color characteristic to the third color characteristic; as well as The second region is changed from the second color characteristic to the fourth color characteristic.

5. The method according to any one of claims 1 to 4, wherein changing to the second display rendering mode comprises: It is determined whether to change the first area or the second area.

6. The method of claim 5, wherein determining whether to change the first region or the second region comprises: determining to change the first region based on determining that the computer-generated content satisfies an integrated color standard; as well as The determination to change the second region is based on a determination that the computer-generated content satisfies a computer-generated content color preservation standard.

7. The method according to any one of claims 1 to 6, wherein the third color characteristic is based on the first color characteristic and the second color characteristic.

8. The method of claim 7, wherein the third color characteristic is coordinated with respect to the first color characteristic and the second color characteristic.

9. The method of any one of claims 1 to 8, wherein changing the first region from the first color characteristic to the third color characteristic comprises: A tone mapping operation is performed on the computer-generated content based on the third color characteristic.

10. The method according to any one of claims 1 to 9, wherein the fourth color characteristic is based on a neutral color tone standard.

11. The method of any one of claims 1 to 10, wherein the fourth color characteristic is independent of the first color characteristic and the second color characteristic.

12. The method of any one of claims 1 to 11, wherein at least one of the third color characteristic or the fourth color characteristic is based on metadata associated with the computer-generated content.

13. The method of claim 12, wherein the metadata indicates an application type associated with the computer-generated content. The method of claim 12 , wherein the metadata indicates an immersion level associated with the computer-generated content.

15. The method of any one of claims 1 to 14, wherein the electronic device comprises an image sensor that captures image data of the physical environment, and wherein changing the second area from the second color characteristic to the fourth color characteristic comprises: modifying a portion of the image data corresponding to the second region to generate modified image data; as well as The modified image data is displayed on the display.

16. The method of any one of claims 1 to 15, wherein the electronic device comprises an image sensor, wherein in the first display rendering mode, the image sensor captures first image data of the physical environment according to a first capture characteristic, wherein the first image data has the second color characteristic, and wherein changing the second area from the second color characteristic to the fourth color characteristic comprises: changing the image sensor from the first capture characteristic to a second capture characteristic; as well as While in the second display rendering mode, second image data of the physical environment is captured according to the second capture characteristic, wherein the second image data has the fourth color characteristic.

17. The method of any one of claims 1 to 14, wherein changing the second region from the second color characteristic to the fourth color characteristic comprises displaying an overlay at a portion of the display corresponding to the second region, wherein the overlay has the fourth color characteristic.

18. A method according to any one of claims 1 to 17, wherein the display is associated with a viewable area of ​​the physical environment, and wherein the second area comprises the entirety of the viewable area excluding the first area.

19. The method of any one of claims 1 to 18, wherein the first region is positioned in a non-overlapping arrangement relative to the second region.

20. The method of any one of claims 1 to 19, wherein the first region is adjacent to the second region.

21. A method according to any one of claims 1 to 20, wherein the display is associated with a visible area of ​​the physical environment, and wherein determining that the display rendering mode change condition is satisfied comprises detecting at least one of the first area or the second area within the visible area.

22. The method according to any one of claims 1 to 21, wherein determining that the display rendering mode change condition is satisfied comprises: A display request to display the computer-generated content is detected.

23. The method of claim 22, wherein detecting the display request comprises: A request to launch an application associated with the computer-generated content is received.

24. The method according to any one of claims 1 to 23, further comprising: When the computer-generated content having the first color characteristic is displayed at a location on the display corresponding to the first area, a user engagement score relative to the computer-generated content is determined, wherein the user engagement score characterizes a level of user engagement relative to the computer-generated content, and wherein determining that the display rendering mode change condition is satisfied includes determining that the user engagement score satisfies an engagement threshold.

25. The method according to any one of claims 1 to 24, wherein the electronic device comprises an input device, the method further comprising: When the computer-generated content having the first color characteristic is displayed at a location on the display corresponding to the first area, a user input requesting a color change is detected via the input device, wherein determining that the display rendering mode change condition is satisfied includes detecting the user input requesting the color change.

26. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic device including a display, cause the electronic device to: obtaining computer-generated content associated with a first color characteristic, wherein the computer-generated content is associated with a first area of ​​a physical environment; detecting a second color characteristic associated with a second area of ​​the physical environment, the second area being different than the first area of ​​the physical environment; When the electronic device is in a first display rendering mode including the first color characteristic and the second color characteristic, determining that a display rendering mode change condition is satisfied; as well as In response to determining that the display rendering mode change condition is satisfied, changing the electronic device from the first display rendering mode to a second display rendering mode, wherein the changing to the second display rendering mode comprises: changing the first region on the display from the first color characteristic to a third color characteristic; or The second region is changed on the display from the second color characteristic to a fourth color characteristic different from the third color characteristic.

27. An electronic device, comprising: monitor; non-transitory memory; and One or more processors configured to: obtaining computer-generated content associated with a first color characteristic, wherein the computer-generated content is associated with a first area of ​​a physical environment; detecting a second color characteristic associated with a second area of ​​the physical environment, the second area being different than the first area of ​​the physical environment; When the electronic device is in a first display rendering mode including the first color characteristic and the second color characteristic, determining that a display rendering mode change condition is satisfied; as well as In response to determining that the display rendering mode change condition is satisfied, changing the electronic device from the first display rendering mode to a second display rendering mode, wherein the changing to the second display rendering mode comprises: changing the first region on the display from the first color characteristic to a third color characteristic; or The second region is changed on the display from the second color characteristic to a fourth color characteristic different from the third color characteristic.

Images