Triggered dimming and lightening for head-mounted devices

Abstract

The invention relates to triggered dimming and lightening of a head-mountable device. In one embodiment, a method of setting a dimming value for a display is performed by a device including an at least partially transparent display having a dimming layer, one or more processors, and a non-transitory memory. The method includes detecting that a user is participating in a conversation with a conversation partner. The method includes determining a dimming amount based at least in part on detecting that the user is participating in a conversation with a conversation partner. The method includes setting a dimming value of at least a portion of the dimming layer to the dimming amount.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 471,807, filed on June 8, 2023, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to systems, methods, and devices for controlling dimming levels of a head-mountable device (HMD). Background Art

[0004] In various specific implementations, a head mounted device (HMD) may include an at least partially transparent optically transparent display. In various specific implementations, the optically transparent display includes a display layer and a dimming layer, the display layer emits light or reflects light projected from a light source according to display data, and the dimming layer dims the light passing through the optically transparent display according to the dimming data. By controlling the dimming data, the HMD can be used as "smart sunglasses" that automatically dim according to the ambient light level. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0006] Figure 1A is a block diagram of an example operational architecture according to some specific implementations.

[0007] Figure 1B is a perspective view of an XR environment according to some specific implementation.

[0008] FIG. 2A to FIG. 2C A first XR environment during a series of time periods according to some implementations is illustrated.

[0009] FIG. 3A to FIG. 3E A second XR environment during a series of time periods according to some implementations is illustrated.

[0010] Figure 4 is a flowchart representation of a method of setting a dimming value for a display according to some specific implementations.

[0011] Figure 5 is a flowchart representation of another method of setting a dimming value for a display according to some specific implementations.

[0012] Figure 6 is a block diagram of an example controller according to some specific implementations.

[0013] Figure 7 is a block diagram of an example electronic device according to some implementations.

[0014] As is common practice, the various features illustrated in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0015] Various embodiments disclosed herein include devices, systems, and methods for setting a dimming value of a display. In various embodiments, the method is performed by a device including an at least partially transparent display having a dimming layer, one or more processors, and a non-transitory memory. The method includes detecting that a user is participating in a conversation with a conversation partner. The method includes determining a dimming amount based at least in part on detecting that the user is participating in a conversation with the conversation partner. The method includes setting a dimming value of at least a portion of the dimming layer to the dimming amount.

[0016] Various embodiments disclosed herein include devices, systems, and methods for setting a dimming value of a display. In various embodiments, the method is performed by a device including an at least partially transparent display having a dimming layer, one or more processors, and a non-transitory memory. The method includes detecting that a user is participating in a conversation with a conversation partner. The method includes displaying virtual content at a display location on the display. The method includes, in response to detecting that the user is participating in a conversation with the conversation partner, increasing a dimming value of the dimming layer in an area surrounding and including the display location.

[0017] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors. One or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and a component for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0018] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0019] Some HMDs can be used as "smart sunglasses" by controlling the operation of the dimming layer of the optical transparent display. For example, when the user is indoors (in low light conditions), the dimming level of the dimming layer is set to a low value that allows more light to pass through, but when the user is outdoors (in strong light conditions), the dimming level is set to a high value that allows less light to pass through. However, in some social environments, controlling the dimming level based solely on the current ambient light level may lead to undesirable results. For example, covering your eyes while participating in a conversation may be considered impolite. Although it is relatively simple to take off your sunglasses, adjusting the dimming level of the dimming layer of the optical transparent display of the HMD may require frequent input. Therefore, in various specific implementations, the dimming level is automatically adjusted according to various factors, such as whether the user is participating in a conversation with a conversation partner, whether the user is indoors or outdoors, the position of the light source relative to the user, and user preferences.

[0020] Figure 1A 1 is a block diagram of an example operating architecture 100 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the example implementations disclosed herein. To this end, as a non-limiting example, the operating architecture 100 includes a controller 110 and a head mounted device (HMD) 120 located in a physical environment 105 including a table 107.

[0021] In some implementations, the controller 110 is configured to manage and coordinate the user's XR experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Figure 6The controller 110 is described in more detail. In some implementations, the controller 110 is a computing device located locally or remotely relative to the physical environment 105. For example, the controller 110 is a local server located within the physical environment 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the physical environment 105. In some implementations, the controller 110 is communicatively coupled to the HMD 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). For another example, the controller 110 is contained within a housing of the HMD 120.

[0022] According to some implementations, HMD 120 provides an XR experience to the user while the user is virtually and / or physically present within physical environment 105. For example, Figure 1B The physical environment 105 is illustrated from the user's perspective, where a table 107 is visible with a virtual object 115 (displayed by HMD 120) on it. In some implementations, HMD 120 includes a suitable combination of software, firmware, and / or hardware. Figure 7 HMD 120 is described in further detail. In some implementations, the functionality of controller 110 is provided by and / or combined with HMD 120.

[0023] In some implementations, the user wears the HMD 120 on his head. Therefore, the HMD 120 includes one or more XR displays provided for displaying XR content. For example, in various implementations, the HMD 120 surrounds the user's field of view. In some implementations, a handheld device (such as a smart phone or tablet) configured to present XR content is used instead of the HMD 120, and the user no longer wears the HMD 120, but holds the device with a display, with the display facing the user's field of view and the camera facing the physical environment 105. In some implementations, the handheld device may be placed in a housing that can be worn on the user's head. In some implementations, an XR cabin, housing, or room configured to present XR content is used instead of the HMD 120, in which the user no longer wears or holds the HMD 120.

[0024] In various implementations, one or more XR displays are at least partially transparent optically transparent displays. In various implementations, the optically transparent display includes a display layer that emits light or reflects light projected from a light source in accordance with display data. In various implementations, the display data is an image comprising a matrix of pixels having corresponding pixel values ​​indicating the amount of light emitted or projected at each pixel location. In various implementations, the pixel values ​​are color triples indicating the amount of light of each of three colors emitted or projected at each pixel location.

[0025] In various embodiments, the optically transparent display includes a dimming layer that dims the light passing through the optically transparent display according to the dimming data. In various embodiments, the dimming data is a single dimming value indicating the amount of dimming applied to the light at any position passing through the optically transparent display. In various embodiments, the dimming value is binary, for example, 0 (indicating that dimming is not applied) or 1 (indicating that full dimming is applied). In various embodiments, the light passing through the optically transparent display is completely dimmed to a degree known as being blocked or occluded. In various embodiments, the dimming value is non-binary. For example, in various embodiments, the dimming value adopts any one of a range of values ​​between 0 and 1. The range of values ​​can be continuous or discrete.

[0026] In various implementations, the dimming data includes a plurality of dimming values ​​for respective regions of the optically transparent display. In various implementations, the dimming data includes a pixel matrix with respective dimming values, which indicate the amount of dimming applied to the light at each pixel position passing through the optically transparent display. In various implementations, the dimming values ​​are binary or non-binary. In various implementations, the dimming values ​​are continuous or discrete. In various implementations, using binary dimming values ​​of an appropriate pattern (e.g., a checkerboard pattern), the region of the optically transparent display can be partially dimmed to a level between fully dimmed and brightened.

[0027] In various implementations, the display layer is closer to the user's eyes than the dimming layer when the user wears the HMD 120. In various implementations, the HMD 120 is a pair of smart glasses that includes smart lenses with a dimming layer but no display layer.

[0028] FIG. 2A to FIG. 2C Illustrated is a first XR environment 200 presented at least in part by a display of an electronic device, such as the HMD 120 of FIG. 1. The first XR environment 200 is based on the physical environment of a street where the electronic device is located. FIG. 2A to FIG. 2C The first XR environment 200 is illustrated during a series of time periods. In various implementations, each time period is an instant, a fraction of a second, a few seconds, a few hours, a few days, or any length of time.

[0029] FIG. 2A to FIG. 2C A gaze location indicator 299 is illustrated that indicates the user's gaze location, for example, where the user is looking in the first XR environment 200. FIG. 2A to FIG. 2C The gaze position indicator 299 is illustrated in FIG. 1 , but in various specific implementations, the gaze position indicator 299 is not displayed by the electronic device.

[0030] Figure 2A A first XR environment 200 during a first time period is illustrated. The first XR environment 200 includes multiple objects, including one or more physical objects of the physical environment (e.g., sidewalk 211, tree 212, person 213, and dog 214) and one or more virtual objects (e.g., virtual clock 221, virtual mile marker 222, and virtual running application window 223). In various implementations, certain objects (such as physical objects and virtual mile marker 222) are presented at a location in the first XR environment 200, for example, at a location defined by three coordinates in a three-dimensional (3D) XR coordinate system, so that while some objects may exist in the physical world and other objects may not, spatial relationships (e.g., distances or orientations) can be defined between them. Therefore, when the electronic device moves (e.g., changes position and / or orientation) in the first XR environment 200, the objects move on the display of the electronic device but maintain their positions in the first XR environment 200. Such virtual objects that move on the display in response to the motion of the electronic device but maintain their position in the first XR environment 200 are referred to as world-locked objects.

[0031] In various implementations, certain virtual objects, such as the virtual clock 221, are displayed at a position on the display such that the object is stationary on the display on the electronic device as the electronic device moves in the first XR environment 200. Such virtual objects that maintain their position on the display in response to the movement of the electronic device are referred to as display-locked objects.

[0032] In various implementations, the position of certain virtual objects (such as the virtual running application window 223) in the first XR environment 200 changes based on the posture of the user's body. Such virtual objects are referred to as body-locked objects. For example, when the user runs, the virtual running application window 223 maintains a position approximately one meter in front of the user and half a meter to the left of the user (e.g., relative to the position and orientation of the user's torso). When the user's head moves and the user's body does not move, the virtual running application window 223 appears at a fixed position in the first XR environment 200.

[0033] During the first time period, the dimming value of the display (or its dimming layer) is set to 1 / 2. In addition, during the first time period, as indicated by the gaze position indicator 299, the user is looking at the dog 214.

[0034] Figure 2B The first XR environment 200 is illustrated during a second time period after the first time period. During the second time period, as indicated by gaze position indicator 299, the user is looking at person 213. Based on the gaze information, as well as other factors in various implementations (such as audio of the user and / or person 213 speaking), the electronic device determines that the user is engaging in a conversation with person 213.

[0035] When the user is engaging in a conversation with person 213, person 213 may find it impolite or rude for the user to cover their eyes (e.g., by wearing dark glasses or an HMD with a high dimming value). Person 213 may think that the user is indifferent or trying to hide something. Therefore, in response to detecting that the user is engaging in a conversation with person 213, the dimming value of the display is set to zero. In response to setting the dimming value of the display to zero, the electronic device may display a dimming notification 224 indicating that the dimming value of the display has been automatically changed (e.g., based on determining that the user is engaging in a conversation with person 213). In various specific implementations, dimming notification 224 is a display lock virtual object.

[0036] The dimming notification 224 may include a confirmation enable indication 231, which, when selected, causes the electronic device to stop displaying the dimming notification 224. The dimming notification 224 may also include a rejection enable indication 232, which, when selected, causes the electronic device to stop displaying the dimming notification 224 by resetting the dimming value of the display back to its previous value (e.g., one-half the dimming value). In addition, user selection of the confirmation enable indication 231 or the rejection enable indication 232 may provide the electronic device with feedback regarding the user's preference for future automatic setting of the dimming value. In various specific implementations, if a threshold amount of time has passed without selecting the confirmation enable indication 231 or the rejection enable indication 232, the electronic device stops displaying the dimming notification 224. This also provides feedback regarding the user's preference for future automatic setting of the dimming value.

[0037] Figure 2C The first XR environment 200 is illustrated during a third time period after the second time period. During the third time period, the electronic device displays a transcription 225 of a portion of the conversation spoken by the person 213. In various implementations, the transcription 225 is a display-locked virtual object.

[0038] During the third time period, as indicated by gaze indicator 299, the user is looking at transcript 225. In various situations, person 213 may misinterpret the user's gaze as being away from person 213. For example, person 213 may believe that the person is not interested in the conversation or is interested in the real object behind transcript 225. Therefore, in various implementations, the electronic device increases the dimming value of the area of ​​the display that includes at least transcript 225. For example, during Figure 2C In the example, region 240 at the bottom of the display has a dimming value of one-half. In various implementations, the dimming is feathered so that the dimming values ​​of pixels in a region adjacent to region 240 decrease as a function of the distance from region 240.

[0039] In various implementations, the electronic device increases the dimming value of region 240 in response to displaying virtual content. For example, in various implementations, the electronic device increases the dimming value of region 240 in response to displaying transcript 225. In various implementations, the electronic device increases the dimming value of region 240 in response to determining that the user is looking at region 240 and / or virtual content within region 240. For example, in various implementations, the electronic device increases the dimming value of region 240 in response to determining that the user is looking at transcript 225. For another example, in various implementations, region 240 is dimmed when the user's gaze is directed toward virtual clock 221.

[0040] By dimming area 240, the electronic device provides a hint to person 213 that the user's averted gaze is due to the presentation (and use) of virtual content rather than due to other reasons. In addition, by dimming area 240, an enhanced contrast is provided for virtual content, making the virtual content easier for users to use.

[0041] While the transcript 225 is shown in Figure 2C At area 240 at the bottom of the display in , various types of virtual content can be displayed in various parts of the display and can trigger dimming. In various specific implementations, the virtual content is based on the conversation. For example, in various specific implementations, the virtual content includes a transcription 225 of the conversation or a translation of the conversation. In various specific implementations, the virtual content includes a notification. For example, in various specific implementations, the notification indicates a received text message, incoming call, alarm, or reminder. In various specific implementations, area 240 is located at the top, bottom, left, right, or middle of the display.

[0042] FIG. 3A to FIG. 3E Illustrated is a second XR environment 300 presented at least in part by a display of an electronic device, such as HMD 120 of FIG. 1. The second XR environment 300 is based on the physical environment of a beach where the electronic device is located. FIG. 3A to FIG. 3EThe second XR environment 300 is illustrated during a series of time periods. In various implementations, each time period is an instant, a fraction of a second, a few seconds, a few hours, a few days, or any length of time.

[0043] FIG. 3A to FIG. 3E A gaze location indicator 399 is illustrated, which indicates the user's gaze location, for example, the location in the second XR environment 300 where the user is looking. FIG. 3A to FIG. 3E The gaze position indicator 399 is illustrated in FIG. 1 , but in various specific implementations, the gaze position indicator 399 is not displayed by the electronic device.

[0044] Figure 3A A second XR environment 300 during a first time period is illustrated. The second XR environment 300 includes a plurality of objects, including one or more physical objects of the physical environment (e.g., sand 311, water 312, tree 313, sun 314, person 315, and sunglasses 316) and one or more virtual objects (e.g., virtual clock 321 and virtual boat 322). In various implementations, certain objects (such as physical objects and virtual boat marker 322) are presented at a location in the second XR environment 300, for example, at a location defined by three coordinates in a three-dimensional (3D) XR coordinate system, such that while some objects may exist in the physical world and others may not, a spatial relationship (e.g., distance or orientation) may be defined between them. Thus, when the electronic device moves (e.g., changes position and / or orientation) in the second XR environment 300, the object moves on the display of the electronic device but maintains its position in the second XR environment 300.

[0045] In various implementations, certain virtual objects, such as the virtual clock 321 , are displayed at a location on the display such that when the electronic device moves in the second XR environment 300 , the object is stationary on the display on the electronic device.

[0046] During the first time period, the dimming value of the display (or its dimming layer) is set to 1 / 2. In addition, during the first time period, as indicated by the gaze position indication mark 399, the user is gazing at the virtual boat 322.

[0047] Figure 3B A second XR environment 300 is illustrated during a second time period following the first time period according to the first embodiment. During the second time period, the user is looking at person 315, as indicated by gaze position indicator 399. Based on this gaze information, as well as other factors in various implementations (such as audio of the user and / or person 315 speaking), the electronic device determines that the user is engaging in a conversation with person 315.

[0048] exist Figure 2B In the embodiment, when the electronic device brightens the display in response to determining that the user is engaging in a conversation with person 213, and Figure 3B In response to determining that the user is engaging in a conversation with person 315, the electronic device does not dim the display. In various implementations, the electronic device uses multiple factors to determine the dimming value of the display. For example, in Figure 3B , although the electronic device determines that the user is engaging in a conversation with person 315, the electronic device further determines that the electronic device is outdoors, a strong light source (e.g., sun 314) is in front of the user, and person 315 is also shielding their eyes (e.g., wearing sunglasses 316). Based on at least one of these factors, and in some implementations, the user's preference regarding weighting of these factors, the electronic device determines that the dimming value should be set to a constant half dimming value.

[0049] Figure 3C The second XR environment 300 is illustrated during a second time period according to the second embodiment. As described above, during the second time period, the user is looking at the person 315, and the electronic device determines that the user is engaging in a conversation with the person 315.

[0050] exist Figure 2B In the example, when the electronic device brightens the entire display in response to determining that the user is engaging in a conversation with person 213, and Figure 3C In response to determining that the user is engaging in a conversation with person 315, the electronic device dims only a portion of the display surrounding person 315. In various implementations, the electronic device uses multiple factors to determine the dimming value of the remaining portion of the display. Figure 3C , although the electronic device determines that the user is engaging in a conversation with person 315, the electronic device further determines that the electronic device is outdoors, a strong light source (e.g., sun 314) is in front of the user, and person 315 is also shielding their eyes (e.g., wearing sunglasses 316). Based on at least one of these factors, and in some implementations, the user's preferences regarding weighting of these factors, the electronic device determines that the dimming value of the portion of the display surrounding person 315 should be set to zero and the dimming value of the remainder of the display should be set to a constant half dimming value.

[0051] Figure 3D The second XR environment 300 is illustrated during a second time period according to the third embodiment. As described above, during the second time period, the user is looking at the person 315, and the electronic device determines that the user is engaging in a conversation with the person 315.

[0052] exist Figure 2B , when the electronic device fully brightens the display in response to determining that the user is engaging in a conversation with person 213, and Figure 3CIn response to determining that the user is engaging in a conversation with person 315, the electronic device partially dims the display. In various implementations, the electronic device uses multiple factors to determine the dimming value of the display. For example, in Figure 3D , although the electronic device determines that the user is engaging in a conversation with person 315, the electronic device further determines that the electronic device is outdoors, a strong light source (e.g., sun 314) is in front of the user, and person 315 is also shielding their eyes (e.g., wearing sunglasses 316). Based on at least one of these factors, and in some implementations, a user preference regarding weighting of these factors, the electronic device determines that the dimming value of the display should be set to a reduced quarter dimming value.

[0053] Figure 3E The second XR environment 300 is illustrated during a third time period after the second time period. During the third time period, the electronic device displays an alarm notification 323. In various implementations, the alarm notification 323 is a display-locked virtual object.

[0054] During the third time period, as indicated by gaze indicator 399, the user is looking at alarm notification 323. In various situations, person 315 may misunderstand that the user's gaze is moving away from person 315. For example, person 315 may think that the person is not interested in the conversation or is interested in the real object behind alarm notification 323 (e.g., a part of person 315's body instead of person 315's face). Therefore, in various specific implementations, the electronic device increases the dimming value of the area of ​​the display that includes at least alarm notification 323. For example, during Figure 3E In the example, region 340 at the bottom of the display has a dimming value of three-quarters. In various implementations, the dimming is feathered so that the dimming values ​​of pixels in a region adjacent to region 340 decrease as a function of the distance from region 340.

[0055] In various implementations, the electronic device increases the dimming value of region 340 in response to displaying virtual content. For example, in various implementations, the electronic device increases the dimming value of region 340 in response to displaying alarm notification 323. In various implementations, the electronic device increases the dimming value of region 340 in response to determining that the user is looking at region 340 and / or virtual content within region 340. For example, in various implementations, the electronic device increases the dimming value of region 340 in response to determining that the user is looking at alarm notification 323. For another example, in various implementations, when the user's gaze is directed at virtual clock 321, region 340 is dimmed.

[0056] By dimming area 340, the electronic device provides a hint to person 315 that the user's averted gaze is due to the presentation (and use) of virtual content rather than due to other reasons. In addition, by dimming area 340, an enhanced contrast is provided for the virtual content, making the virtual content easier for the user to use.

[0057] Figure 4 4 is a flowchart representation of a method for setting a dimming value for a display according to some implementations. In various implementations, method 400 is performed by a device (e.g., HMD 120 of FIG. 1 ) including an at least partially transparent display having a dimming layer, one or more processors, and non-transitory memory. In some implementations, method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 400 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0058] Method 400 begins at box 410, where the device detects that the user is participating in a conversation with a conversation partner. In various embodiments, the device detects that the user is participating in a conversation with the conversation partner using input from various sensors of the device (such as a microphone, an image sensor, and an eye tracker). For example, in various embodiments, the device detects that the user is participating in a conversation with the conversation partner based on speech detected by the microphone from the user to the conversation partner and / or from the conversation partner to the user. In various embodiments, the device detects that the user is participating in a conversation with the conversation partner based on detecting the conversation partner in an image captured with the image sensor and based on detecting that the user is looking at the conversation partner based on eye tracking data from the eye tracker. In various embodiments, the device detects that the user is participating in a conversation based on the amount of time that speech and / or gaze directed to the conversation partner is detected, thereby distinguishing between a conversation and a brief message (e.g., "excuse me").

[0059] In various implementations, the device determines a confidence score indicating the likelihood that the user is engaging in a conversation with the conversation partner based on data from the sensor. In various implementations, the device detects that the user is engaging in a conversation with the conversation partner when the confidence score is greater than a threshold score.

[0060] Method 400 continues at block 420, where the device determines a dimming amount based at least in part on detecting that the user is engaging in a conversation with the conversation partner. In various implementations, determining the dimming amount includes determining a zero dimming amount in response to detecting that the user is engaging in a conversation with the conversation partner. Figure 2B In response to detecting that the user is engaging in a conversation with person 213, the dimming value of the display is set to zero.

[0061] As described in part above, in various specific implementations, in addition to detecting that the user is participating in a conversation with a conversation partner, the dimming amount is also based on multiple factors. These factors can be determined using data from various sensors, including ambient light sensors, position sensors (e.g., GPS sensors), image sensors, eye trackers, and motion sensors. For example, using an ambient light sensor, the device determines the level of ambient light. When the ambient light level is high, in response to detecting that the user is participating in a conversation with a conversation partner, the dimming value is higher or less likely to be reduced. For example, using a position sensor (and / or an image sensor), the device determines whether the user is indoors or outdoors. When the user is indoors, in response to detecting that the user is participating in a conversation with a conversation partner, the dimming value is lower or more likely to be reduced. For another example, using an image sensor, the device determines the location of a strong light source. When the user is facing a strong light source, in response to detecting that the user is participating in a conversation with a conversation partner, the dimming value is higher or less likely to be reduced. For another example, using an image sensor, the device determines the identity of the conversation partner. When the user participates in a conversation with different conversation partners, the dimming value may be different or the dimming value may be reduced. As another example, using an image sensor, the device determines whether the conversation partner is covering their eyes. When the conversation partner is covering their eyes (e.g., wearing sunglasses or a dimmed HMD), the dimming value is more likely to be lowered or lowered in response to detecting that the user is engaged in a conversation with the conversation partner. As another example, using an eye tracker, the device determines whether the user is looking at the conversation partner. In the event that the user is not looking at the conversation partner (e.g., having a conversation while hiking along a trail), the dimming value is more likely to be lowered or lowered in response to detecting that the user is engaged in a conversation with the conversation partner. As another example, using a motion sensor, the device determines the speed of the user. When the user is moving quickly (e.g., having a conversation while driving a vehicle), the dimming value is more likely to be lowered or lowered in response to detecting that the user is engaged in a conversation with the conversation partner.

[0062] In various implementations, the dimming amount is determined based on a user preference. In various implementations, the user preference is provided by a user through a user interface. In various implementations, the user preference is generated based on user feedback. In various implementations, the user feedback includes an action taken to confirm or reject the automatic setting of the dimming value, such as selecting a confirmation indication and / or a rejection indication (e.g., relative to Figure 2BAs described above), manually setting the dimming value, or removing the device. For example, if the device does not dim the display in response to detecting that the user is engaged in a conversation with a conversation partner, the user can remove the device to prevent occlusion of the user's eyes instead of manually setting the dimming value. In various specific implementations, various factors (e.g., confidence scores and data from sensors) are provided to the machine learning algorithm to determine the amount of dimming. In various specific implementations, the machine learning algorithm is further trained based on user feedback. Therefore, in various specific implementations, in response to detecting that the user is engaged in a conversation with a conversation partner, determining the amount of dimming is further based on the user's preference regarding dimming.

[0063] In various implementations, determining the dimming amount is based on the current dimming value of the dimming layer. Figure 3D In response to detecting that the user is engaging in a conversation with person 315, the dimming value is set to half of one-half of the current dimming value, for example, to one-quarter.

[0064] Method 400 continues at block 430, where the device sets the dimming value of at least a portion of the dimming layer to the dimming amount. In various implementations, the device sets the dimming value of the entire dimming layer to the dimming amount. Figure 2B In various implementations, the device sets the dimming value of only the area surrounding the conversation partner in the dimming layer to the dimming amount. Figure 3C , the dimming value of the area around person 315 in the display is set to zero. In various implementations, the device sets the dimming value of the dimming layer except for the area around the virtual content. Figure 2C , the dimming value of the display except for region 240 (and adjacent feathering regions) is set to zero.

[0065] In various implementations, setting the dimming value includes decreasing the dimming value from the current dimming value to the dimming amount. Figure 2B In the embodiment, the dimming value is set from one-half to zero. Therefore, in various embodiments, setting the dimming value includes setting the dimming value to zero. Figure 3D In , the dimming value is set from half to one quarter.

[0066] In various implementations, the method includes displaying a dimming notification in response to setting the dimming value. Figure 2B In the embodiment, the electronic device displays a dimming notification 224. In various specific implementations, setting the dimming value includes setting the dimming value from the current dimming value to the dimming amount, and the dimming notification includes an indication of setting (or resetting) the dimming value to the current dimming value when selected. Figure 2B In the example, the electronic device displays a rejection indication 232.

[0067] By automatically setting the dimming value based on detecting that the user is engaging in a conversation with a conversation partner, the user avoids the cumbersome interaction of adjusting the user's appearance in front of the conversation partner to avoid appearing impolite or otherwise socially discourteous.

[0068] Figure 5 1 is a flowchart representation of a method for setting a dimming value for a display according to some implementations. In various implementations, method 500 is performed by a device (e.g., HMD 120 of FIG. 1 ) with an at least partially transparent display including a dimming layer, one or more processors, and non-transitory memory. In some implementations, method 500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 500 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0069] Method 500 begins at box 510, where the device detects that the user is participating in a conversation with a conversation partner. In various embodiments, the device detects that the user is participating in a conversation with the conversation partner using input from various sensors of the device, such as a microphone, an image sensor, and an eye tracker. For example, in various embodiments, the device detects that the user is participating in a conversation with the conversation partner based on speech detected by the microphone from the user to the conversation partner and / or from the conversation partner to the user. In various embodiments, the device detects that the user is participating in a conversation with the conversation partner based on detecting the conversation partner in an image captured with the image sensor and based on detecting that the user is looking at the conversation partner based on eye tracking data from the eye tracker. In various embodiments, the device detects that the user is participating in a conversation based on the amount of time that speech and / or gaze directed to the conversation partner is detected, thereby distinguishing a conversation from a brief message (e.g., "excuse me").

[0070] In various implementations, the device determines a confidence score indicating the likelihood that the user is engaging in a conversation with the conversation partner based on data from the sensor. In various implementations, the device detects that the user is engaging in a conversation with the conversation partner when the confidence score is greater than a threshold score.

[0071] Method 500 continues at block 520, where the device displays virtual content at a display location on the display. In various implementations, the virtual content is based on the conversation. For example, in various implementations, the virtual content includes a transcription of the conversation or a translation of the conversation. For example, in Figure 2C In various embodiments, the electronic device displays the transcript 225. In various embodiments, the virtual content includes a notification. For example, in various embodiments, the notification indicates a received text message, incoming call, alarm, or reminder. For example, in Figure 3EIn the example, the electronic device displays an alarm notification 323.

[0072] Method 500 continues at block 530, where the device, in response to detecting that the user is engaging in a conversation with a conversation partner, increases a dimming value of the dimming layer in an area surrounding and including the display location. Figure 2C In the example, the dimming value of area 240 is changed from zero (in Figure 2B During the second time period of Figure 3E In the example, the dimming value of area 340 is changed from one quarter (in Figure 3D during the second time period) increased to three quarters.

[0073] In various implementations, increasing the dimming value includes determining a dimming amount and setting the dimming value of the dimming layer in the region to the dimming amount. In various implementations, the dimming amount is a default dimming value (e.g., 1). In various implementations, the dimming amount is based on the current dimming value of the region. In various implementations, the dimming amount is based on the above relative to Figure 4 any of the factors described.

[0074] In various implementations, the region extends to the edge of the dimming layer. In various implementations, the region includes the bottom of the dimming layer. In various implementations, the region includes the top of the dimming layer. In various implementations, the dimming value of the region decreases as a function of the distance from the edge of the dimming layer. The function is not necessarily strictly decreasing. For example, Figure 3E , the dimming value of region 340 is three quarters, and the dimming values ​​of adjacent regions decrease as a function of distance from the bottom of the dimming layer until a value of one quarter is reached.

[0075] In various implementations, increasing the dimming value is further performed in response to determining that the user's gaze is directed to the display location. Thus, in various implementations, even when the device detects that the user is participating in a conversation with a conversation partner and virtual content is displayed, the dimming value is not increased before the user views the virtual content. Similarly, in various implementations, even when the device detects that virtual content is displayed and the user views the virtual content, the dimming value is not increased unless the device detects that the user is participating in a conversation with the conversation partner. Thus, in various implementations, method 500 includes stopping increasing the dimming value in response to detecting that the user is not participating in the conversation with the conversation partner.

[0076] In various specific implementations, Figure 4 Method 400 and Figure 5For example, in various implementations, after setting the dimming value (in block 430) to remove dimming in response to detecting that the user is engaging in a conversation with a conversation partner, the device can display a notification or other virtual content (in block 520) and dim the area surrounding the virtual content (in block 530) because the user is engaging in a conversation. Figure 2B In response to detecting that the user is engaging in a conversation with person 213, the electronic device automatically dims (and displays dimming notification 224). Figure 2C In response to displaying transcript 225, the electronic device dims region 240.

[0077] By dimming the area surrounding and including the virtual content, the device provides a hint to the conversation partner to avoid misinterpreting the user's gaze away from the conversation partner as being uninterested in the conversation or interested in the real object behind the virtual content. In addition, by dimming the area surrounding and including the virtual content, the device provides an enhanced contrast between the virtual content and the physical environment, thereby enhancing readability or other forms of use.

[0078] Figure 6 is a block diagram of an example of a controller 110 according to some implementations. While some specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 602 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 606, one or more communication interfaces 608 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 610, a memory 620, and one or more communication buses 604 for interconnecting these components and various other components.

[0079] In some implementations, the one or more communication buses 604 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices 606 include at least one of a keyboard, a mouse, a trackpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0080] Memory 620 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, memory 620 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 620 optionally includes one or more storage devices remotely located from one or more processing units 602. Memory 620 includes non-transitory computer-readable storage media. In some specific implementations, memory 620 or non-transitory computer-readable storage media of memory 620 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 630 and an XR experience module 640.

[0081] The operating system 630 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the XR experience module 640 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various implementations, the XR experience module 640 includes a data acquisition unit 642, a tracking unit 644, a coordination unit 646, and a data transmission unit 648.

[0082] In some implementations, the data acquisition unit 642 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the HMD 120 of Figure 1. To this end, in various implementations, the data acquisition unit 642 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0083] In some implementations, the tracking unit 644 is configured to map the physical environment 105 and to track at least the position / location of the HMD 120 relative to the physical environment 105 of Figure 1. To this end, in various implementations, the tracking unit 644 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0084] In some implementations, the coordination unit 646 is configured to manage and coordinate the XR experience presented to the user by the HMD 120. To this end, in various implementations, the coordination unit 646 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0085] In some implementations, the data transfer unit 648 is configured to transfer data (e.g., presentation data, position data, etc.) to at least the HMD 120. To this end, in various implementations, the data transfer unit 648 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0086] Although the data acquisition unit 642, tracking unit 644, coordination unit 646, and data transfer unit 648 are shown as residing on a single device (e.g., controller 110), it should be understood that in other specific implementations, any combination of the data acquisition unit 642, tracking unit 644, coordination unit 646, and data transfer unit 648 may be located in separate computing devices.

[0087] also, Figure 6 It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 6 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0088] Figure 71 is a block diagram of an example of an HMD 120 according to some implementations. While some specific features are illustrated, those skilled in the art will appreciate from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. For this purpose, as a non-limiting example, in some implementations, the HMD 120 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, one or more XR displays 712, one or more optional internal-facing and / or external-facing image sensors 714, memory 720, and one or more communication buses 704 for interconnecting these and various other components.

[0089] In some implementations, the one or more communication buses 704 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices and sensors 706 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0090] In some implementations, one or more XR displays 712 are configured to provide an XR experience to the user. In some implementations, one or more XR displays 712 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS) and / or similar display types. In some implementations, one or more XR displays 712 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, HMD 120 includes a single XR display. In another example, HMD 120 includes an XR display for each eye of the user. In some implementations, one or more XR displays 712 can present MR and VR content.

[0091] In some implementations, the one or more image sensors 714 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (either sensor may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 714 are configured to face forward in order to acquire image data corresponding to the physical environment that the user would see when the HMD 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 714 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.

[0092] The memory 720 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, the memory 720 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 720 optionally includes one or more storage devices remotely located from the one or more processing units 702. The memory 720 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 720 or the non-transitory computer-readable storage medium of the memory 720 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 730 and an XR rendering module 740.

[0093] The operating system 730 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the XR rendering module 740 is configured to present XR content to a user via one or more XR displays 712. To this end, in various implementations, the XR rendering module 740 includes a data acquisition unit 742, a dimming unit 744, an XR rendering unit 746, and a data transmission unit 748.

[0094] In some implementations, the data acquisition unit 742 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of FIG1 . To this end, in various implementations, the data acquisition unit 742 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0095] In some implementations, the dimming unit 744 is configured to set a dimming value for at least a portion of the one or more XR displays 712. To this end, in various implementations, the dimming unit 744 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0096] In some implementations, the XR rendering unit 746 is configured to display the transformed image via one or more XR displays 712. To this end, in various implementations, the XR rendering unit 746 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0097] In some implementations, the data transmission unit 748 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110. In some implementations, the data transmission unit 748 is configured to transmit authentication credentials to the electronic device. To this end, in various implementations, the data transmission unit 748 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0098] Although the data acquisition unit 742, the dimming unit 744, the XR rendering unit 746, and the data transmission unit 748 are shown as residing on a single device (e.g., the HMD 120), it should be understood that in other specific implementations, any combination of the data acquisition unit 742, the dimming unit 744, the XR rendering unit 746, and the data transmission unit 748 may be located in separate computing devices.

[0099] also, Figure 7 More as a functional description of various features that may be present in a particular embodiment, as opposed to a schematic diagram of the structure of the specific implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 7 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0100] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0101] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0102] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "said" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0103] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method, comprising: At a device having an at least partially transparent display including a dimming layer, one or more processors, and a non-transitory memory: Detecting that the user is participating in a conversation with a conversation partner; determining an amount of dimming based at least in part on detecting that the user is engaging in a conversation with the conversation partner; as well as The dimming value of at least a portion of the dimming layer is set to the dimming amount. 2 . The method of claim 1 , wherein determining the dimming amount is further based on factors based on data from one or more sensors.

3. The method of claim 2, wherein the one or more sensors include an ambient light sensor, a position sensor, an image sensor, an eye tracker, or a motion sensor.

4. The method of any one of claims 1-3, wherein in response to detecting that the user is engaging in a conversation with the conversation partner, determining the dimming amount is further based on a user preference regarding dimming. 5 . The method according to claim 1 , wherein determining the dimming amount is based on a current dimming value of the dimming layer. 6 . The method according to claim 1 , wherein setting the dimming value comprises setting the dimming value of the entire dimming layer to the dimming amount. 7 . The method according to claim 1 , wherein setting the dimming value comprises setting the dimming value of only a region surrounding the conversation partner in the dimming layer to the dimming amount. 8 . The method according to claim 1 , wherein setting the dimming value comprises setting the dimming value of the dimming layer excluding an area surrounding virtual content.

9. The method of any one of claims 1-8, wherein setting the dimming value comprises decreasing the dimming value from a current dimming value to the dimming amount.

10. The method of any one of claims 1-9, wherein setting the dimming value comprises setting the dimming value to zero.

11. The method according to any one of claims 1 to 10, further comprising: In response to setting the dimming value, a dimming notification is displayed.

12. The method of claim 11, wherein setting the dimming value comprises setting the dimming value from a current dimming value to the dimming amount, wherein the dimming notification comprises an affordance that sets the dimming value to the current dimming value when selected.

13. A device, comprising: an at least partially transparent display including a dimming layer; non-transitory memory; as well as One or more processors configured to: Detecting that the user is participating in a conversation with a conversation partner; determining an amount of dimming based at least in part on detecting that the user is engaging in a conversation with the conversation partner; as well as The dimming value of at least a portion of the dimming layer is set to the dimming amount.

14. The device of claim 13, wherein the one or more processors are configured to determine the dimming amount based on factors based on data from one or more sensors.

15. The device of claim 14, wherein the one or more sensors include an ambient light sensor, a position sensor, an image sensor, an eye tracker, or a motion sensor.

16. The apparatus of any one of claims 13-15, wherein the one or more processors are configured to set the dimming value by setting the dimming value of the entire dimming layer to the dimming amount.

17. The apparatus according to any one of claims 13 to 16, wherein the one or more processors are configured to set the dimming value by setting the dimming value of only an area surrounding the conversation partner in the dimming layer to the dimming amount.

18. The apparatus of any one of claims 13-17, wherein the one or more processors are configured to set the dimming value by setting the dimming value of the dimming layer excluding an area surrounding virtual content.

19. The apparatus of any one of claims 13-18, wherein the one or more processors are configured to set the dimming value by setting the dimming value to zero.

20. A non-volatile memory storing one or more programs which, when executed by one or more processors of a device including an at least partially transparent display having a dimming layer, cause the device to perform any of the methods of claims 1-12.

Images