System and method for controlling light output

Abstract

Some embodiments described in the present disclosure relate to an electronic device that controls one or more objects within and / or integrated with a platform in response to a change in the environment outside the interior of the platform and / or a change in the interior of the platform. In some embodiments, an electronic device adjusts light output from one or more light sources within an interior of a platform in response to a change in ambient light within the interior of the platform. In some embodiments, an electronic device displays or causes a display of a visual indication of one or more environmental factors of an environment outside the interior of the platform.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 587,052, filed September 29, 2023, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0002] This disclosure relates generally to computer user interfaces, and more specifically to techniques for controlling light output from one or more light sources. Background Technology

[0003] Electronic devices typically control the light output in the environment. This manipulation of one or more light sources and / or objects protects user privacy. Summary of the Invention

[0004] However, some technologies for using electronic devices to control one or more objects within and / or integrated with a platform in response to changes in the environment outside the platform and / or changes within the platform itself are often cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may include multiple buttons or keystrokes. Existing technologies require more time than necessary, resulting in wasted user time and device energy. This latter consideration is particularly important in battery-powered devices.

[0005] Therefore, this technology provides electronic devices with faster and more efficient methods and interfaces for controlling one or more objects within and / or integrated with the platform in response to changes in the environment outside and / or within the platform. Such methods and interfaces optionally complement or replace other methods for controlling one or more objects within and / or integrated with the platform in response to changes in the environment outside and / or within the platform. These methods and interfaces reduce the cognitive burden on the user and result in more efficient human-machine interfaces. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charging.

[0006] An electronic device is needed for controlling the light output from one or more light sources within the platform in response to changes in ambient light within the platform. In some embodiments, the electronic device communicates with the platform. In some embodiments, the electronic device detects changes in ambient light within the platform. In some embodiments, in response to detecting changes in ambient light within the platform, the electronic device adjusts the light output within the platform based at least in part on determining that the change in ambient light within the platform is caused by a light source in the environment outside the platform or a change in a light source within the platform. Adjusting the light output from the one or more light sources within the platform in response to changes in ambient light within the platform enables a user to view information and operate the device in environments with changing ambient lighting conditions.

[0007] An electronic device is also needed to display visual indications of one or more environmental factors of the platform's environment in response to changes in the environment outside the platform's interior. In some embodiments, the electronic device communicates with a display component included in the platform. In some embodiments, while the display component is operating in a first mode, the electronic device facilitates visibility of the platform's environment, including one or more environmental factors, into the platform's interior. In some embodiments, in response to detecting data indicating a request for the display component to operate in a second mode different from the first mode, the electronic device displays the visual indications of the one or more environmental factors of the environment via the display component. Displaying the visual indications of the one or more environmental factors of the platform's environment allows a user to view information about the environment while also providing user privacy, thereby protecting user privacy and reducing potential confusion for the user who cannot determine their spatial location within the environment.

[0008] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0009] Therefore, devices are provided with faster and more efficient methods and interfaces for controlling one or more objects within and / or integrated with the platform in response to changes in the environment outside and / or within the platform, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can complement or replace other methods for controlling one or more objects within and / or integrated with the platform in response to changes in the environment outside and / or within the platform. Attached Figure Description

[0010] To better understand the various described embodiments, reference should be made to the following detailed description in conjunction with the accompanying drawings, in which the same reference numerals indicate corresponding parts in all the drawings.

[0011] Figure 1A This is a block diagram illustrating the operating environment of a computer system for providing XR experiences according to some implementation schemes.

[0012] Figures 1B to 1P It is used in Figure 1A Examples of computer systems that provide XR experiences in the operating environment.

[0013] Figure 2 This is a block diagram illustrating a computer system configured to manage and coordinate an XR experience for a user, according to some implementation schemes.

[0014] Figure 3A This is a block diagram illustrating a display generation component of a computer system configured to provide a visual component of an XR experience to a user, according to some implementation schemes.

[0015] Figures 3B to 3G This example demonstrates how to perform an operation using an Application Programming Interface (API).

[0016] Figure 4 This is a block diagram illustrating a hand tracking unit of a computer system configured to capture user gesture input according to some implementation schemes.

[0017] Figure 5A This is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input according to some implementation schemes.

[0018] Figure 5B This is a flowchart illustrating a flare-assisted gaze tracking pipeline according to some implementation schemes.

[0019] Figure 5C This is a block diagram illustrating a system with various components according to some implementation schemes.

[0020] Figures 6A to 6P Examples are illustrated of electronic devices that control one or more objects within and / or integrated with the platform in response to changes in the environment outside the platform and / or changes within the platform, according to some implementation schemes.

[0021] Figure 7 This is a flowchart illustrating a method according to some implementation schemes, in which an electronic device controls the light output from one or more light sources within the platform in response to changes in ambient light within the platform.

[0022] Figure 8 This is a flowchart illustrating a method according to some implementation schemes, in which an electronic device displays or causes a visual indication of one or more environmental factors of the environment outside the platform. Detailed Implementation

[0023] According to some implementations, this disclosure relates to a user interface for providing extended reality (XR) experiences to users.

[0024] The systems, methods, and GUIs described in this paper improve user interface interactions with virtual / augmented reality environments in a variety of ways.

[0025] The following description illustrates exemplary techniques for controlling one or more objects within and / or integrated with the platform in response to changes in the environment outside the platform and / or changes within the platform. This description is not intended to limit the scope of this disclosure, but is provided as a description of specific example implementations.

[0026] Users require electronic devices with efficient technology for controlling one or more objects within and / or integrated with the platform in response to changes in the environment outside and / or within the platform. For example, the electronic device may control the light output from one or more light sources within the platform in response to changes in ambient light within the platform. In another example, the electronic device may display visual indications of one or more environmental factors of the platform's environment in response to changes in the environment outside the platform. Efficient technology reduces the mental burden on users when controlling one or more objects within and / or integrated with the platform. This reduction in mental burden increases user productivity and makes the device easier to use. In some embodiments, the technologies described herein can reduce battery usage and processing time (e.g., by providing a user interface that requires less user input to operate).

[0027] Figure 1A Figure 6 provides a description of an example computer system for providing an XR experience to a user (as described below with respect to methods 800 and / or 1000). Figures 6A to 6P An example is given of adjusting the light output inside the platform and displaying visual indications of one or more environmental factors of the environment outside the platform.

[0028] The processes described below enhance device operability and make the user-device interface more efficient through various technologies (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the device). These technologies include providing users with improved visual feedback, reducing the amount of input required to perform operations, providing additional control options without cluttering the user interface with additional display controls, performing operations without further user input when a set of conditions are met, improving privacy and / or security, providing a more diverse, detailed, and / or realistic user experience while saving storage space, and / or additional technologies. These technologies also reduce power consumption and extend device battery life by enabling users to use the device more quickly and efficiently. This saves battery power and, consequently, weight, improving the device's ergonomics. These technologies also enable real-time communication, allowing the use of fewer and / or less precise sensors, resulting in a more compact, lighter, and cheaper device, and enabling operation in a variety of lighting conditions. These technologies reduce energy consumption, thereby reducing the heat generated by the device. This is especially important for wearable devices, where if a device generates too much heat, even when operating entirely within the parameters of its components, it can become uncomfortable for the user to wear.

[0029] Furthermore, in a method described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the described method can be repeated in multiple repetitions such that, during the repetitions, all conditions determining the steps in the method are satisfied in different repetitions of the method. For example, if the method requires performing a first step (if the conditions are satisfied) and a second step (if the conditions are not satisfied), those skilled in the art will know that the stated steps are repeated until both conditions are satisfied and not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each condition described in the method is satisfied. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing discretionary operations based on the satisfaction of the corresponding one or more conditions, and thus to determine whether possible conditions have been satisfied without explicitly repeating the steps of the method until all conditions determining the steps in the method are satisfied. Those skilled in the art will also understand that, similar to a method having discretionary steps, a system or computer-readable storage medium can repeat the steps of the method multiple times as needed to ensure that all discretionary steps have been performed.

[0030] In some implementation schemes, such as Figure 1AAs shown, an XR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted display (HMD), a monitor, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker or output device 160, a haptic output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a haptic sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., home appliances, wearable devices, etc.). In some implementations, one or more of the input device 125, output device 155, sensor 190, and peripheral device 195 are integrated with the display generation component 120 (e.g., in a head-mounted or handheld device).

[0031] In describing XR experiences, various terms are used to distinguish several related but different environments that a user can sense and / or interact with (e.g., interacting with inputs detected by the computer system 101 that generates the XR experience, causing the computer system to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computer system 101). The following is a subset of these terms: Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. Physical environments, such as physical parks, include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through senses such as sight, touch, hearing, taste, and smell.

[0032] Extended Reality: Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In XR, a subset of a person's physical motion, or a representation thereof, is tracked, and in response, one or more properties of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. For example, an XR system can detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the adjustment of the properties of virtual objects in the XR environment can be done in response to a representation of physical motion (e.g., a voice command). A person can use any of their senses to sense and / or interact with XR objects, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects that create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. In another example, an audio object can enable audio transparency, which selectively introduces ambient sound from the physical environment, with or without computer-generated audio. In some XR environments, people can sense and / or interact only with the audio object.

[0033] Examples of XR include virtual reality and mixed reality.

[0034] Virtual Reality (VR): A virtual reality (VR) environment is a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. Examples of virtual objects include trees, buildings, and computer-generated images representing human avatars. A person can sense and / or interact with virtual objects in a VR environment through the simulation of their presence within the computer-generated environment and / or through the simulation of a subset of their physical movements within the computer-generated environment.

[0035] Mixed Reality: Compared to VR environments, which are designed to be entirely based on computer-generated sensory input, mixed reality (MR) environments refer to simulated environments designed to incorporate sensory input from the physical environment, or its representations, in addition to computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any state between, but not limited to, a purely physical environment as one end and a virtual reality environment as the other. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems used to present an MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical objects or their representations from the physical environment). For example, a system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0036] Examples of mixed reality include augmented reality and augmented virtual reality.

[0037] Augmented Reality (AR): An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation of the physical environment. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or semi-transparent display, allowing a person to perceive the virtual objects overlaid on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects overlaid on the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects onto the physical environment, such as as a hologram or onto a physical surface, allowing a person to perceive the virtual objects superimposed on the physical environment. Augmented reality environments also refer to simulated environments in which the representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system can transform one or more sensor images to apply a selected viewpoint (e.g., viewpoint) different from the viewpoint captured by the imaging sensor. As another example, the representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions of it, such that the modified portions can be representative but not realistic versions of the original captured image. Furthermore, the representation of the physical environment can be transformed by graphically removing or blurring portions of it.

[0038] Augmented Virtual: An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment combines one or more sensory inputs from a physical environment. Sensory input can be a representation of one or more characteristics of the physical environment. For example, an AV park could have virtual trees and virtual buildings, but a person's face could be realistically reproduced from an image taken of a physical person. Similarly, virtual objects could adopt the shape or color of a physical object imaged by one or more imaging sensors. Furthermore, virtual objects could adopt shadows that correspond to the sun's position within the physical environment.

[0039] In augmented reality, mixed reality, or virtual reality environments, a view of the three-dimensional environment is visible to the user. This view is typically visible to the user via a virtual viewport through one or more display generating components (e.g., a display providing stereoscopic content to different eyes of the same user), the virtual viewport having a viewport boundary that defines the extent of the three-dimensional environment visible to the user via the one or more display generating components. In some embodiments, the area defined by the viewport boundary is smaller than the user's visual field in one or more dimensions (e.g., based on the user's visual field, the size of one or more display generating components, optical properties or other physical characteristics, and / or the position and / or orientation of one or more display generating components relative to the user's eyes). In some embodiments, the area defined by the viewport boundary is larger than the user's visual field in one or more dimensions (e.g., based on the user's visual field, the size of one or more display generating components, optical properties or other physical characteristics, and / or the position and / or orientation of one or more display generating components relative to the user's eyes). The viewport and viewport boundary typically move with the movement of one or more display generating components (e.g., with the user's head for head-mounted devices, or with the user's hand for handheld devices such as tablets or smartphones). The user's viewpoint determines what is visible within the viewport. The viewpoint typically specifies the position and orientation relative to the 3D environment, and as the viewpoint moves, the view of the 3D environment also moves within the viewport. For head-mounted devices, the viewpoint is typically based on the position and orientation of the user's head, face, and / or eyes to provide a perceptibly accurate view of the 3D environment that offers an immersive experience when the user is using the head-mounted device. For handheld or fixed devices, the viewpoint shifts with the movement of the handheld or fixed device and / or with changes in the user's positioning relative to the handheld or fixed device (e.g., the user moves towards, away from, up, down, right, and / or left). For a device that includes a display generating component with virtual pass-through, portions of the physical environment visible (e.g., displayed and / or projected) via one or more display generating components are based on the field of view of one or more cameras communicating with the display generating component, which typically move with the movement of the display generating component (e.g., for a head-mounted device, it moves with the movement of the user's head, or for a handheld device such as a tablet or smartphone, it moves with the movement of the user's hand), because the user's viewpoint moves with the movement of the field of view of the one or more cameras (and the appearance of one or more virtual objects displayed via one or more display generating components is updated based on the user's viewpoint (e.g., the display positioning and pose of the virtual objects are updated based on the movement of the user's viewpoint)).For a display generating component with optical transparency, portions of the physical environment visible through one or more display generating components (e.g., optically visible through one or more portions or fully transparent portions of the display generating component) are based on the user's field of view through the portion or fully transparent portion of the display generating component (e.g., for a head-mounted device, it moves with the movement of the user's head, or for a handheld device such as a tablet or smartphone, it moves with the movement of the user's hand), because the user's viewpoint moves with the movement of the user's field of view through the portion or fully transparent portion of the display generating component (and the appearance of one or more virtual objects is updated based on the user's viewpoint).

[0040] In some embodiments, the representation of the physical environment (e.g., displayed via virtual passthrough or optical passthrough) may be partially or completely occluded by the virtual environment. In some embodiments, the amount of virtual environment displayed (e.g., the amount of physical environment not displayed) is based on the immersion level of the virtual environment (e.g., relative to the representation of the physical environment). For example, increasing the immersion level optionally results in more virtual environment being displayed, replacing and / or occluding more physical environment, and decreasing the immersion level optionally results in less virtual environment being displayed, thereby revealing portions of the physical environment that were previously not displayed and / or occluded. In some embodiments, at a particular immersion level, one or more first background objects (e.g., in the representation of the physical environment) are visually de-emphasized more than one or more second background objects (e.g., dimmed, blurred, displayed with increased transparency), and one or more third background objects are de-emphasized. In some embodiments, the level of immersion includes the associated degree to which virtual content (e.g., a virtual environment and / or virtual content) displayed by the computer system occludes background content (e.g., content other than the virtual environment and / or virtual content) around / behind the virtual environment, optionally including the number of items of the displayed background content and / or the displayed visual characteristics of the background content (e.g., color, contrast, and / or opacity), the angular range of the virtual content displayed via the display generating component (e.g., 60 degrees for content displayed at low immersion, 120 degrees for content displayed at medium immersion, or 180 degrees for content displayed at high immersion), and / or the proportion of the field of view displayed via the display generating component occupied by the virtual content (e.g., 33% of the field of view occupied by the virtual content at low immersion, 66% of the field of view occupied by the virtual content at medium immersion, or 100% of the field of view occupied by the virtual content at high immersion). In some embodiments, the background content is included in the background on which the virtual content is displayed (e.g., background content in a representation of the physical environment). In some implementations, background content includes user interfaces (e.g., user interfaces generated by a computer system corresponding to an application), virtual objects not associated with or included in the virtual environment and / or virtual content (e.g., files generated by the computer system or representations of other users), and / or real objects (e.g., transparent objects representing real objects in the user's surrounding physical environment, visible such that they are displayed via display generation components and / or via transparent or semi-transparent components of the display generation components, because the computer system does not obscure / impede their visibility through the display generation components). In some implementations, at a low immersion level (e.g., a first immersion level), the background, virtual, and / or real objects are displayed in an unobstructed manner. For example, a virtual environment with a low immersion level is optionally displayed simultaneously with background content, which is optionally displayed at full brightness, color, and / or semi-transparency.In some implementations, at higher immersion levels (e.g., a second immersion level above the first immersion level), background, virtual, and / or real objects are displayed in an occluded manner (e.g., dimmed, blurred, or removed from the display). For example, a corresponding virtual environment with a high immersion level is displayed without concurrently displaying background content (e.g., in full-screen or fully immersive mode). Alternatively, a virtual environment displayed at a medium immersion level is displayed concurrently with darkened, blurred, or otherwise de-emphasized background content. In some implementations, the visual characteristics of background objects differ between background objects. For example, at a particular immersion level, one or more first background objects are visually de-emphasized more than one or more second background objects (e.g., dimmed, blurred, and / or displayed with increased transparency), and one or more third background objects are stopped from being displayed. In some implementations, zero immersion or a zero immersion level corresponds to a virtual environment that is stopped from being displayed, and instead, a representation of the physical environment (optionally having one or more virtual objects, such as an application, window, or virtual 3D object) is displayed, and the representation of the physical environment is not occluded by the virtual environment. Using physical input elements to adjust immersion levels provides a quick and efficient way to adjust immersion, which enhances the operability of computer systems and makes user-device interfaces more efficient.

[0041] Viewpoint-locked virtual objects: When a computer system displays a virtual object at the same position and / or location within the user's viewpoint, the virtual object remains viewpoint-locked even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the direction forward of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); therefore, the user's viewpoint remains fixed even when the user's gaze shifts without moving the user's head. In embodiments where the computer system has a display generating component (e.g., a display screen) that can be repositioned relative to the user's head, the user's viewpoint is the augmented reality view presented to the user on the display generating component of the computer system. For example, a viewpoint-locked virtual object displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint, even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or orientation of a viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In an implementation where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, so the virtual object is also referred to as a "head-locked virtual object".

[0042] Environment-locked visual objects: When a computer system displays a virtual object at a location and / or position within the user's viewpoint, the virtual object is environment-locked (or, "world-locked"), the location and / or position being based on a location and / or object within a three-dimensional environment (e.g., a physical or virtual environment) (e.g., selected and / or anchored to that location and / or object with reference to it). As the user's viewpoint shifts, the location and / or object in the environment relative to the user's viewpoint changes, causing the environment-locked virtual object to appear at different locations and / or positions within the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user appears at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) so that the tree is now centered to the left in the user's viewpoint (e.g., the tree's position shifts in the user's viewpoint), the environment-locked virtual object locked to the tree appears centered to the left in the user's viewpoint. In other words, the position and / or orientation of an environment-locked virtual object displayed in the user's viewpoint depends on the position to which the virtual object is locked and / or the orientation and / or orientation of the object within the environment. In some implementations, the computer system uses a stationary frame of reference (e.g., a coordinate system anchored to a fixed position and / or object in the physical environment) to determine the position of the environment-locked virtual object displayed in the user's viewpoint. An environment-locked virtual object may be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object), or it may be locked to a movable part of the environment (e.g., a vehicle, animal, person, or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm, or foot), causing the virtual object to move with the viewpoint or that part of the environment to maintain a fixed relationship between the virtual object and that part of the environment.

[0043] In some implementations, environment-locked or viewpoint-locked virtual objects exhibit lazy following behavior, reducing or delaying their movement relative to the movement of a reference point they are following. In some implementations, when exhibiting lazy following behavior, the computer system intentionally delays the movement of the virtual object when movement of the reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 cm and 300 cm from the viewpoint) is detected. For example, when the reference point (e.g., that portion of the environment or the viewpoint) moves at a first rate, the virtual object is moved by the device to remain locked to the reference point, but moves at a second rate that is slower than the first rate (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up). In some implementations, when the virtual object exhibits lazy following behavior, the device ignores small movements of the reference point (e.g., ignores movements of the reference point below a threshold amount, such as 0 to 5 degrees or 0 cm to 50 cm). For example, when the reference point (e.g., a portion of the environment or viewpoint to which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed to maintain a fixed or substantially fixed position relative to a portion of the viewpoint or environment to which the virtual object is locked), and when the reference point (e.g., a portion of the environment or viewpoint to which the virtual object is locked) moves by a second amount greater than the first amount, the distance between the reference point and the virtual object first increases (e.g., because the virtual object is being displayed to maintain a fixed or substantially fixed position relative to a portion of the viewpoint or environment to which the virtual object is locked), and then decreases when the amount of movement of the reference point increases to above a threshold (e.g., a "lazy following" threshold), because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some implementations, maintaining a substantially fixed position of the virtual object relative to a reference point includes displaying the virtual object within a threshold distance (e.g., 1cm, 2cm, 3cm, 5cm, 15cm, 20cm, 50cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the reference point).

[0044] Hardware: Many different types of electronic systems enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headsets / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablet devices, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection techniques that project graphic images onto a person's retina. Projection systems can also be configured to project virtual objects into a physical environment, such as as holograms or onto a physical surface. In some embodiments, controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. The following is relative to... Figure 2Controller 110 is described in more detail. In some embodiments, controller 110 is a computing device located locally or remotely relative to scene 105 (e.g., physical environment). For example, controller 110 is a local server located within scene 105. In another example, controller 110 is a remote server (e.g., cloud server, central server, etc.) located outside scene 105. In some embodiments, controller 110 is communicatively coupled to display generation components 120 (e.g., HMD, monitor, projector, touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included within the housing (e.g., physical enclosure) of the display generating component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more input devices in the input device 125, one or more output devices in the output device 155, one or more sensors in the sensor 190, and / or one or more peripheral devices in the peripheral device 195, or shares the same physical housing or support structure with one or more of the aforementioned devices.

[0045] In some embodiments, the display generation component 120 is configured to provide an XR experience to a user (e.g., at least the visual component of the XR experience). In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. The following section discusses... Figure 3A The display generation component 120 is described in more detail. In some embodiments, the functionality of the controller 110 is provided by and / or combined with the display generation component 120.

[0046] According to some implementation schemes, when a user is virtually and / or physically present within scene 105, display generation component 120 provides the user with an XR experience.

[0047] In some embodiments, the display generating component is worn on a part of the user's body (e.g., on his / her head, his / her hand, etc.). Therefore, the display generating component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generating component 120 surrounds the user's field of view. In some embodiments, the display generating component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user holds the device having a display facing the user's field of view and a camera facing scene 105. In some embodiments, the handheld device is optionally placed within a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generating component 120 is an XR chamber, housing, or room configured to present XR content, wherein the user does not wear or hold the display generating component 120. Many user interfaces described with reference to one type of hardware used for displaying XR content (e.g., a handheld device or a tripod-mounted device) can be implemented on another type of hardware used for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface illustrating interaction with XR content triggered by an interaction occurring in the space in front of a handheld device or tripod-mounted device can be similarly implemented using an HMD, where the interaction occurs in the space in front of the HMD and the response to the XR content is displayed via the HMD. Similarly, a user interface illustrating interaction with XR content triggered by movement of a handheld device or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)) can be similarly implemented using an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)).

[0048] Despite Figure 1A The relevant features of the operating environment 100 are illustrated herein, but those skilled in the art will understand from this disclosure that various other features are not illustrated for the sake of brevity and to avoid obscuring further relevant aspects of the exemplary embodiments disclosed herein.

[0049] Figures 1A to 1PVarious examples of computer systems for performing methods and providing audio, visual, and / or haptic feedback as part of the user interface described herein are illustrated. In some embodiments, the computer system includes one or more display generation components (e.g., first and second display components 1-120a, 1-120b and / or first and second optical modules 11.1.1-104a and 11.1.1-104b) for displaying virtual elements and / or representations of the physical environment to a user of the computer system, the virtual elements and / or the representations of the physical environment optionally being generated based on detected events and / or user input detected by the computer system. The user interface generated by the computer system is optionally corrected by one or more corrective lenses 11.3.2-216 to make it easier for a user who would otherwise use glasses or contact lenses to correct their vision to view the user interface, the one or more corrective lenses optionally being removably attached to one or more optical modules in the optical modules. While many user interfaces illustrated herein represent a single view of the user interface, user interfaces in HMDs optionally employ two optical modules (e.g., first display component 1-120a and second display component 1-120b and / or first optical module 11.1.1-104a and second optical module 11.1.1-104b) for display, one optical module for the user's right eye and a different optical module for the user's left eye, presenting slightly different images to the two different eyes to generate the illusion of stereoscopic depth. A single view of the user interface is typically a right-eye view or a left-eye view; the depth effect is explained in text or using other diagrams or views. In some embodiments, the computer system includes one or more external displays (e.g., display component 1-108) for displaying status information of the computer system to the user of the computer system (when the computer system is not worn) and / or to others near the computer system, the status information optionally generated based on detected events and / or user input detected by the computer system. In some embodiments, the computer system includes one or more audio output components (e.g., electronic components 1-112) for generating audio feedback, which is optionally generated based on detected events and / or user input detected by the computer system. In some embodiments, the computer system includes one or more input devices for detecting input, such as one or more sensors (e.g., sensor components 1-356 and / or sensor components 1-356) for detecting information about the physical environment of the device. Figure 1I One or more sensors), which can be used (optionally with one or more illuminators, such as Figure 1IThe system combines the illuminators described herein to generate digital pass-through images, capture visual media (e.g., photographs and / or videos) corresponding to the physical environment, or determine the pose (e.g., positioning and / or orientation) of physical objects and / or surfaces in the physical environment, enabling the placement of virtual objects based on the detected pose of the physical objects and / or surfaces. In some embodiments, the computer system includes one or more input devices for detecting input, such as one or more sensors (e.g., sensor assemblies 1-356 and / or...) for detecting hand positioning and / or movement. Figure 1I One or more sensors), which can be used (optionally with one or more illuminators, such as Figure 1I The illuminators 6-124 described herein (in combination) determine when one or more air gestures are performed. In some embodiments, the computer system includes one or more input devices for detecting input, such as one or more sensors for detecting eye movement (e.g., Figure 1I Eye-tracking and gaze-tracking sensors in the system), these sensors can be used (optionally combined with one or more lights, such as...) Figure 10The light (11.3.2-110) in the image determines attention or gaze localization and / or gaze movement, which may optionally be used to detect gaze-only input based on gaze movement and / or dwell. Combinations of the various sensors described above can be used to determine user facial expressions and / or hand movements for generating an avatar or representation of the user, such as an anthropomorphic avatar or representation for real-time communication sessions, wherein the avatar has facial expressions, hand movements, and / or body movements detected by the user based on or similar to the device. Gaze and / or attention information may optionally be combined with hand tracking information to determine user interaction with one or more user interfaces based on direct and / or indirect input, such as air gestures or input using one or more hardware input devices, such as one or more buttons (e.g., first buttons 1-128, buttons 11.1.1-114, second buttons 1-132 and / or dials or buttons 1-328), knobs (e.g., first buttons 1-128, buttons 11.1.1-114 and / or dials or buttons 1-328), digital crowns (e.g., pressable and twistable or rotatable first buttons 1-128, buttons 11.1.1-114 and / or dials or buttons 1-328), touchpads, touchscreens, keyboards, mice and / or other input devices. One or more buttons (e.g., first buttons 1-128, buttons 11.1.1-114, second buttons 1-132, and / or dials or buttons 1-328) are optionally used to perform system operations, such as recentering content in the user-visible 3D environment of the device, displaying the main user interface for launching an application, initiating a real-time communication session, or initiating the display of a virtual 3D background. Knobs or digital crowns (e.g., pressable and twistable or rotatable first buttons 1-128, buttons 11.1.1-114, and / or dials or buttons 1-328) are optionally rotatable to adjust parameters of the visual content, such as the level of immersion of the virtual 3D environment (e.g., the extent to which the virtual content occupies the user's viewport in the 3D environment) or other parameters associated with the 3D environment and the virtual content displayed via optical modules (e.g., first display components 1-120a and second display components 1-120b and / or first optical modules 11.1.1-104a and second optical modules 11.1.1-104b).

[0050] Figure 1BExamples of head-mounted display (HMD) devices 1-100 configured to be worn by a user and provide virtual and altered / mixed reality (VR / AR) experiences are illustrated in front, top, and perspective views. The HMD 1-100 may include a display unit 1-102 or assembly, an electronic strip assembly 1-104 connected to and extending from the display unit 1-102, and a strap assembly 1-106 secured at either end to the electronic strip assembly 1-104. The electronic strip assembly 1-104 and the strap 1-106 may be part of a retention assembly configured to wrap around the user's head to hold the display unit 1-102 against the user's face.

[0051] In at least one example, the band assembly 1-106 may include a first band 1-116 configured to wrap around the back of the user's head and a second band 1-117 configured to extend above the top of the user's head. As shown, the second band may extend between the first electronic band 1-105a and the second electronic band 1-105b of the electronic band assembly 1-104. The band assembly 1-104 and the band assembly 1-106 may be part of a fixing mechanism that extends rearward from the display unit 1-102 and is configured to hold the display unit 1-102 against the user's face.

[0052] In at least one example, the fixing mechanism includes a first electronic strip 1-105a, which includes a first proximal end 1-134 coupled to a display unit 1-102 (e.g., a housing 1-150 of the display unit 1-102) and a first distal end 1-136 opposite to the first proximal end 1-134. The fixing mechanism may also include a second electronic strip 1-105b, which includes a second proximal end 1-138 coupled to the housing 1-150 of the display unit 1-102 and a second distal end 1-140 opposite to the second proximal end 1-138. The fixing mechanism may also include a first strip 1-116 and a second strip 1-117, the first strip including a first end 1-142 coupled to the first distal end 1-136 and a second end 1-144 coupled to the second distal end 1-140, and the second strip extending between the first electronic strip 1-105a and the second electronic strip 1-105b. Strips 1-105a to b and strip 1-116 may be coupled via a connecting mechanism or component 1-114. In at least one example, the second strip 1-117 includes a first end 1-146 coupled to the first electronic strip 1-105a between a first proximal end 1-134 and a first distal end 1-136, and a second end 1-148 coupled to the second electronic strip 1-105b between a second proximal end 1-138 and a second distal end 1-140.

[0053] In at least one example, the first electronic strip and the second electronic strips 1-105a to b comprise plastic, metal, or other structural materials forming the shape of the substantially rigid strips 1-105a to b. In at least one example, the first strip and the second strips 1-116, 1-117 are formed of an elastic flexible material (including woven textiles, rubber, etc.). The first strip 1-116 and the second strip 1-117 may be flexible enough to conform to the shape of the user's head when wearing the HMD 1-100.

[0054] In at least one example, one or more of the first electronic stripe and the second electronic stripe 1-105a to b may define an inner stripe volume and include one or more electronic components disposed within the inner stripe volume. In one example, such as Figure 1B As shown, the first electronic strip 1-105a may include electronic components 1-112. In one example, electronic components 1-112 may include a speaker. In another example, electronic components 1-112 may include computing components, such as a processor.

[0055] In at least one example, the housing 1-150 defines a first front opening 1-152. The front opening is located in... Figure 1B The section marked 1-152 with dashed lines is because the display assembly 1-108 is configured to obscure the first opening 1-152 when the HMD 1-100 is assembled, as viewed from above. The housing 1-150 may also define a rearward second opening 1-154. The housing 1-150 also defines an internal volume between the first opening 1-152 and the second opening 1-154. In at least one example, the HMD 1-100 includes a display assembly 1-108, which may include a front cover disposed in or across the front opening 1-152 to obscure the front opening 1-152 and a display screen (shown in other figures). In at least one example, the display screen of the display assembly 1-108, and the display assembly 1-108 in general, has a curvature configured to follow the curvature of a user's face. The display screen of display component 1-108 can be bent as shown to complement the user's facial features and the overall curvature from one side of the face to the other, such as from left to right and / or from top to bottom, wherein display unit 1-102 is pressed.

[0056] In at least one example, the housing 1-150 may define a first hole 1-126 between a first opening 1-152 and a second opening 1-154, and a second hole 1-130 between the first opening 1-152 and the second opening 1-154. The HMD 1-100 may also include a first button 1-126 disposed in the first hole 1-128, and a second button 1-132 disposed in the second hole 1-130. The first button 1-128 and the second button 1-132 are pressable through their respective holes 1-126 and 1-130. In at least one example, the first button 1-126 and / or the second button 1-132 may be a rotary dial and a pressable button. In at least one example, the first button 1-128 is a pressable and rotary dial button, and the second button 1-132 is a pressable button.

[0057] Figure 1C A rear perspective view of HMD 1-100 is illustrated. HMD 1-100 may include a light seal 1-110 extending rearwardly around the periphery of housing 1-150 of display assembly 1-108, as shown. The light seal 1-110 may be configured to extend from housing 1-150 to the user's face, surrounding the user's eyes, to block external light from being visible. In one example, HMD 1-100 may include a first display assembly 1-120a and a second display assembly 1-120b disposed at or within a rearwardly facing second opening 1-154 defined by housing 1-150 and / or disposed within the internal volume of housing 1-150 and configured to project light through the second opening 1-154. In at least one example, each display assembly 1-120a to b may include a corresponding display screen 1-122a, 1-122b configured to project light toward the user's eyes in a rearward direction through the second opening 1-154.

[0058] In at least one example, reference Figure 1B and Figure 1C Both, the display assembly 1-108 can be a front-facing display assembly including a display screen configured to project light in a first forward direction, and the rear display screens 1-122a to b can be configured to project light in a second rearward direction opposite to the first direction. As described above, the light seal 1-110 can be configured to block light from outside the HMD 1-100 from reaching the user's eyes, including a component made of... Figure 1B The front perspective view shows the light projected by the front display screen of the display assembly 1-108. In at least one example, the HMD 1-100 may also include a curtain 1-124 that blocks the second opening 1-154 between the housing 1-150 and the rear display assemblies 1-120a to b. In at least one example, the curtain 1-124 may be elastic or at least partially elastic.

[0059] Figure 1B and Figure 1C Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1D to 1F In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1D to 1F Any of the features, components, and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1B and Figure 1C Examples of devices, features, components, and parts are shown.

[0060] Figure 1D An exploded view of an example HMD 1-200 including its various parts or components, separated according to the modularity and selective coupling of these components. For example, HMD 1-200 may include a strip 1-216 selectively coupled to a first electronic strip 1-205a and a second electronic strip 1-205b. The first fixed strip 1-205a may include a first electronic component 1-212a, and the second fixed strip 1-205b may include a second electronic component 1-212b. In at least one example, the first strip and the second strips 1-205a to 1-205b are removably coupled to a display unit 1-202.

[0061] Furthermore, HMD 1-200 may include a light-sealing member 1-210 configured to be removably coupled to display unit 1-202. HMD 1-200 may also include a lens 1-218, which may be removably coupled to display unit 1-202, for example, on a first display assembly and a second display assembly including a display screen. Lens 1-218 may include a custom prescription lens configured for vision correction. As noted, in Figure 1D The exploded view shows that each component described above can be removably coupled, attached, reattached, and replaced to update the component, or replaced for different users. For example, belts such as belt 1-216, light seals such as light seal 1-210, lenses such as lens 1-218, and electronic strips such as electronic strips 1-205a to b can be replaced according to the user, so that these parts are customized to fit and correspond to a single user of HMD 1-200.

[0062] Figure 1D Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figure 1B , Figure 1C and Figures 1E to 1FIn any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figure 1B , Figure 1C and Figures 1E to 1F Any of the features, components, and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1D Examples of devices, features, components, and parts are shown.

[0063] Figure 1E An exploded view illustrating an example of a display unit 1-306 of an HMD is shown. The display unit 1-306 may include a front display assembly 1-308, a frame / housing assembly 1-350, and a curtain assembly 1-324. The display unit 1-306 may also include a sensor assembly 1-356, a logic board assembly 1-358, and a cooling assembly 1-360 disposed between the frame assembly 1-350 and the front display assembly 1-308. In at least one example, the display unit 1-306 may also include a rear display assembly 1-320, which includes a first rear display screen 1-322a and a second rear display screen 1-322b disposed between the frame 1-350 and the curtain assembly 1-324.

[0064] In at least one example, the display unit 1-306 may further include a motor assembly 1-362 configured as an adjustment mechanism for adjusting the positioning of the display screens 1-322a to b of the display unit 1-320 relative to the frame 1-350. In at least one example, the display unit 1-320 is mechanically coupled to the motor assembly 1-362, and each display screen 1-322a to b has at least one motor, such that the motor is capable of translating the display screens 1-322a to b to match the interpupillary distance of the user's eyes.

[0065] In at least one example, display unit 1-306 may include a dial or button 1-328 that is pressable relative to frame 1-350 and accessible to a user outside frame 1-350. Button 1-328 may be electrically connected to motor assembly 1-362 via a controller, such that button 1-328 can be operated by a user to cause the motor of motor assembly 1-362 to adjust the positioning of display screens 1-322a to b.

[0066] Figure 1E Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1B to 1D and Figure 1F In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1B to 1D and Figure 1FAny of the features, components, and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1E Examples of devices, features, components, and parts are shown.

[0067] Figure 1F An exploded view of another example of a display unit 1-406 of an HMD device similar to other HMD devices described herein is illustrated. The display unit 1-406 may include a front display assembly 1-402, a sensor assembly 1-456, a logic board assembly 1-458, a cooling assembly 1-460, a frame assembly 1-450, a rear display assembly 1-421, and a curtain assembly 1-424. The display unit 1-406 may also include a motor assembly 1-462 for adjusting the positioning of a first display sub-assembly 1-420a and a second display sub-assembly 1-420b of the rear display assembly 1-421, including a first and second corresponding display screen for interpupillary adjustment, as described above.

[0068] References in this article Figures 1B to 1E The following figures, which are referenced in this disclosure, will be used to describe the subject in more detail. Figure 1F The exploded view shows the various parts, systems, and components. Figure 1F The display unit 1-406 shown can be connected with Figures 1B to 1E The shown fixture assembly and integration includes electronic strips, belts, and other components (including light seals, connecting assemblies, etc.).

[0069] Figure 1F Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1B to 1E In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1B to 1E Any of the features, components, and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1F Examples of devices, features, components, and parts are shown.

[0070] Figure 1G An example is the front cover assembly 3-100 of the HMD device described herein (e.g., Figure 1G An exploded perspective view of the front cover assembly 3-1) of the HMD 3-100 shown or any other HMD device shown and described herein. Figure 1GThe front cover assembly 3-100 shown may include a transparent or translucent cover 3-102, a shield 3-104 (or “cover”), an adhesive layer 3-106, a display assembly 3-108 including a biconvex lens panel or array 3-110, and a structural decorative element 3-112. The adhesive layer 3-106 secures the shield 3-104 and / or the transparent cover 3-102 to the display assembly 3-108 and / or the decorative element 3-112. The decorative element 3-112 secures various components of the front cover assembly 3-100 to the frame or base of the HMD device.

[0071] In at least one example, such as Figure 1G As shown, the transparent cover 3-102, the protective cover 3-104, and the display assembly 3-108 including a biconvex lens array 3-110 can be bent to adapt to the curvature of a user's face. The transparent cover 3-102 and the protective cover 3-104 can be bent in two or three dimensions, for example, vertically in and out of the Z-plane along the Z direction, and horizontally in and out of the ZX-plane along the X direction. In at least one example, the display assembly 3-108 may include the biconvex lens array 3-110 and a display panel with pixels configured to project light through the protective cover 3-104 and the transparent cover 3-102. The display assembly 3-108 can be bent in at least one direction (e.g., the horizontal direction) to adapt to the curvature of a user's face from one side (e.g., the left) to the other (e.g., the right). In at least one example, each layer or component of the display assembly 3-108 (which will be shown and described in more detail in the following figures, but may include the biconvex lens array 3-110 and the display layer) may be similarly or concentrically curved in the horizontal direction to accommodate the curvature of the user's face.

[0072] In at least one example, the cover 3-104 may include a transparent or translucent material through which the display component 3-108 projects light. In one example, the cover 3-104 may include one or more opaque portions, such as opaque ink-printed portions or other opaque film portions on the back of the cover 3-104. When the HMD device is worn, the rear surface may be the surface of the cover 3-104 facing the user's eyes. In at least one example, the opaque portion may be on the front surface of the cover 3-104 opposite the rear surface. In at least one example, one or more opaque portions of the cover 3-104 may include peripheral portions that visually conceal any components surrounding the outer periphery of the display screen of the display component 3-108. In this way, the opaque portions of the cover conceal any other components of the HMD device that would otherwise be visible through the transparent or translucent cover 3-102 and / or the cover 3-104, including electronic components, structural components, etc.

[0073] In at least one example, the housing 3-104 may define one or more transparent aperture portions 3-120 through which the sensor can transmit and receive signals. In one example, portion 3-120 is an aperture through which the sensor can extend or transmit and receive signals. In one example, portion 3-120 is a transparent portion, or a portion more transparent than the surrounding translucent or opaque portion of the housing, through which the sensor can transmit and receive signals through the housing and via the transparent cover 3-102. In one example, the sensor may include a camera, an IR sensor, a LUX sensor, or any other visual or non-visual environmental sensor of the HMD device.

[0074] Figure 1G Any of the features, components, and / or parts shown herein (including their arrangement and configuration) may be included individually or in any combination of any other examples of the devices, features, components, and parts described herein. Similarly, any of the features, components, and / or parts shown and described herein (including their arrangement and configuration) may be included individually or in any combination of any other examples of the devices, features, components, and parts described herein. Figure 1G Examples of devices, features, components, and parts are shown.

[0075] Figure 1H An exploded view of an example HMD device 6-100 is shown. The HMD device 6-100 may include a sensor array or system 6-102, which includes one or more sensors, cameras, projectors, etc., mounted to one or more components of the HMD 6-100. In at least one example, the sensor system 6-102 may include a bracket 1-338 on which one or more sensors of the sensor system 6-102 may be fixed / secured.

[0076] Figure 1I A portion of an HMD device 6-100, including a front transparent cover 6-104 and a sensor system 6-102, is illustrated. The sensor system 6-102 may include multiple different sensors, transmitters, and receivers, including cameras, IR sensors, projectors, etc. The transparent cover 6-104 is illustrated in front of the sensor system 6-102 to illustrate the relative positioning of the various sensors and transmitters and the orientation of each sensor / transmitter in system 6-102. As referenced herein, "side," "side," "lateral," "horizontal," and other similar terms refer to... Figure 1J The orientation or direction indicated by the X-axis. Terms such as "vertical," "upward," "downward," and similar terms refer to the orientation or direction indicated by... Figure 1JThe orientation or direction indicated by the Z-axis. Terms such as "frontward," "rearward," "forward," "backward," and similar terms refer to the orientation or direction indicated by the Z-axis. Figure 1J The orientation or direction indicated by the Y-axis shown.

[0077] In at least one example, a transparent cover 6-104 may define the front outer surface of an HMD device 6-100, and a sensor system 6-102, including various sensors and their components, may be positioned behind the cover 6-104 in the Y-axis / direction. The cover 6-104 may be transparent or translucent to allow light to pass through it, including both light detected by the sensor system 6-102 and light emitted therefrom.

[0078] As described elsewhere herein, the HMD device 6-100 may include one or more controllers, which include processors for electrically coupling various sensors and transmitters of the sensor system 6-102 to one or more motherboards, processing units, and other electronic devices such as displays. Furthermore, as will be shown in more detail below with reference to other accompanying drawings, various sensors, transmitters, and other components of the sensor system 6-102 may be coupled to the HMD device 6-100. Figure 1I Various structural frame components, brackets, etc., not shown. For clarity, Figure 1I The components of the sensor system 6-102 are shown, which are not attached to or electrically coupled to other components.

[0079] In at least one example, the device may include one or more controllers having a processor configured to execute instructions stored on a memory component electrically coupled to the processor. These instructions may include, or cause the processor to execute, one or more algorithms for self-correcting the angle and position of the various cameras described herein as the camera's initial position, angle, or orientation is affected by collisions or deformations due to accidental drop events or other events over time.

[0080] In at least one example, the sensor system 6-102 may include one or more scene cameras 6-106. System 6-102 may include two scene cameras 6-102, respectively positioned on either side of the nose bridge or arch structure of the HMD device 6-100, such that each of the two cameras 6-106 approximately corresponds to the positioning of the user's left and right eyes behind the cover 6-103. In at least one example, the scene cameras 6-106 are generally oriented forward in the Y direction to capture images in front of the user during use of the HMD 6-100. In at least one example, the scene cameras are color cameras and, when the HMD device 6-100 is used, provide images and content for MR video pass-through to a display screen facing the user's eyes. The scene cameras 6-106 may also be used for environment and object reconstruction.

[0081] In at least one example, the sensor system 6-102 may include a first depth sensor 6-108 that is generally pointing forward in the Y direction. In at least one example, the first depth sensor 6-108 may be used for environment and object reconstruction as well as user hand and body tracking. In at least one example, the sensor system 6-102 may include a second depth sensor 6-110 centrally located along the width of the HMD device 6-100 (e.g., along the X-axis). For example, the second depth sensor 6-110 may be located above the central bridge of the nose or on an adapter feature above the nose when the user wears the HMD 6-100. In at least one example, the second depth sensor 6-110 may be used for environment and object reconstruction as well as hand and body tracking. In at least one example, the second depth sensor may include a LiDAR sensor.

[0082] In at least one example, the sensor system 6-102 may include a depth projector 6-112, which is typically forward-facing to project electromagnetic waves (e.g., in the form of a predetermined spot pattern) into or within the field of view of the user and / or scene camera 6-106, or into or beyond the field of view of the user and / or scene camera 6-106. In at least one example, the depth projector is capable of projecting electromagnetic waves of light in the form of a spot pattern, which are reflected from an object and back into the aforementioned depth sensors, including depth sensors 6-108 and 6-110. In at least one example, the depth projector 6-112 may be used for environment and object reconstruction, as well as hand and body tracking.

[0083] In at least one example, the sensor system 6-102 may include a downward-facing camera 6-114, whose field of view is generally directed downwards relative to the HMD device 6-100 on the Z-axis. In at least one example, the downward-facing camera 6-114 may be positioned as shown on the left and right sides of the HMD device 6-100 and used for hand and body tracking, head-mounted device tracking, and facial avatar detection and creation for displaying a user avatar on the front display screen of the HMD device 6-100 as described elsewhere herein. For example, the downward-facing camera 6-114 may be used to capture facial expressions and movements of the user's face below the HMD device 6-100, including the cheeks, mouth, and chin.

[0084] In at least one example, the sensor system 6-102 may include a jaw camera 6-116. In at least one example, the jaw camera 6-116 may be positioned as shown on the left and right sides of the HMD device 6-100 and used for hand and body tracking, head-mounted device tracking, and facial avatar detection and creation for displaying a user avatar on the front display screen of the HMD device 6-100 as described elsewhere herein. For example, the jaw camera 6-116 may be used to capture facial expressions and movements of the user's face below the HMD device 6-100, including the user's jaw, cheeks, mouth, and chin. Used for hand and body tracking, head-mounted device tracking, and facial avatar creation. In at least one example, the sensor system 6-102 may include a side camera 6-118. The side camera 6-118 may be oriented to capture left and right views along the X-axis or in a direction relative to the HMD device 6-100. In at least one example, the side camera 6-118 may be used for hand and body tracking, head-mounted device tracking, and facial avatar detection and reconstruction.

[0085] In at least one example, the sensor system 6-102 may include multiple eye-tracking and gaze-tracking sensors for determining identity, status, and the user's gaze direction during and / or prior to use. In at least one example, the eye / gaze-tracking sensor may include a nose-eye camera 6-120 positioned on either side of the user's nose and adjacent to the user's nose when wearing the HMD device 6-100. The eye / gaze sensor may also include a bottom eye camera 6-122 positioned below the respective user's eye for capturing images of the eye for use in facial avatar detection and creation, gaze tracking, and iris identification functions.

[0086] In at least one example, sensor system 6-102 may include an infrared illuminator 6-124 that is pointed outward from HMD device 6-100 to illuminate the external environment and any objects therein with IR light for IR detection using one or more IR sensors of sensor system 6-102. In at least one example, sensor system 6-102 may include a flicker sensor 6-126 and an ambient light sensor 6-128. In at least one example, flicker sensor 6-126 may detect the refresh rate of the overhead light to avoid display flicker. In one example, infrared illuminator 6-124 may include a light-emitting diode and may be specifically designed for low-light environments to illuminate a user's hands and other objects in low light for detection by the infrared sensors of sensor system 6-102.

[0087] In at least one example, multiple sensors (including scene camera 6-106, downward camera 6-114, chin camera 6-116, side camera 6-118, depth projector 6-112, and depth sensors 6-108, 6-110) can be used in combination with an electrically coupled controller to combine depth data with camera data for hand tracking and for sizing, thereby improving the hand tracking and object recognition and tracking functions of the HMD device 6-100. In at least one example, as described above and Figure 1I The downward-facing camera 6-114, the chin camera 6-116, and the side camera 6-118 shown can be wide-angle cameras capable of operating in both the visible and infrared spectra. In at least one example, these cameras 6-114, 6-116, and 6-118 can operate solely in black-and-white light detection to simplify image processing and achieve sensitivity.

[0088] Figure 1I Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1J to 1L In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1J to 1L Any of the features, components, and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1I Examples of devices, features, components, and parts are shown.

[0089] Figure 1JA lower perspective view of an example HMD 6-200 including a cover or shield 6-204 fixed to a frame 6-230 is shown. In at least one example, a sensor 6-203 of a sensor system 6-202 may be disposed around the periphery of the HMD 6-200 such that the sensor 6-203 is disposed outwardly around the periphery of the display area or region 6-232 so as not to obstruct the view of the displayed light. In at least one example, the sensor may be disposed behind the shield 6-204 and aligned with a transparent portion of the shield, thereby allowing light to pass back and forth through the shield 6-204 by the sensor and projector. In at least one example, an opaque ink or other opaque material or film / layer may be disposed on the shield 6-204 around the display area 6-232 to conceal components of the HMD 6-200 outside the display area 6-232 rather than through a transparent portion defined by the opaque portion through which the sensor and projector transmit and receive light and electromagnetic signals during operation. In at least one example, the shield 6-204 allows light to pass through the display (e.g., within the display area 6-232), but does not allow light to pass radially outward from the display area surrounding the periphery of the display and the shield 6-204.

[0090] In some examples, the shield 6-204 includes a transparent portion 6-205 and an opaque portion 6-207, as described above and elsewhere herein. In at least one example, the opaque portion 6-207 of the shield 6-204 may define one or more transparent areas 6-209 through which the sensor 6-203 of the sensor system 6-202 transmits and receives signals. In the illustrated examples, the sensor 6-203 of the sensor system 6-202, which transmits and receives signals through the shield 6-204, or more specifically through the transparent area 6-209 defined by the opaque portion 6-207 of the shield 6-204, may include... Figure 1I The examples illustrate those same or similar sensors, such as depth sensors 6-108 and 6-110, depth projector 6-112, first scene camera and second scene camera 6-106, first downward camera and second downward camera 6-114, first side camera and second side camera 6-118, and first infrared illuminator and second infrared illuminator 6-124. These sensors also... Figure 1K and Figure 1L The example is shown. Other sensors, sensor types, number of sensors, and their relative positioning can be included in one or more other examples of the HMD.

[0091] Figure 1J Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figure 1I and Figures 1K to 1LIn any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figure 1I and Figures 1K to 1L Any of the features, components, and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1J Examples of devices, features, components, and parts are shown.

[0092] Figure 1K A front view of a portion of an example of an HMD device 6-300, including a display 6-334, brackets 6-336, 6-338, and a frame or housing 6-330, is shown. Figure 1K The examples shown do not include a front cover or shield to illustrate brackets 6-336 and 6-338. For example, Figure 1J The shield 6-204 shown includes an opaque portion 6-207 that visually covers / blocks the view of anything outside the display / display area 6-334 (e.g., radially / peripherally outside the display / display area), including the sensor 6-303 and the bracket 6-338.

[0093] In at least one example, various sensors of sensor system 6-302 are coupled to brackets 6-336, 6-338. In at least one example, scene camera 6-306 includes strict tolerances for angles relative to each other. For example, the tolerance for the mounting angle between two scene cameras 6-306 may be 0.5 degrees or less, such as 0.3 degrees or less. To achieve and maintain such strict tolerances, in one example, scene camera 6-306 may be mounted to bracket 6-338 instead of a housing. The bracket may include a cantilever on which scene camera 6-306 and other sensors of sensor system 6-302 may be mounted to maintain their positioning and orientation in the event of a drop event caused by a user that results in any deformation of other brackets 6-226, housing 6-330, and / or housing.

[0094] Figure 1K Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1I to 1J and Figure 1L In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1I to 1J and Figure 1L Any of the features, components, and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1K Examples of devices, features, components, and parts are shown.

[0095] Figure 1L A bottom view illustrating an example of an HMD 6-400 including a front display / cover assembly 6-404 and a sensor system 6-402 is shown. The sensor system 6-402 is compatible with the above and other parts of this document (including references). Figures 1I to 1K Other sensor systems described are similar. In at least one example, the jaw camera 6-416 may be oriented downwards to capture images of the user's lower facial features. In one example, the jaw camera 6-416 may be directly coupled to a frame or housing 6-430 or one or more internal brackets that are directly coupled to the frame or housing 6-430 shown. The frame or housing 6-430 may include one or more holes / openings 6-415 through which the jaw camera 6-416 transmits and receives signals.

[0096] Figure 1L Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1I to 1K In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1I to 1K Any of the features, components, and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1L Examples of devices, features, components, and parts are shown.

[0097] Figure 1M A rear perspective view of an interpupillary distance (IPD) adjustment system 11.1.1-102 is illustrated. This IPD adjustment system includes a first optical module and a second optical module 11.1.1-104a-104a-104a-105a, slidably engaged / coupled to corresponding guide rods 11.1.1-108a ... In at least one example, buttons 11.1.1-114 can be electrically communicated with the first motor and the second motors 11.1.1-110a to b via a processor or other circuit components to activate the first motor and the second motors 11.1.1-110a to b and respectively cause the first optical module and the second optical modules 11.1.1-104a to b to change their positions relative to each other.

[0098] In at least one example, the first and second optical modules 11.1.1-104a to b may include corresponding display screens configured to project light toward the user's eyes when the HMD 11.1.1-100 is worn. In at least one example, the user can manipulate (e.g., press and / or rotate) buttons 11.1.1-114 to activate positional adjustment of the optical modules 11.1.1-104a to b to match the interpupillary distance of the user's eyes. The optical modules 11.1.1-104a to b may also include one or more cameras or other sensors / sensor systems for imaging and measuring the user's IPD, such that the optical modules 11.1.1-104a to b can be adjusted to match the IPD.

[0099] In one example, a user can manipulate buttons 11.1.1-114 to cause automatic positional adjustment of the first and second optical modules 11.1.1-104a to b. In another example, a user can manipulate buttons 11.1.1-114 to cause manual adjustment, moving the optical modules 11.1.1-104a to b further or closer (e.g., when the user rotates buttons 11.1.1-114 in one way or another) until the user visually matches their own IPD. In one example, manual adjustment is communicated electronically via one or more circuits, and the power for moving the optical modules 11.1.1-104a to b via motors 11.1.1-110a to b is supplied by a power source. In another example, the adjustment and movement of the optical modules 11.1.1-104a to b via the manipulation buttons 11.1.1-114 are mechanically actuated via the movement buttons 11.1.1-114.

[0100] Figure 1M Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included, individually or in any combination, in any other example of the devices, features, components, and parts shown and described herein in any other illustrated figures. Similarly, any of the features, components, and / or parts shown and described herein (including their arrangement and configuration) may be included, individually or in any combination, in any other example of the devices, features, components, and / or parts shown and / or described herein. Figure 1M Examples of devices, features, components, and parts are shown.

[0101] Figure 1N A front perspective view of a portion of HMD 11.1.2-100 is shown, including an outer structural frame 11.1.2-102 defining first and second holes 11.1.2-106a, 11.1.2-106b, and an inner or intermediate structural frame 11.1.2-104. Holes 11.1.2-106a to b are located in... Figure 1NThe holes 11.1.2-106a to b are shown in dashed lines because viewing the HMD 11.1.2-100 may be obstructed by one or more other components coupled to the inner frame 11.1.2-104 and / or the outer frame 11.1.2-102, as illustrated. In at least one example, the HMD 11.1.2-100 may include a first mounting bracket 11.1.2-108 coupled to the inner frame 11.1.2-104. In at least one example, the mounting bracket 11.1.2-108 is coupled to the inner frame 11.1.2-104 between the first and second holes 11.1.2-106a to b.

[0102] Mounting brackets 11.1.2-108 may include intermediate or central portions 11.1.2-109 coupled to the inner frame 11.1.2-104. In some examples, the intermediate or central portions 11.1.2-109 may not be the geometric center or middle of the brackets 11.1.2-108. Instead, the intermediate / central portions 11.1.2-109 may be positioned between a first cantilever extension arm and a second cantilever extension arm extending away from the intermediate portions 11.1.2-109. In at least one example, mounting bracket 108 includes first cantilever arms 11.1.2-112 and second cantilever arms 11.1.2-114 extending away from the intermediate portions 11.1.2-109 of the mounting brackets 11.1.2-108 coupled to the inner frame 11.1.2-104.

[0103] like Figure 1N As shown, the outer frame 11.1.2-102 may define a curved geometry on its lower side to adapt to the user's nose when the user wears the HMD 11.1.2-100. This curved geometry may be referred to as the bridge of the nose 11.1.2-111 and is centrally located on the lower side of the HMD 11.1.2-100 as shown. In at least one example, the mounting bracket 11.1.2-108 may be connected to the inner frame 11.1.2-104 between holes 11.1.2-106a and b, such that the cantilever 11.1.2-112, 11.1.2-114 extend downward and laterally outward away from the central portion 11.1.2-109 to complement the nose bridge geometry of the outer frame 11.1.2-102. In this way, the mounting bracket 11.1.2-108 is configured to adapt to the user's nose, as described above. The geometry of the bridge of the nose 11.1.2-111 adapts to the nose, as it provides a curvature that conforms to the shape of the user's nose, offering a comfortable fit from above, above, and around.

[0104] The first cantilever 11.1.2-112 may extend in a first direction away from the middle portion 11.1.2-109 of the mounting bracket 11.1.2-108, and the second cantilever 11.1.2-114 may extend in a second direction opposite to the first direction away from the middle portion 11.1.2-109 of the mounting bracket 11.1.2-108. The first cantilever 11.1.2-112 and the second cantilever 11.1.2-114 are referred to as “cantilever” or “cantilever” arms because each arm 11.1.2-112, 11.1.2-114 includes free distal ends 11.1.2-116, 11.1.2-118, respectively, which are not attached to the inner frame 11.1.2-102 and the outer frame 11.1.2-104. In this way, arms 11.1.2-112 and 11.1.2-114 extend from the middle section 11.1.2-109, which can be connected to the inner frame 11.1.2-104, while the distal ends 11.1.2-102 and 11.1.2-104 are not attached.

[0105] In at least one example, the HMD 11.1.2-100 may include one or more components coupled to the mounting bracket 11.1.2-108. In one example, the components include multiple sensors 11.1.2-110a-f. Each of the multiple sensors 11.1.2-110a-f may include various types of sensors, including cameras, IR sensors, etc. In some examples, one or more of the sensors 11.1.2-110a-f may be used for object recognition in three-dimensional space, making it important to maintain the precise relative positioning of two or more of the multiple sensors 11.1.2-110a-f. The cantilever nature of the mounting bracket 11.1.2-108 protects the sensors 11.1.2-110a-f from damage and displacement in the event of an accidental drop by the user. Because the sensors 11.1.2-110a-f cantilevered on the arms 11.1.2-112 and 11.1.2-114 of the mounting bracket 11.1.2-108, the stress and deformation of the internal frame and / or the external frames 11.1.2-104 and 11.1.2-102 are not transmitted to the cantilever arms 11.1.2-112 and 11.1.2-114, and therefore do not affect the relative position of the sensors 11.1.2-110a-f coupled to / mounted to the mounting bracket 11.1.2-108.

[0106] Figure 1NAny of the features, components, and / or parts shown herein (including their arrangement and configuration) may be included individually or in any combination of any other example of the devices, features, components, and other examples described herein. Similarly, any of the features, components, and / or parts shown and described herein (including their arrangement and configuration) may be included individually or in any combination of any other example. Figure 1N Examples of devices, features, components, and parts are shown.

[0107] Figure 10 An example of optical modules 11.3.2-100 for use in electronic devices, such as HMDs, including the HDM devices described herein, is illustrated. As shown in one or more other examples described herein, optical module 11.3.2-100 may be one of two optical modules within an HMD, wherein each optical module is aligned to project light toward a user's eye. In this way, a first optical module may project light toward a user's first eye via a display screen, and a second optical module of the same device may project light toward a user's second eye via another display screen.

[0108] In at least one example, the optical module 11.3.2-100 may include an optical frame or housing 11.3.2-102, which may also be referred to as a tube or optical module tube. The optical module 11.3.2-100 may also include a display 11.3.2-104 coupled to the housing 11.3.2-102, the display including one or more display screens. The display 11.3.2-104 may be coupled to the housing 11.3.2-102 such that the display 11.3.2-104 is configured to project light toward the user's eyes when the HMD to which the display module 11.3.2-100 belongs is worn during use. In at least one example, the housing 11.3.2-102 may surround the display 11.3.2-104 and provide connection features for coupling other components of the optical module described herein.

[0109] In one example, the optical module 11.3.2-100 may include one or more cameras 11.3.2-106 coupled to the housing 11.3.2-102. The cameras 11.3.2-106 may be positioned relative to the display 11.3.2-104 and the housing 11.3.2-102 such that the cameras 11.3.2-106 are configured to capture one or more images of a user's eye during use. In at least one example, the optical module 11.3.2-100 may also include a light strip 11.3.2-108 surrounding the display 11.3.2-104. In one example, the light strip 11.3.2-108 is disposed between the display 11.3.2-104 and the camera 11.3.2-106. The light strip 11.3.2-108 may include a plurality of lights 11.3.2-110. The plurality of lights may include one or more light-emitting diodes (LEDs) or other lights configured to project light toward the user's eyes when the HMD is worn. The individual lights 11.3.2-110 in the light strips 11.3.2-108 may be spaced apart around the light strips 11.3.2-108, and are therefore uniformly or non-uniformly spaced around the display 11.3.2-104 at various locations on the light strips 11.3.2-108 and around the display 11.3.2-104.

[0110] In at least one example, the housing 11.3.2-102 defines a viewing opening 11.3.2-101 through which a user can view the display 11.3.2-104 when wearing the HMD device. In at least one example, the LEDs are configured and arranged to emit light onto the user's eyes through the viewing opening 11.3.2-101. In one example, a camera 11.3.2-106 is configured to capture one or more images of the user's eyes through the viewing opening 11.3.2-101.

[0111] As mentioned above, Figure 10 Each of the components and features of the optical modules 11.3.2-100 shown can be replicated in another (e.g., a second) optical module set together with the HMD to interact with the user's other eye (e.g., projecting light and capturing images).

[0112] Figure 10 Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figure 1P Any other example of the device, feature, component, and part shown or otherwise described herein. Similarly, refer to... Figure 1P Any of the features, components, and / or parts shown, described, or otherwise referred to herein (including their arrangement and configuration) may be included individually or in any combination. Figure 10 Examples of devices, features, components, and parts are shown.

[0113] Figure 1P A cross-sectional view of an example optical module 11.3.2-200 is shown, which includes a housing 11.3.2-202, a display assembly 11.3.2-204 coupled to the housing 11.3.2-202, and a lens 11.3.2-216 coupled to the housing 11.3.2-202. In at least one example, the housing 11.3.2-202 defines a first aperture or channel 11.3.2-212 and a second aperture or channel 11.3.2-214. Channels 11.3.2-212 and 11.3.2-214 can be configured to slidably engage corresponding tracks or guides of an HMD device to allow the optical module 11.3.2-200 to be adjusted and positioned relative to the user's eye to match the user's interpupillary distance (IPD). The housing 11.3.2-202 can slidably engage the guide rod to secure the optical module 11.3.2-200 in the appropriate position within the HMD.

[0114] In at least one example, the optical module 11.3.2-200 may further include a lens 11.3.2-216 coupled to the housing 11.3.2-202 and positioned between the display assembly 11.3.2-204 and the user's eye when the HMD is worn. The lens 11.3.2-216 may be configured to direct light from the display assembly 11.3.2-204 to the user's eye. In at least one example, the lens 11.3.2-216 may be part of a lens assembly including a corrective lens removably attached to the optical module 11.3.2-200. In at least one example, lenses 11.3.2-216 are positioned above light strips 11.3.2-208 and one or more eye-tracking cameras 11.3.2-206, such that cameras 11.3.2-206 are configured to capture an image of a user's eye through lenses 11.3.2-216, and light strips 11.3.2-208 include lamps configured to project light onto the user's eye through lenses 11.3.2-216 during use.

[0115] Figure 1P Any of the features, components, and / or parts shown herein (including their arrangement and configuration) may be included individually or in any combination of any other example of the devices, features, components, and parts described herein and of any other example. Similarly, any of the features, components, and / or parts shown and described herein (including their arrangement and configuration) may be included individually or in any combination of any other example. Figure 1P Examples of devices, features, components, and parts are shown.

[0116] Figure 2This is a block diagram of an example controller 110 according to some implementation schemes. Although some specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features have not been illustrated for the sake of brevity and to avoid obscuring further relevant aspects of the implementation schemes disclosed herein. Therefore, as a non-limiting example, in some embodiments, controller 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0117] In some embodiments, one or more communication buses 204 include circuitry for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0118] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures, or subsets thereof, including optional operating system 230 and XR experience module 240.

[0119] Operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR experience module 240 is configured to manage and coordinate single or multiple XR experiences for one or more users (e.g., single XR experiences for one or more users, or multiple XR experiences for corresponding groups of one or more users). To this end, in various embodiments, XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0120] In some implementations, the data acquisition unit 241 is configured to acquire data from... Figure 1A The data acquisition unit 241 includes at least a display generation component 120 and optionally acquires data (e.g., presentation data, interactive data, sensor data, location data, etc.) from one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data acquisition unit 241 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0121] In some implementations, the tracking unit 242 is configured to map scene 105, and the tracking at least shows the generated component 120 relative to... Figure 1A The tracking unit 242 tracks the location / position of scene 105, and optionally tracks the position of one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the location / position of one or more portions of the user's hand, and / or the location of one or more portions of the user's hand relative to the user's hand. Figure 1A The movement of scene 105 relative to the display generating component 120 and / or relative to a coordinate system (defined relative to the user's hand). The following refers to the movement relative to... Figure 4 The hand tracking unit 244 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the user's gaze (or more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hand)) or relative to XR content displayed via display generation component 120. The following description is relative to... Figure 5A The eye-tracking unit 243 is described in more detail.

[0122] In some implementations, coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by display generation component 120, and optionally by one or more of output device 155 and / or peripheral device 195. To this end, in various implementations, coordination unit 246 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0123] In some embodiments, the data sending unit 248 is configured to send data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data sending unit 248 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0124] Although the data acquisition unit 241, the tracking unit 242 (e.g., including eye tracking unit 243 and hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 are shown residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including eye tracking unit 243 and hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 may reside in a separate computing device.

[0125] also, Figure 2 This is used more as a functional description of various features that can exist in a particular specific implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, individually shown items can be combined, and some items can be separated. For example, Figure 2 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0126] Figure 3AThis is a block diagram illustrating an example of generating component 120 according to some embodiments. Although some specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features have not been illustrated for the sake of brevity and to avoid obscuring further relevant aspects of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the display generating component 120 (e.g., HMD) includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, and / or similar interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional internal and / or external image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0127] In some embodiments, one or more communication buses 304 include circuitry for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 306 include inertial measurement units (IMUs), accelerometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, haptic engines, and / or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0128] In some embodiments, one or more XR displays 312 are configured to provide an XR experience to a user. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, one or more XR displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, display generation component 120 (e.g., HMD) includes a single XR display. In another example, display generation component 120 includes XR displays for each of the user's eyes. In some embodiments, one or more XR displays 312 are capable of presenting MR and VR content. In some embodiments, one or more XR displays 312 are capable of presenting either MR or VR content.

[0129] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand and optionally the user's arm (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward in order to acquire image data corresponding to the scene that the user would see in the absence of a display generation component 120 (e.g., an HMD) (and may be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0130] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures, or subsets thereof, including optional operating system 330 and XR rendering module 340.

[0131] Operating system 330 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR rendering module 340 is configured to present XR content to a user via one or more XR displays 312. Therefore, in various embodiments, XR rendering module 340 includes a data acquisition unit 342, an XR rendering unit 344, an XR mapping generation unit 346, and a data transmission unit 348.

[0132] In some implementations, the data acquisition unit 342 is configured to acquire data from at least... Figure 1A The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). To this end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0133] In some implementations, the XR rendering unit 344 is configured to render XR content via one or more XR displays 312. To this end, in various implementations, the XR rendering unit 344 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0134] In some implementations, the XR mapping generation unit 346 is configured to generate XR maps based on media content data (e.g., 3D maps of mixed reality scenes or maps in which computer-generated objects can be placed to generate extended reality physical environments). To this end, in various implementations, the XR mapping generation unit 346 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0135] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data transmission unit 348 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0136] Although the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data sending unit 348 are shown residing in a single device (e.g., Figure 1A The data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data sending unit 348 are located on the display generation component 120, but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data sending unit 348 may be located in a separate computing device.

[0137] also, Figure 3A This is used more as a functional description of various features that may exist in a particular specific implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 3A Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0138] Specific embodiments within the scope of this disclosure may be implemented, in whole or in part, using a tangible computer-readable storage medium (or a plurality of tangible computer-readable storage media of one or more types) that encodes one or more computer-readable instructions. It should be understood that computer-readable instructions may be organized in any format, including applications, components, processes, software, and / or parts.

[0139] Specific embodiments within the scope of this disclosure include computer-readable storage media that encode instructions organized as an application (e.g., application 3160) that, when executed by one or more processing units, control the execution of an electronic device (e.g., device 3150). Figure 3B Methods Figure 3C The methods and / or one or more other processes and / or methods described herein.

[0140] It should be recognized that the application of 3160 ( Figure 3DThe application 3160 (as shown in the diagram) can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets, or other applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, messaging applications, and / or map applications. In some embodiments, application 3160 is an application pre-installed on device 3150 at the time of purchase (e.g., a first-party application). In some embodiments, application 3160 is an application provided to device 3150 via operating system update files (e.g., a first-party or second-party application). In some embodiments, application 3160 is an application provided via an app store. In some embodiments, the app store can be an app store pre-installed on device 3150 at the time of purchase (e.g., a first-party app store). In some embodiments, the app store is a third-party app store (e.g., an app store provided by another app store, downloaded via a network, and / or read from a storage device).

[0141] refer to Figure 3B and Figure 3F Application 3160 acquires information (e.g., 3010). In some embodiments, at 3010, information is acquired from at least one hardware component of device 3150. In some embodiments, at 3010, information is acquired from at least one software module of device 3150. In some embodiments, at 3010, information is acquired from at least one hardware component external to device 3150 (e.g., peripheral devices, accessory devices, and / or servers). In some embodiments, the information acquired at 3010 includes location information, time information, notification information, user information, environmental information, electronic device status information, weather information, media information, historical information, event information, hardware information, and / or motion information. In some embodiments, in response to acquiring information at 3010 and / or after this step, application 3160 provides the information to the system (e.g., 3020).

[0142] In some implementations, the system (e.g., Figure 3E The 3110 shown is the operating system hosted on the device 3150. In some implementations, the system (e.g., Figure 3E 3110 shown in the figure is an external device (e.g., a server, peripheral device, accessory and / or personal computing device) that includes an operating system.

[0143] refer to Figure 3C and Figure 3GApplication 3160 acquires information (e.g., 3030). In some embodiments, the information acquired at 3030 includes location information, time information, notification information, user information, environmental information, electronic device status information, weather information, media information, historical information, event information, hardware information, and / or motion information. In response to acquiring information at 3030 and / or after that step, application 3160 performs an operation on the information (e.g., 3040). In some embodiments, the operation performed at 3040 includes: providing notifications based on the information, sending messages based on the information, displaying information, controlling the user interface of a fitness application based on the information, controlling the user interface of a health application based on the information, controlling focus mode based on the information, setting reminders based on the information, adding calendar entries based on the information, and / or calling the API of system 3110 based on the information.

[0144] In some implementations, execution is performed in response to a trigger. Figure 3B Methods and / or Figure 3C The method involves one or more steps. In some implementations, triggering includes the detection of an event, a notification received from system 3110, user input, and / or a response to a call to an API provided by system 3110.

[0145] In some implementations, when the instructions of application 3160 are executed, control device 3150 executes them by calling an application programming interface (API) (e.g., API 3190) provided by system 3110. Figure 3B Methods and / or Figure 3C The method. In some implementations, application 3160 executes without calling API 3190. Figure 3B Methods and / or Figure 3C At least a part of the method.

[0146] In some implementation schemes, Figure 3B Methods and / or Figure 3C One or more steps of the method involve calling the API (e.g., API 3190) using one or more parameters defined by the API. In some implementations, one or more parameters include constants, keys, data structures, objects, object classes, variables, data types, pointers, arrays, lists, or pointers to functions or methods and / or references to data or other items to be passed via the API in another way.

[0147] refer to Figure 3D Example 3150 is shown. In some embodiments, device 3150 is a personal computing device, smartphone, smartwatch, fitness tracker, head-mounted display (HMD) device, media device, public utility, speaker, television, and / or tablet computer. Figure 3DAs illustrated, device 3150 includes application 3160 and operating system (e.g., Figure 3E System 3110 is shown in the diagram. Application 3160 includes application implementation module 3170 and API call module 3180. System 3110 includes API 3190 and implementation module 3100. It should be understood that device 3150, application 3160 and / or system 3110 may include... Figure 3D and Figure 3E The examples shown have more, fewer, and / or different components.

[0148] In some implementations, application implementation module 3170 includes a set of one or more instructions corresponding to one or more operations performed by application 3160. For example, when application 3160 is a messaging application, application implementation module 3170 may include operations for receiving and sending messages. In some implementations, application implementation module 3170 communicates with API calling module 3180 via API 3190 (in... Figure 3E (As shown in the figure) communicates with system 3110.

[0149] In some implementations, API 3190 is a software module (e.g., a set of computer-readable instructions) that provides an interface allowing different modules (e.g., API calling module 3180) to access and / or use one or more functions, methods, procedures, data structures, classes, and / or other services provided by implementation module 3100 of system 3110. For example, API calling module 3180 can access features of implementation module 3100 through one or more API calls or enablements (e.g., embodied by function or method calls) exposed by API 3190 (e.g., software and / or hardware modules capable of receiving, responding to, and / or transmitting API calls), and can pass data and / or control information via API calls or enablements using one or more parameters. In some implementations, API 3190 allows application 3160 to use services provided by a software development kit (SDK) library. In some implementations, application 3160 combines calls to functions or methods provided by the SDK library and API 3190, or uses data types or objects defined in the SDK library and provided by API 3190. In some implementations, API calling module 3180 makes API calls via API 3190 to access and use features of implementation module 3100 specified by API 3190. In such implementations, implementation module 3100 may return a value to API calling module 3180 via API 3190 in response to an API call. This value may report to application 3160 the capabilities or status of hardware components of device 3150, including those capabilities or statuses related to aspects such as input capabilities and status, output capabilities and status, processing capabilities, power status, storage capacity and status, and / or communication capabilities. In some implementations, API 3190 is implemented in part by firmware, microcode, or other low-level logic executed in part on the hardware components.

[0150] In some implementations, API 3190 allows the developer of API calling module 3180 (which may be a third-party developer) to utilize features provided by implementation module 3100. In such implementations, one or more API calling modules (e.g., including API calling module 3180) may exist that communicate with implementation module 3100. In some implementations, API 3190 allows multiple API calling modules written in different programming languages ​​to communicate with implementation module 3100 (e.g., API 3190 may include features for translating calls and returns between implementation module 3100 and API calling module 3180), and API 3190 is implemented in a specific programming language. In some implementations, API calling module 3180 calls APIs from different providers, such as a set of APIs from an OS provider, another set of APIs from a plugin provider, and / or another set of APIs from another provider (e.g., a software library provider) or the creator of another set of APIs.

[0151] Examples of API 3190 may include one or more of the following: pairing API (e.g., for establishing a secure connection, such as with an accessory), device detection API (e.g., for locating nearby devices, such as media devices and / or smartphones), payment API, UIKit API (e.g., for generating user interfaces), location detection API, locator API, map API, health sensor API, sensor API, messaging API, push notification API, streaming API, collaboration API, video conferencing API, app store API, advertising service API, web browser API (e.g., WebKit API), transportation API, networking API, WiFi API, Bluetooth API, NFC API, UWB API, fitness API, smart home API, contact transfer API, photo API, camera API, and / or image processing API. In some implementations, a sensor API is an API for accessing data associated with sensors of device 3150. For example, a sensor API may provide access to raw sensor data. Alternatively, a sensor API may provide data derived (and / or generated) from raw sensor data. In some implementations, sensor data includes temperature data, image data, video data, audio data, heart rate data, IMU (Inertial Measurement Unit) data, LiDAR data, location data, GPS data, and / or camera data. In some implementations, sensors include one or more of accelerometers, temperature sensors, infrared sensors, optical sensors, heart rate sensors, barometers, gyroscopes, proximity sensors, and / or biometric sensors.

[0152] In some embodiments, implementation module 3100 is a system (e.g., an operating system and / or server system) software module (e.g., a set of computer-readable instructions) configured to perform operations in response to receiving an API call via API 3190. In some embodiments, implementation module 3100 is configured to provide an API response (via API 3190) as a result of processing the API call. For example, implementation module 3100 and API call module 3180 can each be any of an operating system, library, device driver, API, application, or other module. It should be understood that implementation module 3100 and API call module 3180 can be the same or different types of modules. In some embodiments, implementation module 3100 is at least partially embodied in firmware, microcode, or hardware logic.

[0153] In some implementations, implementation module 3100 returns a value via API 3190 in response to an API call from API call module 3180. While API 3190 defines the syntax and results of the API call (e.g., how to enable the API call and what the API call does), API 3190 may not reveal how implementation module 3100 performs the functionality specified by the API call. Various API calls are passed via one or more application programming interfaces between API call module 3180 and implementation module 3100. Passing API calls may include issuing, initiating, enabling, calling, receiving, returning, and / or responding to function calls or messages. In other words, passing can describe the actions of API call module 3180 or implementation module 3100. In some implementations, function calls or other enablements of API 3190 transmit and / or receive one or more parameters via parameter lists or other structures.

[0154] In some implementations, implementation module 3100 provides more than one API, each API providing a different view or aspect of the functionality implemented by implementation module 3100. For example, one API of implementation module 3100 may provide a first set of functions and be exposed to third-party developers, while another API of implementation module 3100 may be hidden (e.g., not exposed) and provide a subset of the first set of functions, and also provide another set of functions, such as test or debug functions not in the first set of functions. In some implementations, implementation module 3100 calls one or more other components via lower-level APIs, thus acting as both an API calling module and an implementation module. It should be recognized that implementation module 3100 may include additional functions, methods, classes, data structures, and / or other features not specified through API 3190 and not available to API calling module 3180. It should also be recognized that API calling module 3180 may be on the same system as implementation module 3100, or may be remotely located and accessed via a network using API 3190. In some implementations, implementation module 3100, API 3190, and / or API calling module 3180 are stored in a machine-readable medium, which includes any means for storing information in a machine-readable (e.g., computer or other data processing system) form. For example, a machine-readable medium may include a magnetic disk, optical disk, random access memory, read-only memory, and / or flash memory devices.

[0155] An Application Programming Interface (API) is an interface between a first software process and a second software process, specifying the format for communication between the two processes. Limited APIs (e.g., private or partner APIs) are APIs accessible to a limited set of software processes (e.g., only software processes within the operating system or only software processes authorized to access the limited API). Public APIs are accessible to a broader set of software processes. Some APIs enable a software process to communicate or set the state of one or more input devices (e.g., one or more touch sensors, proximity sensors, vision sensors, motion / or orientation sensors, pressure sensors, intensity sensors, sound sensors, wireless proximity sensors, biometric sensors, buttons, switches, rotatable elements, and / or external controllers). Some APIs enable a software process to communicate and / or set the state of one or more output generation components (e.g., one or more audio output generation components, one or more display generation components, and / or one or more haptic output generation components). Some APIs enable specific capabilities (e.g., scrolling, handwriting, text input, image editing, and / or image creation) to be accessed, executed, and / or used by a software process (e.g., generating output for use by the software process based on input from the software process). Some APIs enable content from software processes to be inserted into templates and displayed in user interfaces with layouts and / or behaviors specified by the templates.

[0156] Many software platforms include a set of frameworks that provide core objects and behaviors that software developers need to build software applications that can be used on the platform. Software developers use these objects to display content on a screen, interact with that content, and manage interactions with the software platform. The basic behavior of a software application depends on this framework, and this framework provides software developers with numerous ways to customize the application's behavior to match the specific needs of the application. Many of these core objects and behaviors are accessed via APIs. APIs typically specify the format for communication between software processes, including specifying and grouping available variables, functions, and protocols. API calls (sometimes called API requests) are typically passed from a sending software process to a receiving software process as a way to achieve one or more of the following: the sending software process requests information from the receiving software process (e.g., for the sending software process to take an action); the sending software process provides information to the receiving software process (e.g., for the receiving software process to take an action); the sending software process requests an action from the receiving software process; or the sending software process provides information to the receiving software process about the action taken by the sending software process. In some cases, interaction with a device (e.g., using a user interface) will involve passing and / or receiving one or more API calls (e.g., multiple API calls) between multiple different software processes (e.g., different parts of an operating system, applications and operating systems, or different applications) via one or more APIs (e.g., via multiple different APIs). For example, when input is detected, direct sensor data is frequently processed into one or more input events, which are provided (e.g., via an API) to a receiving software process, which makes some determinations based on the input events and then (e.g., via an API) transmits information to the software process to perform an operation (e.g., change the device state and / or the user interface) based on the determinations. While the determinations and the operations performed in response can be made by the same software process, alternatively, the determinations can be made in a first software process and relayed (e.g., via an API) to a second software process different from the first software process, allowing the operation to be performed by the second software process. Alternatively, the second software process can relay instructions (e.g., via an API) to a third software process different from the first and / or second software processes to perform the operation. It should be understood that some or all user interactions with a computer system may involve one or more API calls within the steps of interacting with the computer system (e.g., between different software components of the computer system or between software components of the computer system and software components of one or more remote computer systems).It should be understood that some or all user interactions with a computer system may involve one or more API calls between steps of interaction with the computer system (e.g., between different software components of the computer system or between software components of the computer system and software components of one or more remote computer systems).

[0157] In some implementations, the application can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets or other applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, messaging applications and / or map applications.

[0158] In some embodiments, the application is an application pre-installed on the first computer system at the time of purchase (e.g., a first-party application). In some embodiments, the application is an application provided to the first computer system via an operating system update file (e.g., a first-party application). In some embodiments, the application is an application provided via an app store. In some embodiments, the app store is pre-installed on the first computer system at the time of purchase (e.g., a first-party app store) and allows the download of one or more applications. In some embodiments, the app store is a third-party app store (e.g., an app store provided by another device, downloaded via a network, and / or read from a storage device). In some embodiments, the application is a third-party application (e.g., an application provided by an app store, downloaded via a network, and / or read from a storage device). In some embodiments, the application controls the first computer system to execute method 700 by calling an application programming interface (API) provided by a system process using one or more parameters. Figure 7 ).

[0159] In some implementations, exemplary APIs provided by system processes include one or more of the following: pairing API (e.g., for establishing a secure connection, such as with an accessory), device detection API (e.g., for locating nearby devices, such as media devices and / or smartphones), payment API, UIKit API (e.g., for generating user interfaces), location detection API, locator API, map API, health sensor API, sensor API, messaging API, push notification API, streaming API, collaboration API, video conferencing API, app store API, advertising service API, web browser API (e.g., WebKit API), transportation API, networking API, WiFi API, Bluetooth API, NFC API, UWB API, fitness API, smart home API, contact transfer API, photo API, camera API, and / or image processing API.

[0160] In some embodiments, at least one API is a software module (e.g., a set of computer-readable instructions) that provides an interface allowing different modules (e.g., an API calling module) to access and use one or more functions, methods, procedures, data structures, classes, and / or other services provided by an implementation module of a system process. The API may define one or more parameters passed between the API calling module and the implementation module. In some embodiments, API 3190 defines a first API call that can be provided by API calling module 3180. An implementation module is a system software module (e.g., a set of computer-readable instructions) configured to perform operations in response to receiving an API call via the API. In some embodiments, the implementation module is configured to provide an API response (via the API) as a result of processing the API call. In some embodiments, the implementation module is included in a device (e.g., 3150) running an application. In some embodiments, the implementation module is included in an electronic device separate from the device running the application. Figure 4 This is a schematic illustration of an example embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1A ) by hand tracking unit 244 ( Figure 2 To control and track the location / position of one or more parts of the user's hand, and / or the location of one or more parts of the user's hand relative to the user's hand. Figure 1A The scenario 105 involves movement relative to a portion of the user's surrounding physical environment, relative to display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hand. In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0161] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures at least three-dimensional scene information including the human user's hand 406. The image sensor 404 captures images of the hand at sufficient resolution to distinguish the fingers and their corresponding positions. The image sensor 404 typically captures images of other parts of the user's body, or possibly all parts of the body, and may have scaling capabilities or be a dedicated sensor with increased magnification to capture images of the hand at the desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of scene 105, or serves as the image sensor for capturing the physical environment of scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a way that uses the field of view of the image sensor 404 or a portion thereof to define an interaction space in which hand movements captured by the image sensor are considered input to the controller 110.

[0162] In some implementations, image sensor 404 outputs a sequence of frames containing 3D image data (and, in addition, possibly color image data) to controller 110, which extracts high-level information from the image data. This high-level information is typically provided via an application programming interface (API) to an application running on the controller, which in turn drives display generation component 120. For example, a user can interact with software running on controller 110 by moving his hand 406 and changing his hand pose.

[0163] In some embodiments, image sensor 404 projects a speckle pattern onto a scene containing hand 406 and captures an image of the projected pattern. In some embodiments, controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) via triangulation based on the lateral offset of the specks in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. This method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from image sensor 404. In this disclosure, it is assumed that image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis such that the depth coordinates of points in the scene correspond to the z-component measured by the image sensor. Alternatively, image sensor 404 (e.g., a hand-tracking device) may use other 3D mapping methods, such as stereo imaging or time-of-flight measurement, based on a single or multiple cameras or other types of sensors.

[0164] In some implementations, hand tracking device 140 captures and processes time-series depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on a processor in image sensor 404 and / or controller 110 processes the 3D map data to extract image block descriptors of the hand from these depth maps. The software may match these descriptors with image block descriptors stored in database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and fingertips.

[0165] The software can also analyze the trajectories of the hand and / or fingers across multiple frames in a sequence to identify gestures. The pose estimation function described herein can be alternated with motion tracking, such that patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to find pose changes occurring in the remaining frames. Pose, motion, and gesture information is provided to an application running on controller 110 via the aforementioned API. The program can, for example, move and modify the image presented on display generation unit 120 in response to pose and / or gesture information, or perform other functions.

[0166] In some implementations, gestures include air gestures. An air gesture is a gesture detected without the user touching an input element that is part of the device (e.g., computer system 101, one or more input devices 125 and / or hand tracking device 140) (or independent of an input element that is part of the device) and based on the detected movement of a part of the user's body (e.g., head, one or two arms, one or two hands, one or more fingers and / or one or two legs) through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., including a tapping gesture in which the hand moves a predetermined amount and / or speed in a predetermined pose, or a shaking gesture including a predetermined speed or amount of rotation of a part of the user's body)).

[0167] In some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures for interacting with an XR environment (e.g., a virtual or mixed reality environment) performed by the movement of a user's fingers relative to other fingers or portions of the user's hand. In some embodiments, air gestures are detected without the user touching an input element that is part of the device (or independently of an input element that is part of the device) and are based on the detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or portion of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tapping gesture that includes the hand moving a predetermined amount and / or speed in a predetermined pose, or a shaking gesture that includes a predetermined speed or amount of rotation of a part of the user's body)).

[0168] In some implementations where the input gesture is an air gesture (e.g., where, in the absence of physical contact with the input device, the input device provides the computer system with information about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touchscreen, or contact with a mouse or touchpad to move the cursor to the user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Therefore, in specific implementations involving air gestures, for example, the input gesture is combined with (e.g., simultaneously) movement of the user's fingers and / or hand to detect attention (e.g., gaze) toward a user interface element to perform pinch and / or tap input, as described below.

[0169] In some implementations, input gestures directed to a user interface object are performed, either directly or indirectly, by referencing the user interface object. For example, user input is performed directly on the user interface object based on the user's hand performing an input gesture at a location corresponding to the user interface object's position in the three-dimensional environment (e.g., determined based on the user's current viewpoint). In some implementations, when user attention to the user interface object (e.g., gazing) is detected, input gestures are performed indirectly on the user interface object based on the user's hand not being positioned at a location corresponding to the user interface object's position in the three-dimensional environment while the user is performing the input gesture. For example, for direct input gestures, the user can guide their input to the user interface object by initiating a gesture at or near a location corresponding to the user interface object's display position (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge or center of the option). For indirect input gestures, the user can guide their input to the user interface object by focusing on it (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location not corresponding to the user interface object's display position).

[0170] In some implementations, the input gestures (e.g., air gestures) used in the various examples and implementations described herein include pinch input and tap input for interacting with a virtual or mixed reality environment. For example, the pinch input and tap input described below are performed as air gestures.

[0171] In some implementations, pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes the movement of two or more fingers of the hand to contact each other, i.e., optionally followed by an immediate (e.g., within 0 to 1 second) interruption of contact. A long pinch gesture as an air gesture includes the movement of two or more fingers of the hand to contact each other for at least a threshold amount of time (e.g., at least 1 second) before an interruption of contact is detected. For example, a long pinch gesture includes the user holding a pinch gesture (e.g., where two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some implementations, a double pinch gesture as an air gesture includes two (e.g., more) pinch inputs (e.g., performed by the same hand) that are detected consecutively with each other immediately (e.g., within a predefined time period). For example, a user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts the contact between two or more fingers), and performs a second pinch input within a predefined time period after releasing the first pinch input (e.g., within 1 second or within 2 seconds).

[0172] In some embodiments, pinch-and-drag gestures as air gestures (e.g., air drag gestures or air swipe gestures) include pinch gestures (e.g., pinch gestures or long pinch gestures) performed in conjunction with (e.g., following) a drag input that changes the user's hand position from a first position (e.g., the start position of the drag) to a second position (e.g., the end position of the drag). In some embodiments, the user holds the pinch gesture while performing the drag input and releases the pinch gesture (e.g., opening two or more of their fingers) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and drag input are performed by the same hand (e.g., the user pinches two or more fingers together to touch each other and uses the drag gesture to move the same hand to the second position in the air). In some implementations, pinch input is performed by the user's first hand, and drag input is performed by the user's second hand (e.g., while the user continues pinch input with the user's first hand, the user's second hand moves in the air from a first position to a second position). In some implementations, input gestures as air gestures include inputs performed using both of the user's hands (e.g., pinch and / or tap inputs). For example, input gestures include two (e.g., more) pinch inputs performed in combination with each other (e.g., concurrently or within a predefined time period). For example, a first pinch gesture (e.g., pinch input, long pinch input, or pinch and drag input) is performed using the user's first hand, and a second pinch input is performed using the other hand (e.g., the second hand in the user's two hands).

[0173] In some implementations, a tap input performed as an air gesture (e.g., pointing at a user interface element) includes movement of a user's finger toward the user interface element, movement of the user's hand toward the user interface element (optionally, the user's finger extends toward the user interface element), downward movement of the user's finger (e.g., mimicking a mouse click or a tap on a touchscreen), or other predefined movements of the user's hand. In some implementations, the tap input performed as an air gesture is detected based on the movement characteristics of the finger or hand performing the tap gesture movement, which is the finger or hand moving away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by the end of the movement. In some implementations, the end of the movement is detected based on changes in the movement characteristics of the finger or hand performing the tap gesture (e.g., the end of movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of finger or hand movement, and / or a reversal of the acceleration direction of finger or hand movement).

[0174] In some implementations, the user's attention is determined to be directed to a portion of the 3D environment based on the detection of a gaze directed to that portion of the 3D environment (optionally, no other conditions are required). In some implementations, the user's attention is determined to be directed to that portion of the 3D environment based on the detection of a gaze directed to that portion of the 3D environment using one or more additional conditions, such as requiring the gaze to be directed to that portion of the 3D environment for at least a threshold duration (e.g., dwell time) and / or requiring the gaze to be directed to that portion of the 3D environment when the user's viewpoint is within a distance threshold from that portion of the 3D environment, so that the device determines that the user's attention is directed to that portion of the 3D environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed to the portion of the 3D environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0175] In some implementations, the detection of the readiness configuration of a user or a portion of a user is performed by a computer system. The detection of the hand's readiness configuration is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the readiness of the hand is determined based on whether it has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart in preparation for a pinch or grasping gesture, or a pre-tap shape with one or more fingers extended and the back of the hand facing the user), whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular manner (e.g., moving towards the area in front of the user above the user's waist and below the user's head, or moving away from the user's body or legs). In some implementations, the readiness state is used to determine whether an interactive element of the user interface responds to attentional (e.g., gaze) input.

[0176] In scenarios where input is described by reference to air gestures, it should be understood that hardware input devices attached to or held by one or both of the user's hands can be used to detect such gestures. Optical tracking, one or more accelerometers, one or more gyroscopes, one or more magnetometers, and / or one or more inertial measurement units can be used to track the spatial positioning of the hardware input device, and the positioning and / or movement of the hardware input device can be used in place of the positioning and / or movement of the one or two hands corresponding to the air gesture. Similarly, in scenarios where input is described by reference to air pose, it should be understood that hardware input devices attached to or held by one or both of the user's hands can be used to detect such poses. User input can be detected using controls contained in hardware input devices, such as one or more touch-sensitive input elements, one or more pressure-sensitive input elements, one or more buttons, one or more knobs, one or more dials, one or more joysticks, a hand or finger cover that can detect the position or positional change of a portion of a hand and / or finger relative to each other, relative to the user's body, and / or relative to the user's physical environment, and / or other hardware input device controls, wherein user input using controls contained in the hardware input device replaces hand and / or finger gestures such as air taps or air pinches in corresponding air gestures. For example, a selection input described as being performed using an air tap or air pinch input can alternatively be detected using button presses, taps on touch-sensitive surfaces, presses on pressure-sensitive surfaces, or other hardware inputs. For example, motion input described as being performed using air pinching and dragging (e.g., air dragging gestures or air swipe gestures) can alternatively be detected based on interaction with hardware input controls (such as pressing and holding a button, touching on a touch-sensitive surface, pressing on a pressure-sensitive surface, or other hardware input following movement of a hardware input device (e.g., a hand associated with the hardware input device) through space). Similarly, two-handed input involving movement of hands relative to each other can be performed using an air gesture and a hardware input device not in the hand performing the air gesture, two hardware input devices held in different hands, or two air gestures performed by different hands using air gestures and / or inputs detected by one or more of the aforementioned hardware input devices.

[0177] In some embodiments, the software may be downloaded to controller 110 electronically, for example, via a network, or alternatively, may be provided on a tangible, non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, database 408 is also stored in memory associated with controller 110. Alternatively or additionally, some or all of the described functions of the computer may be implemented in dedicated hardware, such as custom or semi-custom integrated circuits or programmable digital signal processors (DSPs). Although in Figure 4The controller 110 is shown, but for example, as a separate unit from the image sensor 404, some or all of the controller's processing functions may be performed by a suitable microprocessor and software, or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand-tracking device), or by other devices associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing function of the image sensor 404 may also be integrated into a computer or other computerized device controlled by the sensor output.

[0178] Figure 4 It also includes a schematic diagram of a depth map 410 captured by image sensor 404 according to some embodiments. As described above, the depth map comprises a matrix of pixels with corresponding depth values. Pixel 412 corresponding to hand 406 has been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z-distance from image sensor 404), where gray shadows become darker as depth increases. Controller 110 processes these depth values ​​to identify and segment components of the image that have human hand characteristics (i.e., a group of adjacent pixels). These characteristics may include, for example, overall size, shape, and frame-to-frame motion from the depth map sequence.

[0179] Figure 4 The controller 110 also schematically illustrates, according to some embodiments, the hand skeleton 414 ultimately extracted from the depth map 410 of the hand 406. Figure 4 In this configuration, the hand skeleton 414 is superimposed on the hand background 416, which has already been segmented from the original depth map. In some embodiments, key feature points of the hand, and optionally those on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, fingertips, the center of the palm, the end of the hand connecting to the wrist, etc.), are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points across multiple image frames to determine, according to some embodiments, the gesture performed by the hand or the current state of the hand.

[0180] Figure 5A An eye-tracking device 130 is illustrated. Figure 1A Example implementation of ). In some implementations, the eye-tracking device 130 consists of an eye-tracking unit 243 ( Figure 2The eye-tracking device 130 is controlled to track the positioning and movement of a user's gaze relative to scene 105 or relative to XR content displayed via display generation component 120. In some embodiments, the eye-tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as a head-mounted device, helmet, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating XR content for the user to view and components for tracking the user's gaze relative to the XR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or an XR room, the eye-tracking device 130 is optionally a separate device from the handheld device or XR room. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or not head-mounted. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally used in conjunction with head-mounted display generation components. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally part of non-head-mounted display generation components.

[0181] In some embodiments, the display generation unit 120 uses display mechanisms (e.g., a left near-eye display panel and a right near-eye display panel) to display frames including left and right images in front of the user's eyes, thereby providing the user with a 3D virtual view. For example, the head-mounted display generation unit may include left and right optical lenses (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation unit may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation unit may have a transparent or semi-transparent display on which virtual objects are displayed, allowing the user to view the physical environment directly through the transparent or semi-transparent display. In some embodiments, the display generation unit projects virtual objects onto the physical environment. The virtual objects may be projected, for example, onto a physical surface or as holograms, allowing an individual to observe virtual objects superimposed on the physical environment using the system. In this case, separate display panels and image frames for the left and right eyes may not be necessary.

[0182] like Figure 5AAs shown, in some embodiments, eye-tracking device 130 (e.g., gaze tracking device) includes at least one eye-tracking camera (e.g., an infrared (IR) or near-infrared (NIR) camera) and an illumination source (e.g., an array or ring of IR or NIR light sources, such as LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be pointed at the user's eye to receive IR or NIR light reflected directly from the eye, or alternatively, it may be pointed at "hot" mirrors located between the user's eye and the display panel, which reflect the IR or NIR light from the eye back to the eye-tracking camera while allowing visible light to pass through. Eye-tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps), analyzes these images to generate gaze tracking information, and transmits the gaze tracking information to controller 110. In some embodiments, the user's two eyes are tracked separately using corresponding eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked using corresponding eye-tracking cameras and illumination sources.

[0183] In some implementations, a device-specific calibration procedure is used to calibrate the eye-tracking device 130 to determine parameters for the eye-tracking device in a specific operating environment 100, such as the 3D geometry and parameters of the LEDs, camera, thermal mirror (if present), eye lenses, and display. The device-specific calibration procedure can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration procedure can be automated or manual. According to some implementations, a user-specific calibration procedure may include estimations of eye parameters for a specific user, such as pupil position, foveal position, optical axis, visual axis, interocular distance, etc. According to some implementations, once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, a flash-assisted method can be used to process the images captured by the eye-tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0184] like Figure 5AAs shown, the eye-tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system. The gaze tracking system includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-infrared (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye-tracking camera 540 may be pointed toward a mirror 550 located between the user's eye 592 and a display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, projector, etc.). These mirrors reflect the IR or NIR light from the eye 592 while allowing visible light to pass through. Figure 5A (as shown in the top portion), or alternatively, it can be pointed towards the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion), Figure 5A (As shown in the bottom part).

[0185] In some implementations, controller 110 renders AR or VR frames 562 (e.g., left and right frames for the left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye-tracking camera 540 for various purposes, such as processing frame 562 for display. Controller 110 optionally estimates the user's gaze point on display 510 based on the gaze tracking input 542 obtained from eye-tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated based on gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0186] The following describes several possible use cases for the user's current gaze direction and is not intended to be limiting. As an example use case, controller 110 may render virtual content differently based on the determined user gaze direction. For example, controller 110 may generate virtual content at a higher resolution in the concave area determined according to the user's current gaze direction than in the peripheral area. As another example, the controller may position or move virtual content in the view based at least partially on the user's current gaze direction. As yet another example, the controller may display specific virtual content in the view based at least partially on the user's current gaze direction. As another example use case in AR applications, controller 110 may guide an external camera used to capture the physical environment of an XR experience to focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface in the environment that the user is currently looking at on display 510. As another example use case, eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of eye lens 520 so that the virtual object the user is currently looking at has appropriate convergence / divergence to match the convergence of the user's eyes 592. The controller 110 can use gaze tracking information to guide the eye lens 520 to adjust its focus so that the nearby object that the user is looking at appears at the correct distance.

[0187] In some embodiments, the eye-tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lens 520), an eye-tracking camera (e.g., eye-tracking camera 540), and a light source (e.g., illumination source 530 (e.g., IR or NIR LED)). The light source emits light (e.g., IR or NIR light) toward the user's eyes 592. In some embodiments, the light source may be arranged in a ring or circle around each lens in the head-mounted device, such as... Figure 5A As shown. In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0188] In some embodiments, the display 510 emits light in the visible light range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. It should be noted that the positions and angles of the eye-tracking camera 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, cameras 540 with a wider field of view (FOV) and cameras 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, cameras 540 operating at one wavelength (e.g., 850 nm) and cameras 540 operating at different wavelengths (e.g., 940 nm) may be used on each side of the user's face.

[0189] like Figure 5A The illustrated gaze tracking system implementation can be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide users with computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experiences.

[0190] Figure 5B Examples of flash-assisted gaze tracking pipelines according to some embodiments are illustrated. In some embodiments, the gaze tracking pipeline uses a flash-assisted gaze tracking system (e.g., such as...) Figure 1A This is achieved using the eye-tracking device 130 illustrated in Figure 5. The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and flash in the current frame. When not in tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and flash in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in tracking state.

[0191] like Figure 5B As shown, the gaze-tracking camera captures left and right images of the user's left and right eyes. The captured images are then fed into a gaze-tracking pipeline for processing to begin at 510b. As indicated by the arrow returning to element 500b, the gaze-tracking system can continue capturing images of the user's eyes, for example, at a rate of 60 to 120 frames per second. In some embodiments, each set of captured images can be fed into the pipeline for processing. However, in some embodiments or under certain conditions, not all captured frames are processed by the pipeline.

[0192] At 510b, for the currently captured image, if the tracking status is "Yes", the method proceeds to element 540b. At 510b, if the tracking status is "No", the image is analyzed to detect the user's pupil and flash in the image, as indicated at 520b. At 530b, if the pupil and flash are successfully detected, the method proceeds to element 540b. Otherwise, the method returns to element 510b to process the next image of the user's eye.

[0193] At 540b, if proceeding from element 510b, the current frame is analyzed to track the pupil and flashes in part based on previous information from the previous frame. At 540b, if proceeding from element 530b, the tracking state is initialized based on the pupils and flashes detected in the current frame. The processing result at element 540b is checked to verify the reliability of the tracking or detection result. For example, the result can be checked to determine whether the pupils and a sufficient number of flashes used to perform gaze estimation were successfully tracked or detected in the current frame. At 550b, if the result is not reliable, the tracking state is set to "No" at element 560b, and the method returns to element 510b to process the next image of the user's eye. At 550b, if the result is reliable, the method proceeds to element 570b. At 570b, the tracking state is set to "Yes" (if it is not already "Yes"), and the pupil and flash information is passed to element 580b to estimate the user's gaze point.

[0194] Figure 5B This is intended as an example of an eye-tracking technology that can be used in a particular specific implementation. As will be recognized by those skilled in the art, in a computer system 101 for providing an XR experience to a user, other eye-tracking technologies that are currently available or will be developed in the future may be used to replace or in combination with the flash-assisted eye-tracking technology described herein, depending on the various implementations.

[0195] In some implementations, a captured portion of the real-world environment is used to provide an XR experience to the user, such as a mixed reality environment in which one or more virtual objects are overlaid on a representation of the real-world environment.

[0196] Therefore, this description describes some embodiments of a three-dimensional environment (e.g., an XR environment) that includes representations of real-world objects and virtual objects. For example, the three-dimensional environment optionally includes a representation of a table existing in a physical environment, which is captured and displayed in the three-dimensional environment (e.g., actively displayed via a camera and display of a computer system or passively displayed via a transparent or semi-transparent display of a computer system). As previously described, the three-dimensional environment is optionally a mixed reality system, wherein the three-dimensional environment is based on a physical environment captured by one or more sensors of a computer system and displayed via a display generation component. As a mixed reality system, the computer system is optionally capable of selectively displaying portions and / or objects of the physical environment such that the corresponding portions and / or objects of the physical environment appear as if they exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system is optionally capable of displaying virtual objects in the three-dimensional environment to appear as if the virtual objects exist in the real world (e.g., the physical environment) by placing virtual objects in the three-dimensional environment at corresponding locations in the real world that have corresponding positions in the three-dimensional environment. For example, the computer system optionally displays a vase such that the vase appears as if a real vase were placed on top of a table in the physical environment. In some implementations, a corresponding location in the three-dimensional environment has a corresponding location in the physical environment. Therefore, when a computer system is described as displaying a virtual object at a corresponding location relative to a physical object (e.g., such as at or near a user's hand or at or near a physical table), the computer system displays the virtual object at a specific location in the three-dimensional environment such that it appears as if the virtual object were at or near a physical object in the physical environment (e.g., the virtual object is displayed in the three-dimensional environment at a location in the physical environment that would be displayed if the virtual object were a real object at that specific location).

[0197] In some implementations, real-world objects that exist in a physical environment and are displayed in a 3D environment (e.g., and / or visible via display-generated components) can interact with virtual objects that exist only in the 3D environment. For example, the 3D environment may include a table and a vase placed on top of the table, where the table is a view (or representation) of a physical table in the physical environment, and the vase is a virtual object.

[0198] In a three-dimensional environment (e.g., a real environment, a virtual environment, or a hybrid environment including both real and virtual objects), an object is sometimes referred to as having depth or simulated depth, or as being visible, displayed, or placed at different depths. In this context, depth refers to a dimension other than height or width. In some embodiments, depth is defined relative to a fixed set of coordinates (e.g., where a room or object has a height, depth, and width defined relative to a fixed set of coordinates). In some embodiments, depth is defined relative to a user's position or viewpoint, in which case the depth dimension varies based on the user's position and / or the position and angle of the user's viewpoint. In some embodiments where depth is defined relative to the user's location relative to a surface of the environment (e.g., the surface of the environment's floor or ground), objects further away from the user along lines extending parallel to the surface are considered to have greater depth in the environment, and / or the depth of an object is measured along an axis extending outward from the user's position and parallel to the surface of the environment (e.g., depth is defined in a cylindrical or substantially cylindrical coordinate system, where the user's position is at the center of a cylinder extending from the user's head toward the user's feet). In some embodiments where depth is defined relative to the user's viewpoint (e.g., a direction relative to a point in space that determines which part of the environment is visible via a head-mounted device or other display), objects further away from the user's viewpoint along a line extending parallel to the user's viewpoint are considered to have greater depth in the environment, and / or the depth of an object is measured along an axis extending outward from a line extending from and parallel to the user's viewpoint (e.g., defining depth in a spherical or substantially spherical coordinate system, where the origin of the viewpoint is at the center of a sphere extending outward from the user's head). In some embodiments, depth is defined relative to a user interface container (e.g., a window or application displaying application and / or system content), where the user interface container has a height and / or width, and depth is a dimension orthogonal to the height and / or width of the user interface container. In some implementations, when a depth is defined relative to a user interface container, when the container is placed in a three-dimensional environment or initially displayed (e.g., such that the container's depth dimension extends outward away from the user or the user's viewpoint), the container's height and / or width are typically orthogonal or substantially orthogonal to a straight line extending from the user's location (e.g., the user's viewpoint or the user's position) to the user interface container (e.g., the center of the user interface container or another feature point of the user interface container). In some implementations, when a depth is defined relative to a user interface container, the object's depth relative to the user interface container refers to the object's positioning along the depth dimension of the user interface container. In some implementations, multiple different containers may have different depth dimensions (e.g., different depth dimensions extending away from the user or the user's viewpoint in different directions and / or from different starting points).In some implementations, when depth is defined relative to a user interface container, the orientation of the depth dimension remains constant relative to the user interface container as the position of the user interface container changes, or as the user and / or the user's viewpoint changes (e.g., when multiple different viewers are viewing the same container in a 3D environment, such as during a collaborative session and / or when multiple participants are in a real-time communication session with shared virtual content including the container). In some implementations, for curved containers (e.g., containers including those with curved surfaces or curved content areas), the depth dimension optionally extends into the surface of the curved container. In some cases, z-interval (e.g., the distance between two objects in the depth dimension), z-height (e.g., the distance of one object from another in the depth dimension), z-position (e.g., the position of an object in the depth dimension), z-depth (e.g., the position of an object in the depth dimension), or simulated z-dimensionality (e.g., depth used as a dimension of an object, a dimension of the environment, an orientation in space, and / or an orientation in simulated space) are used to refer to the concept of depth as described above.

[0199] In some implementations, a user may optionally be able to interact with virtual objects in a three-dimensional environment using one or both hands as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the computer system may optionally capture one or both of the user's hands and display a representation of the user's hands in the three-dimensional environment (e.g., in a manner similar to displaying real-world objects in the three-dimensional environment described above). Alternatively, in some implementations, the user's hands may be seen via the display generating component, through the ability to see the physical environment through the user interface, due to the transparency / semi-transparency of a portion of the user interface being displayed by the display generating component, or due to the projection of the user interface onto a transparent / semi-transparent surface or onto the user's eyes or into the user's field of view. Thus, in some implementations, the user's hands are displayed at corresponding locations in the three-dimensional environment and are treated as if they were objects in the three-dimensional environment that can interact with virtual objects in the three-dimensional environment as if these virtual objects were physical objects in the physical environment. In some implementations, the computer system may update the display of the user's hand representation in the three-dimensional environment in conjunction with the movement of the user's hands in the physical environment.

[0200] In some embodiments described below, the computer system optionally determines the “effective” distance between a physical object in the physical world and a virtual object in a three-dimensional environment, for example, to determine whether a physical object is directly interacting with a virtual object (e.g., whether a hand is touching, grasping, holding, or within a threshold distance of a virtual object). For example, a hand directly interacting with a virtual object optionally includes one or more of the following: a finger pressing a virtual button, a user’s hand grasping a virtual vase, a user’s hand clasped together and pinching / holding the application’s user interface, and two fingers performing any other type of interaction described herein. For example, the computer system optionally determines the distance between a user’s hand and a virtual object when determining whether and / or how a user is interacting with a virtual object. In some embodiments, the computer system determines the distance between a user’s hand and a virtual object by determining the distance between the position of a hand in the three-dimensional environment and the position of the virtual object of interest in the three-dimensional environment. For example, a user's one or both hands are located at a specific location in the physical world. The computer system optionally captures the one or both hands and displays them at a specific corresponding location in a three-dimensional environment (e.g., the location where the hand would be displayed in the three-dimensional environment if it were a virtual hand rather than a physical hand). Optionally, the location of the hand in the three-dimensional environment is compared with the location of a virtual object of interest in the three-dimensional environment to determine the distance between the user's one or both hands and the virtual object. In some embodiments, the computer system optionally determines the distance between a physical object and a virtual object by comparing locations in the physical world (e.g., rather than comparing locations in the three-dimensional environment). For example, when determining the distance between a user's one or both hands and a virtual object, the computer system optionally determines the corresponding location of the virtual object in the physical world (e.g., the location where the virtual object would be located in the physical world if it were a physical object rather than a virtual object), and then determines the distance between the corresponding physical location and the user's one or both hands. In some embodiments, the same technique is optionally used to determine the distance between any physical object and any virtual object. Therefore, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, the computer system may optionally perform any of the techniques described above to map the position of the physical object to the three-dimensional environment and / or map the position of the virtual object to the physical environment.

[0201] In some implementations, the same or similar techniques are used to determine where and what the user's gaze is directed at, and / or where and what the physical stylus held by the user is pointing at. For example, if the user's gaze is directed at a specific location in the physical environment, the computer system optionally determines a corresponding location in the three-dimensional environment (e.g., a virtual location of the gaze), and if a virtual object is located at that corresponding virtual location, the computer system optionally determines that the user's gaze is directed at that virtual object. Similarly, the computer system may optionally be able to determine the direction in which the stylus is pointing in the physical environment based on the orientation of the physical stylus. In some implementations, based on this determination, the computer system determines a corresponding virtual location in the three-dimensional environment corresponding to the location pointed at by the stylus in the physical environment, and optionally determines that the stylus is pointing at the corresponding virtual location in the three-dimensional environment.

[0202] Similarly, the embodiments described herein may refer to the location of a user (e.g., a user of a computer system) in a three-dimensional environment and / or the location of the computer system in a three-dimensional environment. In some embodiments, the user of the computer system is holding, wearing, or otherwise located at or near the computer system. Thus, in some embodiments, the location of the computer system serves as a proxy for the location of the user. In some embodiments, the location of the computer system and / or the user in the physical environment corresponds to a corresponding location in the three-dimensional environment. For example, the location of the computer system would be its location in the physical environment (and its corresponding location in the three-dimensional environment) such that, if the user stands at that location facing the corresponding portion of the physical environment visible via the display generation component, the user will see from that location objects in the physical environment that are positioned, oriented, and / or sized (e.g., in an absolute sense and / or relative to each other) in the same way as objects displayed or visible in the three-dimensional environment by or via the display generation component of the computer system. Similarly, if the virtual objects displayed in a 3D environment are physical objects in a physical environment (e.g., physical objects placed in the physical environment at the same location as these virtual objects in the 3D environment, and physical objects in the physical environment having the same size and orientation as in the 3D environment), then the position of the computer system and / or the user is the position from which the user will see these virtual objects in the physical environment at the same location, orientation, and / or size (e.g., in an absolute sense and / or relative to each other and real-world objects) as the virtual objects displayed in the 3D environment by the display generation components of the computer system.

[0203] In this disclosure, various input methods are described in relation to interaction with a computer system. When an example is provided using one input device or method, and another example is provided using another input device or method, it should be understood that each example is compatible with and optionally utilizes the input device or method described with respect to the other example. Similarly, various output methods are described in relation to interaction with a computer system. When an example is provided using one output device or method, and another example is provided using another output device or method, it should be understood that each example is compatible with and optionally utilizes the output device or method described with respect to the other example. Similarly, various methods are described in relation to interaction with a virtual or mixed reality environment via a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and optionally utilizes the methods described with respect to the other example. Therefore, this disclosure discloses embodiments that are combinations of features of a plurality of examples without exhaustively listing all features of the embodiments in the description of each example embodiment.

[0204] Figure 5C An example of an exemplary device is provided for performing techniques for controlling one or more objects within and / or integrated with the platform in response to changes in the environment outside the platform and / or changes within the platform. Figures 6A to 6P Examples are illustrated of electronic devices that control one or more objects within and / or integrated with the platform in response to changes in the environment outside the platform and / or changes within the platform, according to some implementation schemes. Figures 6A to 6G The user interface in the document is used to illustrate the processes described below, including Figures 7 to 8 The process in.

[0205] The process described below describes various techniques for making user interfaces and / or human-computer interaction more efficient (e.g., by helping users provide input quickly and easily and preventing user errors when operating the device). These techniques sometimes reduce the amount of input required for users (e.g., people and / or users) to perform actions, provide users with clear and / or meaningful feedback (e.g., visual, acoustic, and / or haptic feedback) so that users know what is happening or what is expected, provide additional information and controls without cluttering the user interface, and / or perform certain actions without further input from the user. Because users can use the device more quickly and easily, these techniques sometimes extend battery life and / or reduce the device's power consumption.

[0206] In a method described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the described method may be repeated in multiple repetitions such that, during the repetitions, all the conditions determining the steps in the method have been satisfied in different repetitions of the method. For example, if the method requires performing a first step (if the conditions are satisfied) and a second step (if the conditions are not satisfied), it should be understood that these steps are repeated until both conditions are satisfied and not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each of the conditions described in the method has been satisfied. However, system or computer-readable medium claims do not require such multiple repetitions, wherein the system or computer-readable medium contains instructions for performing conditional operations that require the satisfaction of one or more conditions prior to the operation. Those skilled in the art will also understand that, similar to a method having conditional steps, a system or computer-readable storage medium may repeat the steps of the method multiple times as needed to ensure that all conditional steps have been performed.

[0207] The terminology used in the description of the various implementation schemes is for the purpose of describing a particular implementation scheme only and is not intended to be limiting.

[0208] The following describes user interfaces for electronic devices and associated processes for using these devices. In some embodiments, the device may be a desktop computer with a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). In other embodiments, the device is a portable, mobile, and / or mobile electronic device (e.g., a processor, smartphone, smartwatch, tablet, fitness tracker, laptop computer, head-mounted display (HMD) device, public utility, vehicle, media device, smart speaker, smart display, robot, television, and / or personal computing device).

[0209] In some embodiments, the electronic device is a computer system that communicates with the display component (e.g., via wireless or wired communication). The display component may be integrated into the computer system or may be separate from the computer system. Additionally, the display component may be configured to provide visual output to a display (e.g., a liquid crystal display, an OLED display, or a CRT display). As used herein, "display" content includes content that causes a display (e.g., video data rendered or decoded by a display controller) by transmitting data (e.g., image data or video data) to an integrated or external display component via a wired or wireless connection to visually generate content. In some embodiments, visual output is any output that can be perceived by the human eye, including but not limited to images, videos, graphics, charts, and other graphical representations of data.

[0210] In some implementations, the electronic device is a computer system that communicates with the audio generation component (e.g., via wireless or wired communication). The audio generation component may be integrated into the computer system or may be separate from it. Additionally, the audio generation component may be configured to provide audio output. Examples of audio generation components include speakers, home theater systems, soundbars, headphones, earphones, earbuds, television speakers, augmented reality headset speakers, audio jacks, optical audio outputs, Bluetooth audio outputs, and / or HDMI audio outputs. In some implementations, the audio output is any output perceptible to the human ear, including but not limited to sound waves, music, speech, and / or other audible representations of data.

[0211] In the following discussion, an electronic device including specific input and output devices is described. However, it should be understood that the electronic device may optionally include one or more other input and / or output devices, such as physical user interface devices (e.g., physical keyboard, mouse, and / or joystick).

[0212] Figure 5C An example system 100c is illustrated for implementing the techniques described herein. System 100c is executable. Figure 7 and / or Figure 8 Any of the methods described herein (e.g., method 700 and / or method 800) or a portion thereof.

[0213] In Figure 1, system 100c includes various components such as processor 103, RF circuitry 105c, memory 107, sensors 156 (e.g., image sensors, orientation sensors, position sensors, heart rate monitors, temperature sensors), input components 158 (e.g., cameras (e.g., periscope cameras, telephoto cameras, wide-angle cameras, and / or ultra-wide-angle cameras), depth sensors, microphones, touch-sensitive surfaces, hardware input mechanisms, and / or rotatable input mechanisms), mobility components (e.g., actuators (e.g., pneumatic actuators, hydraulic actuators, and / or electric actuators), motors, wheels, movable bases, rotatable components, translational components, and / or rotatable bases), and output components 160 (e.g., speakers, display components, audio generation components, haptic output devices, displays, projectors, and / or touch-sensitive displays). These components optionally communicate via the system's communication bus 123. Although shown as separate components, in some specific embodiments, various components may be combined and used as a single component; for example, a sensor may be an input component.

[0214] In some implementations, system 100c is a mobile and / or mobile device (e.g., a tablet, smartphone, laptop, head-mounted display (HMD) device, and / or smartwatch). In other implementations, system 100c is a desktop computer, embedded computer, and / or server.

[0215] In some embodiments, processor 103 includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some embodiments, memory 107 is one or more non-transitory computer-readable storage media (e.g., flash memory and / or random access memory) storing computer-readable instructions configured to be executed by processor 103 to perform the techniques described herein.

[0216] In some embodiments, RF circuit 105c includes circuitry for communicating with electronic devices and / or networks (e.g., the Internet, intranets, and / or wireless networks such as cellular networks and wireless local area networks (LANs)). In some embodiments, RF circuit 105c includes circuitry for using near-field communication and / or short-range communication (such as Bluetooth). ® Circuits that communicate (or ultra-wideband).

[0217] In some embodiments, display 121 includes one or more monitors, projectors, and / or screens. In some embodiments, display 121 includes a first display for displaying an image to a user's first eye and a second display for displaying an image to a user's second eye. In such embodiments, corresponding images may be displayed simultaneously on the first and second displays. Optionally, the corresponding images include representations of the same virtual objects and / or the same physical objects viewed from different viewpoints, thereby creating a parallax effect that provides the user with the illusion that objects have depth on the display. In some embodiments, display 121 is a single display. In such embodiments, for each of the user's eyes, corresponding images are simultaneously displayed in a first and a second area of ​​a single display. Optionally, the corresponding images include representations of the same virtual objects and / or the same physical objects viewed from different viewpoints, thereby creating a parallax effect that provides the user with the illusion that objects have depth on a single display.

[0218] In some embodiments, system 100c includes one or more touch-sensitive surfaces 115 for receiving user input, such as tap and swipe input. In some embodiments, display 121 and touch-sensitive surfaces 115 form a touch-sensitive display.

[0219] In some embodiments, sensor 156 includes sensors for detecting various conditions. In some embodiments, sensor 156 includes orientation sensors (e.g., orientation sensor 111) for detecting the orientation and / or movement of the platform. For example, system 100c uses orientation sensors to track changes in the position and / or orientation (sometimes collectively referred to as positioning) of system 100c, such as changes relative to physical objects in the physical environment. In some embodiments, sensor 156 includes one or more gyroscopes, one or more inertial measurement units, and / or one or more accelerometers. In some embodiments, sensor 156 includes a Global Positioning Sensor (GPS) for detecting the GPS position of the platform. In some embodiments, sensor 156 includes a radar system, a LiDAR system, a sonar system, an image sensor (e.g., image sensor 109, a visible light image sensor, and / or an infrared sensor), a depth sensor, a rangefinder, and / or a motion detector. In some embodiments, sensor 156 includes sensors located within an internal portion of system 100c and / or sensors located externally to system 100c. In some embodiments, system 100c uses sensors 156 (e.g., internal sensors) to detect the presence and / or state (e.g., location and / or orientation) of passengers within the interior of system 100c. In some embodiments, system 100c uses sensors 156 (e.g., external sensors) to detect the presence and / or state of objects outside system 100c. In some embodiments, system 100c uses sensors 156 to receive user input, such as gestures and / or other air gestures. In some embodiments, system 100c uses sensors 156 to detect the location and / or orientation of system 100c in the physical environment. In some embodiments, system 100c uses sensors 156 to navigate system 100c along a planned route, around obstacles, and / or to a destination location. In some embodiments, sensors 156 include one or more sensors for identifying and / or authenticating users of system 100c, such as fingerprint sensors and / or facial recognition sensors.

[0220] In some embodiments, the image sensor includes one or more visible light image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide-semiconductor (CMOS) sensors, capable of acquiring images of a physical object. In some embodiments, the image sensor includes one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light. For example, an active IR sensor may include an IR emitter, such as an IR point emitter, for emitting infrared light. In some embodiments, the image sensor includes one or more cameras configured to capture movement of a physical object. In some embodiments, the image sensor includes one or more depth sensors configured to detect the distance between the physical object and system 100c. In some embodiments, system 100c uses a combination of CCD sensors, cameras, and depth sensors to detect the physical environment surrounding system 100c. In some embodiments, the image sensor includes a first image sensor and a second image sensor different from the first image sensor. In some embodiments, system 100c uses the image sensor to receive user input, such as gestures and / or other air gestures. In some embodiments, system 100c uses the image sensor to detect the position and orientation of system 100c in the physical environment.

[0221] In some implementations, system 100c uses an orientation sensor to detect the orientation and / or movement of system 100c. For example, system 100c may use an orientation sensor to track changes in the position and / or orientation of system 100c, such as changes relative to physical objects in the physical environment. In some implementations, the orientation sensor includes one or more gyroscopes, one or more inertial measurement units, and / or one or more accelerometers.

[0222] In some embodiments, system 100c uses a microphone to detect sound from one or more users and / or the physical environment of those users. In some embodiments, the microphone includes a microphone array (comprising multiple microphones) that optionally operates in series, such as to identify ambient noise or locate sound sources in a space (e.g., inside and / or outside system 100c).

[0223] In some embodiments, input device 158 includes one or more mechanical and / or electrical devices for detecting input, such as buttons, sliders, knobs, switches, remote controls, joysticks, touch-sensitive surfaces, keypads, microphones, and / or cameras. In some embodiments, input device 158 includes one or more input devices internal to system 100c. In some embodiments, input device 158 includes one or more input devices external to system 100c (e.g., touch-sensitive surfaces and / or keypads).

[0224] In some embodiments, output device 160 includes one or more devices such as a display, monitor, projector, speaker, lamp, and / or haptic output device. In some embodiments, output device 160 includes one or more external output devices such as an external display screen, external lamp, and / or external speaker. In some embodiments, output device 160 includes one or more internal output devices such as an internal display screen, internal lamp, and / or internal speaker.

[0225] In some embodiments, environmental controls 162 include mechanical and / or electrical systems for monitoring and / or controlling the condition of the interior portion (e.g., cabin) of system 100c. In some embodiments, environmental controls 162 include fans, heaters, air conditioners, and / or thermostats for controlling temperature and / or airflow within the interior portion of system 100c.

[0226] In some embodiments, the mobility component includes mechanical and / or electrical components that enable and / or assist the platform in moving. In some embodiments, the mobility system 164 includes a power system, drivetrain, motor (e.g., electric motor), engine, power source (e.g., battery), transmission, suspension system, speed control system, and / or steering system. In some embodiments, one or more elements of the mobility component are configured for autonomous or manual control (e.g., via system 100c and / or input device 158).

[0227] In some implementations, system 100c performs monetary transactions with or without another computer system. For example, system 100c, or another computer system associated with and / or communicating with system 100c (e.g., via a user account described below), is associated with a user's payment account (such as a credit card account or checking account). To complete the transaction, system 100c may send a key to the entity purchasing goods and / or services from it, enabling the entity to charge the payment account for the transaction. As another example, system 100c stores encrypted payment account information and sends this information to the entity purchasing goods and / or services from it to complete the transaction.

[0228] System 100c may optionally engage in other transactions with other systems, computers, and / or devices. For example, system 100c may engage in transactions to unlock another system, computer, and / or device and / or be unlocked by another system, computer, and / or device. Unlocking transactions may optionally include using, for example, RF circuit 105c to send and / or receive one or more secure cryptographic keys.

[0229] In some implementations, system 100c is capable of communicating with other computer systems and / or electronic devices. For example, system 100c may use RF circuitry 105c to access a network connection that enables the transmission of data between systems for communication purposes. Example communication sessions include telephone calls, emails, SMS messages, and / or video conferencing communication sessions.

[0230] In some embodiments, a video conferencing communication session includes sending and / or receiving video and / or audio data between systems participating in the video conferencing communication session (including system 100c). In some embodiments, system 100c uses sensor 156 to capture video and / or audio content for transmission to other systems in the video conferencing communication session using RF circuitry 105c. In some embodiments, system 100c uses RF circuitry 105c to receive video and / or audio from other systems in the video conferencing communication session and uses output components 160 (such as display 121 and / or speakers) to present the video and / or audio. In some embodiments, the transmission of audio and / or video between systems is near real-time, such as presenting it to other systems with a delay of less than 0.1 seconds, 0.5 seconds, 1 second, or 3 seconds from the time the corresponding portion of the audio and / or video was captured.

[0231] In some embodiments, system 100c uses output component 160 to generate tactile (e.g., haptic) output. In some embodiments, output component 160 generates tactile output by displacing a movable mass block relative to a neutral position. In some embodiments, the tactile output is inherently periodic, optionally including the frequency and / or amplitude of two-dimensional or three-dimensional movement. In some embodiments, system 100c generates a variety of different tactile outputs varying in the frequency, amplitude, and / or duration / number of cycles of the included movement. In some embodiments, the tactile output pattern includes a start buffer and / or an end buffer during which the movable mass block gradually accelerates and / or decelerates at the beginning and / or end of the tactile output, respectively.

[0232] In some implementations, the haptic output has a corresponding characteristic frequency that affects the "pitch" of the haptic sensation perceived by the user. For example, a higher frequency corresponds to faster movement of the movable mass, while a lower frequency corresponds to slower movement of the movable mass. In some implementations, the haptic output has a corresponding characteristic amplitude that affects the "intensity" of the haptic sensation perceived by the user. For example, a higher amplitude corresponds to movement of the movable mass over a greater distance, while a lower amplitude corresponds to movement of the movable mass over a smaller distance. In some implementations, the "pitch" and / or "intensity" of the haptic output varies over time.

[0233] In some embodiments, the tactile output is distinct from the movement of system 100c. For example, system 100c may include a tactile output device that moves a movable mass block to generate the tactile output, and may include other moving parts that control the movement of system 100c, such as motors, wheels, axles, control arms, and / or brakes. Although the movement and / or cessation of movement of system 100c may produce vibrations and / or other physical sensations in some cases, these vibrations and / or other physical sensations are distinct from the tactile output. In some embodiments, system 100c may generate the tactile output independently of the movement of system 100c. For example, system 100c may generate the tactile output without accelerating, decelerating, and / or moving system 100c to a new position.

[0234] In some implementations, system 100c detects gesture input made by a user. In some implementations, gesture input includes touch gestures and / or air gestures, as described herein. In some implementations, the touch-sensitive surface 115 identifies touch gestures based on contact patterns (e.g., varying intensities, timings, and / or movements of an object touching or nearly touching the touch-sensitive surface 115). Thus, the touch-sensitive surface 115 detects gestures by detecting the corresponding contact patterns. For example, detecting a finger press event and then detecting a finger lift-off (e.g., lift-off) event at the same location as the finger press event (e.g., at the location of a user interface element), which could correspond to detecting a tap gesture on a user interface element. As another example, detecting a finger press event, then detecting a movement of the contact, and subsequently detecting a finger lift-off (e.g., lift-off) event, which could correspond to detecting a swipe gesture. Additional and / or alternative touch gestures are possible.

[0235] In some embodiments, an air gesture is a gesture performed by a user without touching the input component 158. In some embodiments, an air gesture is based on the detected movement of a portion of the user (e.g., hand, fingers, and / or body) in the air. In some embodiments, an air gesture includes the movement of that portion of the user relative to a reference object. Example reference objects include the distance of the user's hand relative to a physical object (such as the ground), the angle of the user's arm relative to a physical object, and / or the movement of a first portion of the user (e.g., hand or fingers) relative to a second portion of the user (e.g., shoulder, another hand, or another finger). In some embodiments, detecting an air gesture includes detecting the absolute movement of that portion of the user, such as a tapping gesture including the hand moving in a predetermined pose at a predetermined amount and / or speed, or a shaking gesture including a portion of the user at a predetermined speed or amount of rotation.

[0236] In some implementations, detecting one or more inputs includes detecting the user's speech. In some implementations, system 100c uses one or more microphones of input component 158 ​​to detect when the user utters one or more words. In some implementations, system 100c parses information and / or transmits information to one or more other systems to determine the content of the user's speech, including identifying words and / or acquiring semantic understanding of the words. For example, system processor 103 may be configured to perform natural language processing to detect one or more words and / or determine the possible meaning of one or more words in a sequence spoken by the user. Additionally or alternatively, in some implementations, system 100c determines the meaning of one or more words in a sequence spoken based on the context determined by system 100c.

[0237] In some embodiments, system 100c outputs spatial audio via output component 160. In some embodiments, spatial audio is output at a specific location. For example, system 100c may play a notification ringtone having one or more characteristics that cause the notification ringtone to be generated as if it were emitted from a first location relative to the user's current viewpoint (e.g., "spatialization" includes modifying the audio in amplitude, filtering, and / or delaying it to provide the user with perceived spatial quality).

[0238] In some embodiments, system 100c presents visual and / or audio feedback indicating the user's current position relative to another user's current viewpoint, thereby notifying the other user of the user's updated position. In some embodiments, playing audio corresponding to the user includes altering one or more characteristics of audio acquired from another computer system to simulate the effect of placing an audio source generating the audio playback within the user's position (such as a position the user moves to, appears in, and / or is assigned to within a 3D environment). In some embodiments, the relative magnitude of the audio at one or more frequencies and / or frequency groups is altered, one or more filters are applied to the audio (e.g., directional audio filters), and / or the magnitude of the audio provided via one or more channels is altered (e.g., increased or decreased) to produce the perceived effect of a physical audio source. In some embodiments, the simulated position of the audio source relative to the floor of the 3D environment is matched to the height of the participant's head, or one or more predetermined heights relative to the floor of the 3D environment, when the user's position is determined to correspond to a second position different from a first position and meets one or more first criteria, system 100c presents feedback including generating audio that appears to originate from the second position.

[0239] In some embodiments, system 100c communicates with one or more accessory devices. In some embodiments, one or more accessory devices are integrated with system 100c. In some embodiments, one or more accessory devices are external to system 100c. In some embodiments, system 100c uses RF circuitry 105c and / or a wired connection to communicate with the accessory devices. In some embodiments, system 100c controls the operation of accessory devices (such as doors, windows, locks, speakers, lights, and / or cameras). For example, system 100c can control the operation of an electric door of system 100c. As another example, system 100c can control the operation of an electric window included in system 100c. In some embodiments, accessory devices used as input devices (such as remote controls and / or other computer systems (e.g., smartphones, media players, tablets, computers, and / or wearable devices)) control the operation of system 100c. For example, a wearable device (e.g., a smartwatch) is used as a key to initiate the operation of the actuation system of system 100c. In some implementations, system 100c acts as an input device to control the operation of another system, device, and / or computer, such as platform 100c serving as a key to initiate the operation of an actuation system associated with another system, device, and / or computer.

[0240] In some implementations, the digital assistant uses system 100c to assist the user in performing various functions. For example, the digital assistant may provide weather updates, set alarms, and perform searches locally and / or using a network connection (e.g., the Internet) via a natural language interface. In some implementations, the digital assistant accepts requests that are at least in part in the form of natural language commands, narrations, requests, statements, and / or inquiries. In some implementations, the user uses the digital assistant to request informational answers and / or the execution of tasks. For example, in response to receiving the question “What is the current temperature?”, the digital assistant answers “30 degrees.” As another example, in response to receiving a request to perform a task, such as “Please invite my family to dinner tomorrow,” the digital assistant may confirm the request by playing spoken words (such as “Okay, right away”) and then, on behalf of the user, send the requested calendar invitations to each family member listed in the user’s contact list. In some implementations, the digital assistant engages in a continuous dialogue with the user during the execution of a user-requested task, which involves multiple exchanges of information over a period of time. Other ways of interacting with the digital assistant may involve requesting the execution of tasks and / or requesting information. For example, digital assistance may respond to the user in other forms, such as displayed prompts, text, video, animation, music, etc. In some embodiments, the digital assistant includes a client-side portion executing on system 100c and a server-side portion executing on a server communicating with system 100c. The client-side portion may communicate with the server via a network connection using RF circuitry 105c. For example, the client-side portion may provide client-side functionality, input and / or output processing, and / or communication with the server. In some embodiments, the server-side portion provides server-side functionality for any number of client-side portions across multiple systems.

[0241] In some implementations, system 100c is associated with one or more user accounts. In some implementations, system 100c stores and / or encrypts user data, including files, settings, and / or preferences associated with a specific user account. In some implementations, user accounts are password protected, and system 100c requires user authentication before accessing user data associated with an account. In some implementations, user accounts are associated with other systems, devices, and / or servers. In some implementations, associating a user account with multiple systems enables these systems to access, update, and / or synchronize user data associated with that user account. For example, systems associated with a user account may access purchased media content, contact lists, communication sessions, payment information, stored passwords, and other user data. Therefore, in some implementations, user accounts provide security mechanisms for a customized user experience.

[0242] User interface and related processes Now turn our attention to implementations of user interfaces (“UIs”) and associated processes that can be implemented on computer systems (such as portable multifunction devices or head-mounted devices) having display generation components, one or more input devices, and (optionally) one or more cameras.

[0243] Users interact with electronic devices in a variety of different ways. In some implementations, the electronic device adjusts the light output from one or more light sources within the platform in response to changes in ambient light within the platform. In some implementations, adjusting the light output enables users within the platform to view information and operate the device without requiring subsequent user input to manipulate the one or more light sources, thereby reducing power consumption and extending the battery life of the electronic device and saving power to the one or more light sources through faster and more efficient automatic light adjustment.

[0244] In some implementations, the electronic device displays visual indications of one or more environmental factors in the environment outside the platform. In some implementations, displaying visual indications allows users within the platform to view information about the environment while also providing privacy for users, thereby protecting user privacy and reducing potential confusion for users within the platform who cannot determine their spatial location relative to the environment.

[0245] Figures 6A to 6P Examples of electronic devices that control one or more objects within and / or integrated with the platform in response to changes in the environment outside the platform and / or changes within the platform are illustrated. For example, the electronic devices adjust the light output from one or more light sources within the platform in response to changes in ambient light within the platform. In some embodiments, the electronic devices adjust the light output based on the type or kind of change in ambient light within the platform, as well as other information such as user activity, platform operating mode, and / or location.

[0246] In other example implementations, the electronic device displays or causes the display of visual indications of one or more environmental factors in the environment outside the platform's interior. In some implementations, the one or more environmental factors include pedestrians, drivers and passengers in vehicles, cyclists, aircraft, and / or other environmental factors described below. In some implementations, the visual indications of the one or more environmental factors in the platform's environment include, based on the appearance of the one or more environmental factors, enabling the user to identify objects in the environment while also providing the user with privacy.

[0247] Figure 6AAn example is illustrated of the interior 600b of a platform 600a having display components 602a and 602b. In some embodiments, the platform 600a is a vehicle (e.g., a car, bus, truck, airplane, train, ship, or other vehicle configured to transport users), a house, a shop, or other building. In some embodiments, the display components 602a and 602b are displays integrated with the vehicle or other platform (optionally, one or more of their exterior windows or windshields) for projecting a user interface or making the user interface visible to one or more users (e.g., user 610) within the interior 600b of the platform 600a, as will be referenced. Figures 6I to 6P As described. In some implementations, platform 600a is associated with system 100c (e.g., Figure 1A Operating environment 100, and / or Figure 1B and Figure 1C The system communicates with the HMD 1-100, which is configured to control one or more light sources associated with the platform to output light, as described herein. In some embodiments, system 100c is a mobile device, such as... Figure 6A As shown. In some embodiments, system 100c is a device other than the mobile device described in reference method 700.

[0248] like Figure 6A As shown, the interior 600b of platform 600a includes light sources 608a, 608b, and 606. In some embodiments, and as will be described herein and with reference to methods 700 and 800, system 100c is configured to control one or more light sources to output light. For example, system 100c detects changes in ambient light within the interior of the platform, and in response, system 100c initiates and / or controls the light output of the one or more light sources. The ambient light level is illustrated by an ambient light indicator 612 having a first ambient light quantity value. As described herein and with reference to methods 700 and 800, ambient light optionally refers to light present in the interior space 600b of platform 600a, such as light present in the interior space 600b of platform 600a generated by interior ceiling lights (e.g., light sources 608a and 608b), floor lamps, dashboard / instrument group lights and / or other light present in the interior space 600b of platform 600a generated by one or more light sources discussed herein and with reference to methods 700 and 800 (e.g., light source 606 and / or light entering platform 600a from the outside of platform 600a through display components 602a and / or 602b).

[0249] Figure 6A It also includes light source indicators: 614a corresponds to light source 608a; 614b corresponds to light source 608b; and 616 corresponds to light source 606. Light source indicators 614a, 614b, and 616 include corresponding values ​​of the output light in response to changes in ambient light, as shown in [the diagram / document / reference]. Figure 6BThe figures described herein refer to light quantity values, but such lighting adjustments may optionally be applied to the intensity, luminance, and / or color temperature of the lighting, as described in more detail with reference to methods 700 and / or 800. Figure 6A The user 610 and one or more objects within the platform are further illustrated, such as portions of the right, left and rear walls, the floor and sofa 604 within the platform.

[0250] In some implementations, system 100c outputs light within the platform based on ambient light as described herein. For example, system 100c adjusts the light output from one or more light sources within the platform in response to changes in ambient light within the platform. Figure 6B The ambient light indicator 612, corresponding to the ambient light inside the interior 600b of platform 600a, indicates a second light intensity value, which is less than... Figure 6A The ambient light indicator 612 corresponds to the first light intensity value. In some embodiments, system 100c detects from... Figure 6A The ambient light value shown is... Figure 6B The change in ambient light intensity value is shown (e.g., a decrease in ambient light intensity value). In some embodiments, in response to detecting a change from... Figure 6A The ambient light value shown is... Figure 6B The change in ambient light intensity shown is due to the output of one or more light sources within the platform by system 100c. Figure 6A The light shown Figure 6B The light shown. For example, in Figure 6B In response to changes in ambient light, system 100c adjusts the light output from light sources 608a and 606 in a first manner (e.g., with...). Figure 6A Compared to light indicators 614a and 616, the corresponding light quantity values ​​shown by indicators 614a and 616 are increased respectively.

[0251] exist Figure 6A In the middle, the ambient light indicator 612 corresponding to the ambient light inside the platform 600a and 600b indicates a ratio of Figure 6B The ambient light indicator 612 has a larger light intensity value. In some embodiments, the ambient light intensity value is determined by light sources in the environment outside the interior 600b of the platform 600a (such as...). Figure 6B The moon in 618 or Figure 6F Caused by the sun (628).

[0252] In some implementations, systems 100c operate specific light sources independently of each other. For example, and as... Figure 6BAs shown, system 100c operates light source 608a but not light source 608b (illustrated as indicated by indicators 614a and 614b, respectively) because system 100c determines that user 610 is closer to light source 608a (e.g., within a predetermined distance described in method 700) than to light source 608b. In some embodiments, although light source 606 is farther from user 610 than a predetermined distance, system 100c operates light source 606 (illustrated as indicated by indicator 616) to satisfy or achieve a predetermined target illumination amount as described in method 700. In some embodiments, system 100c operates light source 606 instead of light source 608b because system 100c determines that operating light source 606 requires less energy than operating light source 608b to satisfy the predetermined target illumination amount in space. In some embodiments, system 100c selects to operate the light output from a particular light source based on various factors and / or conditions described with reference to methods 700 and 800.

[0253] In some implementations, system 100c outputs light from one or more light sources within the platform in a manner corresponding to determining that a user within the platform is interacting with an object. As described with reference to methods 700 and 800, the platform may optionally be a space or room within a vehicle, house, shop, building, or other enclosed space. For example, Figure 6C The diagram illustrates users 610 and 620 within the platform when user 620 is interacting with an object (e.g., reading book 622). In some embodiments, in response to determining that user 620 is reading book 622, system 100c outputs light from a light source 608b located above user 620 at a level corresponding to the level of reading. For example, in Figure 6C In this context, the light quantity value of the light source indicator 614b corresponding to the light output from the light source 608b is greater than the light quantity value associated with the light source indicator 614a corresponding to the light output from the light source 608a.

[0254] In some embodiments, system 100c activates one or more light sources and / or initiates light output from the one or more light sources in response to determining that the one or more light sources are within a predetermined distance from the user, as described in method 700. In some embodiments, system 100c controls the light output from the one or more light sources within the platform based on the user's position, as described herein. For example, in Figure 6B In this embodiment, the light source indicator 614b corresponding to the light output from light source 608b is not activated, and is thus exemplified as having a zero value (e.g., light source 608b does not output light). In some embodiments, such as Figure 6CAs shown, system 100c activates light source 608b to output light, wherein light source indicator 614b includes a light quantity value greater than zero. In some embodiments, system 100c detects the presence of user 620 within the platform, such as... Figure 6C As shown. In some implementations, in response to from Figure 6B The user 620 shown as non-existent has been changed to Figure 6C As shown, in the presence of user 620, system 100c activates light source 608b located at a predetermined distance from user 620. For example, Figure 6C The light quantity value included in the light source indicator 614b corresponding to the light output from light source 608b is greater than that of light source indicator 614b. Figure 6B The light quantity value associated with the light source indicator 614b having a zero quantity value. In some embodiments, system 100c activates and / or adjusts the light output from light source 608b regardless of whether user 620 is interacting with an object (e.g., reading book 622). In some embodiments, system 100c activates and / or adjusts the light output from light source 608b in a different manner than when electronic device detects that user 620 is not interacting with an object. For example, in response to system 100c detecting that user 620 is present inside the platform, and based on determining that user 620 is not interacting with an object (e.g., book 622), electronic device activates and / or adjusts the light output from light source 608b in a manner different from that when electronic device detects that user 620 is not interacting with an object. Figure 6C The light source 608b, located at a predetermined distance from the user 620, is activated and / or adjusted in a manner similar to or corresponding to the light source indicator 614a in the system. In another example, in response to system 100c detecting the presence of user 620 inside the platform, and based on determining that user 620 is interacting with an object, the electronic device activates and / or adjusts the light source 608b located at a predetermined distance from the user 620 in a manner different from the manner in which system 100c activates and / or adjusts the light source 608b when it is determined that the user is not interacting with an object (e.g., Figure 6C The light quantity value included in the light source indicator 614b is greater than the light quantity value associated with the light source indicator 614a.

[0255] In some implementations, system 100c controls the light output from one or more light sources within the platform in a manner corresponding to a defined time of day (e.g., early morning, morning, noon, afternoon, evening, or night). For example, in Figure 6D In the middle, the ambient light indicator 612 corresponding to the ambient light inside the platform 600a and 600b indicates a ratio of Figure 6C The ambient light indicator 612 in the system has a larger light intensity value. In some implementations, system 100c detects light from... Figure 6C The ambient light value shown is... Figure 6DThe change in ambient light intensity value is shown (e.g., an increase in ambient light intensity value). In some embodiments, in response to detecting a change from... Figure 6C The ambient light value shown is... Figure 6D The change in ambient light intensity shown is due to the output of one or more light sources within the platform by system 100c. Figure 6C The light shown Figure 6D The light shown. For example, from Figure 6C The ambient light value shown is... Figure 6D The variations in ambient light values ​​shown are optionally caused by environmental changes (such as from night to day), such as the absence of a moon 618 indicating night, which indicates the moon's position. Figure 6C The environment visible via display component 602b, which is located outside the platform's interior.

[0256] In some implementations, system 100c is configured to detect optical output from one or more second electronic devices different from system 100c. For example, in Figure 6E In the image, user 610 is interacting with a second electronic device 624 (e.g., a laptop computer). Electronic device 624 includes a display screen that projects light with a first magnitude, such as... Figure 6E Illustrated by light source indicator 626 in . Figure 6E The ambient light indicator 612 in the middle includes more than Figure 6D The ambient light indicator 612 in the image has a larger light intensity value. In some implementations, Figure 6E The light intensity value of the ambient light indicator 612 is caused by the electronic device 624.

[0257] In some implementations, to minimize or prevent glare on the display screen of electronic device 624, system 100c adjusts the light output from light source 608a, such as... Figure 6E The light intensity value indicated by the light source indicator 614a is now less than, as shown in the image. Figure 6D (In this diagram, user 610 is not using electronic device 624) The light quantity value indicated by light source indicator 614a corresponds to the light source 608a. In some embodiments, system 100c sets the light output from electronic device 624 according to the lighting conditions in the platform. For example, in Figure 6F In the middle, system 100c detected a ratio Figure 6E The increased ambient light value is caused by one or more environmental factors (e.g., the sun 628) from the environment outside the interior 600b of platform 600a. This increased ambient light is due to... Figure 6F The ambient light level of the ambient light indicator 612 is greater than that of the ambient light indicator 612. Figure 6E The ambient light indicator 612 is used as an example to represent the corresponding ambient light intensity value. In some embodiments, it is based on the ambient light intensity value derived from... Figure 6F The system 100c sets the light output from the display of the electronic device 624 to a value indicated by the indicator 626, based on the ambient light caused by direct sunlight from the sun 628. Figure 6F Medium greater than by Figure 6E The value indicated by indicator 626 in the text. Figure 6F The display of the electronic device 624 uses increased brightness to minimize or reduce glare on the screen, which may be caused by direct sunlight from the sun 628.

[0258] In some implementation schemes, according to Figure 6F The ambient light intensity (including direct sunlight from the sun 628) is used by system 100c to operate light sources 608a, 608b, and 606 as indicated by light source indicators 614a, 614b, and 616, respectively. For example, the corresponding light intensity values ​​associated with light source indicators 614a, 614b, and 616 are now... Figure 6F When there is less ambient light in the medium Figure 6E The amount is lower. In some embodiments, to further reduce glare on the display screen of electronic device 624, system 100c operates light source 608a in a different manner than light source 608b, because light source 608a is closer (e.g., within a predetermined distance as described in reference method 700) to user 610 interacting with electronic device 624. This difference is caused by Figure 6F The light quantity value associated with the light source indicator 614a corresponding to light source 608a is illustrated as being less than the corresponding light quantity value associated with the light source indicator 614b corresponding to light source 608b. In some embodiments, and as described in more detail with reference to methods 700 and 800, system 100c controls the light output from one or more light sources (e.g., 608a, 606b, 606, and / or 624) in a manner based on one or more conditions or combinations of conditions.

[0259] In some embodiments, when system 100c determines that the amount of illumination (e.g., direct sunlight caused by sun 628) exceeds a predetermined target amount of illumination and / or falls within the range of illumination values ​​described in methods 700 and / or 800, system 100c applies a certain amount of color and / or shadow (e.g., color / shadow 630a and 630b) to display components 602a and 602b, such as... Figure 6G As shown. Therefore, in some embodiments, display components 602a and 602b serve as electrically adjustable light shields for platform 600a. In some embodiments, system 100c applies adjustable amounts of color and / or shading to reflect, transmit, and / or absorb light from... Figure 6FThe direct sunlight of the sun 628 is projected into the interior 600b of platform 600a, as described in more detail with reference to methods 700 and / or 800.

[0260] Figure 6G As further illustrated, because system 100c applies this amount of color / shade 630a and 630b to display components 602a and 602b respectively, the light quantity value associated with ambient light indicator 612 is now reduced to a certain amount, which is less than... Figure 6F The amounts shown are associated with ambient light indicator 612 before the application of color / shading 630a and 630b. In response to the aforementioned change in ambient light, system 100c adjusts the light output from light sources 608a, 608b, 606, and 624. For example, the corresponding light quantity values ​​of light source indicators 614a, 614b, 616, and 626 are now greater than those associated with ambient light indicator 612. Figure 6F The light intensity values ​​associated with the light source indicators 614a, 614b, 616, and 626 (e.g., before changes in ambient light caused by the color / shading 630a and 630b applied by system 100c to display components 602a and 602b). In some embodiments, and as... Figure 6G As shown, due to one or more factors, including determining that user 610 is interacting with laptop computer 624, the light output from light source 608a is different from the light output from light source 608b (e.g., less light). Therefore, in order to minimize glare on the display of electronic device 624, system 100c optionally adjusts the light output from light source 608a differently than light source 608b. System 100c considers and describes in more detail one or more other factors and / or combinations of such one or more other factors with those optionally described herein, with reference to methods 700 and / or 800.

[0261] In some embodiments, system 100c detects and / or captures information about the environment outside the interior 600b of platform 600a and / or objects in that environment via display components 602a and / or 602b. See reference method 800 and... Figures 6H to 6P As described, system 100c displays visual indications of the one or more objects (e.g., one or more environmental factors described in method 800) via display components 602a and / or 602b. For example, in Figure 6H In the process of operating display components 602a and 602b in a privacy mode as illustrated by indicator 636 with a value of "off", system 100c refrains from adjusting the translucency and / or hue of display components 602a and 602b to make the interior 600b of platform 600a indistinct from the exterior of platform 600a, and vice versa. Therefore, in Figure 6HIn the process, users 610 and 620 can view external objects, such as pedestrians 632a and buildings 634a, via display components 602a and 602b.

[0262] In some implementations, system 100c receives input to display components 602a and / or 602b according to a privacy mode (such as...). Figure 6I The system 100c displays a request for operation of the privacy mode (illustrated by privacy mode indicator 636 with the value "on"). In some embodiments, privacy mode is an operating mode in which an observer outside the platform is not allowed to easily view the internal workings of the platform via display components, and vice versa. In response to receiving a request for operation of display components 602a and / or 602b according to privacy mode, the system 100c displays visual indicator 632b via display components 602a and / or 602b. For example, in Figure 6I In the middle, system 100c displays the information via display unit 602a. Figure 6H Visual indication 632b for pedestrian 632a shown. Figure 6I The system 100c further illustrates the display of information via the display unit 602b. Figure 6H Visual indicator 634b is shown for building 634a. In some embodiments, visual indicator 632b is displayed having a shape similar to its corresponding object, as shown in visual indicator 632b. In some embodiments, visual indicators are displayed with an abstract visual appearance, as shown in visual indicator 634b. The visual appearance of visual indicators is described in more detail with reference to method 800.

[0263] In some implementations, system 100c moves the visual cues 632b to the external object based on detected movement of the external object. For example, from Figures 6I to 6J System 100c detects movement of an external object (e.g., pedestrian 632a) from left to right (e.g., relative to display unit 602a) of platform 600a via one or more sensors as described in method 800. In response to the detected movement of pedestrian 632a from left to right of platform 600a, system 100c displays visual indication 632b via display unit 602a as a movement from left to right of display unit 602a (e.g., the magnitude, velocity, acceleration, and / or direction of movement of pedestrian 632a). In some embodiments, even when system 100c detects no movement of the external object, system 100c displays visual indications 632b and 634b via display unit 602a at corresponding locations corresponding to the external objects (e.g., pedestrian 632a and building 634a).

[0264] In some implementations, system 100c displays visual indications that indicate the location information of platform 600a, such as... Figure 6L As shown. For example, from Figure 6K to Figure 6LDisplay components 602a and 602b are from, according to, Figure 6K The privacy mode instruction with the value "off" indicates that the normal (non-privacy) mode operation exemplified by 636 is changed according to, for example, Figure 6L The privacy mode instruction with the value "On" is exemplified by 636. Figure 6K In the middle, a street-level view 638a of the environment outside the interior 600b of platform 600a is visible via display component 602a. Figure 6K The physical building 634a visible via display component 602b is also illustrated. In response to the display components 602a and 602b changing from operation according to normal mode to operation according to... Figure 6L The privacy mode operation illustrated in the system 100c display indicates... Figure 6K The street-level view 638a and the visual indication 638b. Figure 6L In the middle, visual indicator 638b is similar to a street-level view, but without the same... Figure 6K The level of detail shown is associated with the street-level view 638a presented via display component 602a when operating in normal mode. In some embodiments, system 100c displays visual indications 638c associated with location information used by system 100c to perform functions. For example, in Figure 6L In this system, visual indication 638c includes the name of the street in which platform 600a is currently navigating. Additionally and / or alternatively, system 100c displays visual indication 640 representing a map via display components 602a and / or 602b, which includes visual indication of the route, visual indication of the destination 642, and visual indication of the current location and orientation of platform 644.

[0265] In some implementations, system 100c controls the light output within the platform to enhance the environment outside the platform 600a. For example, in Figure 6N In this context, the environment outside the interior 600b of platform 600a includes a sunset 648 presented via display component 602b. In some embodiments, in response to detecting the sunset 648, system 100c controls the light output in the interior 600b of platform 600a to visually emphasize the sunset 648, such that, to users 610 and 620, the color of the sunset 648 appears to be more emphasized within the platform than if the light output in the interior 600b of platform 600a were not adjusted according to the sunset. For example, in Figure 6N In this context, a color temperature of 650 has color values ​​similar to or corresponding to those of sunset 648. Figure 6NIn this embodiment, system 100c displays user interface elements 654 via a display 652 of a second electronic device. These user interface elements include controls 656a and 656b, which are interactive to allow system 100c to adjust the color temperature of the interior 600b of platform 600a. In some embodiments, system 100c displays contextual information, such as a representation of a horizon line 646, via display components 602a and / or 602b to indicate the location and / or position of a visual indicator (e.g., sunset 648) relative to the representation of the horizon line 646. In some embodiments, system 100c controls the light output within the interior 600b of platform 600a to visually reduce one or more environmental factors, such as smoke and / or wildfire smoke. For example, system 100c applies color correction to the light output within the interior of platform 600a in response to the color of smoke and / or wildfire smoke, such that the color of smoke and / or wildfire smoke is minimized to a lower degree compared to the case where the light output within the interior 600b of platform 600a is not controlled.

[0266] In some implementations, system 100c displays visual indications of one or more environmental factors, including, for example... Figure 6O The simulation of the moon 662 is shown. For example, when display components 602a and 602b are operating in a privacy mode as indicated by a privacy mode indicator 636 with a value of "on," preventing users 610 and 620 located within platform 600a from easily viewing the environment outside the platform via display components 602a and 602b, system 100c detects the physical moon and displays the simulation of the moon 662 as an overlay on the rendering of the physical environment 660b outside the interior 600b of platform 600a. For example, the simulation of the moon 662 may optionally be overlaid on the rendering of the sky of the physical environment. In some embodiments, system 100c displays the simulation of the moon 662, including one or more visual characteristics corresponding to one or more visual characteristics of the simulated moon, such as the simulated moon's position, phase, color, hue, brightness, luminance, and / or texture, as described in more detail with reference to method 800.

[0267] In some implementations, when a physical object in the environment is partially or completely invisible, system 100c displays a representation of the physical object in a display component corresponding to the corresponding location of the physical object in the environment. For example, in Figure 6P In this configuration, display components 602a and 602b operate in normal mode, as indicated by privacy indicator 636 with the value "off," and the physical environments 664a and 664b are visible to users 610 and 620. Figure 6PIn the configuration, physical environments 664a and 664b include fog, which makes physical objects in the environment, such as clouds and buildings, partially or completely invisible. In response to system 100c determining that physical objects included in the environment are not visible to the interior of the platform, system 100c displays representations of the physical objects (e.g., representations 668a, 666a, 668b, and 666b) via display components 602a and 602b. Representations 668a, 666a, 668b, and 666b correspond to clouds and buildings in the physical environment that would otherwise be visible in the absence of foggy weather environments 664a and 664b.

[0268] Figure 7 This is a flowchart illustrating a method in which an electronic device controls the light output from one or more light sources within the platform in response to changes in ambient light within the platform. Method 700 is optionally performed at a first device and / or electronic device (such as system 100c described above with reference to FIG. 1). Some operations in method 700 are optionally combined, and / or the order of some operations is optionally changed.

[0269] In some implementations, method 700 is used in conjunction with platforms (such as...) Figure 6A The platform 600a) communicates with an electronic device (e.g., 100c). In some embodiments, the electronic device is a mobile device (e.g., a tablet, smartphone, and / or media player), a computer (e.g., a desktop computer and / or laptop computer), an in-vehicle computer (e.g., an in-vehicle computer, a vehicle information and entertainment system, or an infotainment system), and / or a wearable device (e.g., a watch and / or a head-mounted device). In some embodiments, the platform is a vehicle (e.g., a car, bus, truck, airplane, train, ship, or other vehicle configured to transport users), a house, a shop, or other building. In some embodiments, the electronic device communicates with the platform using wired or wireless communication. In some embodiments, the electronic device is located within the platform. In some embodiments, and as will be described herein, the electronic device is configured to control one or more light sources to output light. In some embodiments, and as will be described herein with respect to method 700, the electronic device detects changes in ambient light within the interior of the platform (e.g., a confined space, such as the interior of a vehicle or a room in a house), and in response, the electronic device is configured to initiate or control light output via the one or more light sources. In some embodiments, method 700 is performed at or by a vehicle (e.g., at the lighting system, multi-display system, and / or infotainment system of a car having or communicating with one or more display components and / or input devices).

[0270] In some implementations, when light is output within the platform using one or more light sources (e.g., interior ceiling lights, floor lamps, dashboard / instrument group lights, and / or other light present within the platform and generated by one or more light sources discussed herein), system 100c detects (702a) changes in ambient light within the platform, such as Figure 6A Ambient light indicator 612 to Figure 6B The ambient light indicator 612 indicates a change. For example, ambient light optionally refers to light present within the interior space of the platform, such as light provided by interior ceiling lights, floor lamps, dashboard / instrument cluster lights, and / or other light present within the platform generated by one or more light sources discussed herein. In some embodiments, the electronic device uses one or more light sensors to measure the intensity and / or luminance of ambient light. For example, the electronic device uses multiple light sensors (e.g., located at one or more locations within the platform) to determine changes in ambient light at different locations within the platform and / or to determine the average ambient light within the platform. In some embodiments, the electronic device detects changes in ambient light within the platform when no light is output within the platform.

[0271] In some implementations, in response to detecting a change in ambient light inside the platform (702b), and based on determining that the change in ambient light inside the platform is caused by a light source in the environment outside the platform (such as...) Figure 6B The first change in the moon (618) causes the electronic equipment to adjust the light output inside the platform (702c) in a first manner, such as Figure 6BThe light source indicators 614a and 616 are shown in the diagram. In some embodiments, the one or more light sources outputting light include lamps, light-emitting diodes, display components, lasers, light source arrays, individual light sources, backlight units of display components, light sources emitting beams, and / or other electrically controlled light sources. In some embodiments, the platform includes an ambient light sensor or other light-based sensors configured to measure the amount of ambient light within the platform. In some embodiments, the electronic device includes an ambient light sensor. In some embodiments, the electronic device determines, as described herein, a process for initiating a change in the light output within the platform based on the measured amount of ambient light (e.g., measured in lux and / or lumens), the measured ambient light intensity (e.g., measured in candela (cd), the measured luminance (e.g., measured in lux (lx)), and / or the measured color temperature (e.g., measured in Kelvin). For example, the electronic device optionally detects a change in the measured amount of ambient light within the platform from a first ambient light amount to a second ambient light amount, wherein the first ambient light amount is greater than the second ambient light amount. In some embodiments, the first ambient light amount is less than the second ambient light amount. In some embodiments, changes in ambient light within the platform indicate that the ambient lighting is too bright or too dim (e.g., the measured ambient light level within the platform is above or below a lighting threshold, such as 20 lux, 40 lux, 60 lux, 80 lux, 100 lux, 150 lux, 200 lux, 250 lux, 300 lux, 350 lux, 400 lux, 600 lux, 800 lux, or 1000 lux), making it difficult for a user to read and / or write comfortably. In some embodiments, the first change in light source in the environment outside the platform originates from external entities located outside the platform, such as the sun, moon, streetlights, reflective parts of vehicles or buildings, floodlights, and / or other external entities that project light into the platform's interior. In some embodiments, adjusting the light output within the platform in a first manner includes modifying the illumination level from a first (initial) illumination level to a second illumination level greater than the first illumination level. In some embodiments, the second illumination level is less than the first illumination level. In some embodiments, adjusting the light output within the platform in a first manner includes controlling the platform's sunshade, the platform's electrically controlled window tint, or other electrically adjustable components of the platform, as described in more detail below. It should be understood that although the embodiments described herein involve modifying the amount of illumination, such illumination adjustments may optionally apply to the intensity, luminance, and / or color temperature of the illumination, as described in more detail below. In some embodiments, as described herein and below, adjusting the light output within the platform in a first manner is based on the type or kind of variation in ambient light within the platform's interior, as well as other information (e.g., user activity, platform operating mode, and / or location).

[0272] In some implementations, in response to detecting a change in ambient light within the platform, and based on determining that the change in ambient light within the platform is caused by a light source within the platform (such as from...) Figure 6E The second change in the light source of the laptop computer 624 causes the electronic devices to adjust the light output inside the platform (702d) in a second manner different from the first manner, such as... Figure 6E The light source indicator 614a is shown in the document. In some embodiments, the second variation of the light source within the platform originates from one or more light sources described herein. In some embodiments, the light source within the platform includes artificial light. For example, artificial light (e.g., floor lamps, ceiling lamps, wall lamps, reading lamps, door lamps, and / or dashboard / instrument group lamps) may be optionally provided on the inner surface of the platform. In another example, the artificial light may optionally be provided by devices (such as computer monitors, mobile displays, wearable displays, television displays, and / or displays of devices located within the platform that are different from electronic devices). In some embodiments, the electronic devices adjust the light output within the platform in a second manner greater than the first manner (e.g., increasing the amount of illumination). In some embodiments, the second manner is less than the first manner (e.g., decreasing the amount of illumination). In some embodiments, adjusting the light output within the platform in a second manner includes maintaining the light output from the first light source within the platform and adjusting the light output from a second light source within the platform that is different from the first light source within the platform. In some embodiments, adjusting the light output within the platform in a second manner includes adjusting the light source within the platform. In some embodiments, adjusting the light output within the platform in a first or second manner includes controlling the corresponding amount of illumination from a light source at a predetermined distance (e.g., 5 cm, 10 cm, 12 cm, 14 cm, 16 cm, 18 cm, 20 cm, 40 cm, 60 cm, 80 cm, 100 cm, 120 cm, 140 cm, 160 cm, 180 cm, 200 cm, or 250 cm) from electronic devices or a user (optionally equipped with electronic devices), as will be described in more detail below. It should be understood that although the embodiments described herein involve controlling the amount of illumination, such illumination adjustments may optionally apply to the intensity, luminance, and / or color temperature of the illumination, as will be described in more detail below.

[0273] In some implementations, as described herein and below, the light output within the platform is adjusted in a second manner based on the type or kind of change in ambient light within the platform, as well as other information (e.g., user activity, platform operating mode, and / or location). Automatic adjustment of the light output from one or more light sources within the platform in response to changes in ambient light within the platform enables the user to view information and operate the device in environments with varying ambient lighting conditions. This reduces the need for subsequent user input to manipulate the one or more light sources, thereby reducing power consumption and extending the battery life of electronic devices and saving power from the one or more light sources through faster and more efficient automatic light adjustment.

[0274] In some implementations, in response to detecting a change in ambient light within the platform (e.g., a change similar to or corresponding to the change in ambient light within the platform as described in reference method 700 above), and based on determining that a user within the platform is interacting with an object (such as...) Figure 6C The book 622) interacts with electronic devices in a third way, adjusting the light output within the platform, such as Figure 6C The light source indicator 614b is shown in the diagram. In some embodiments, a third method of adjusting the light output is based on user interaction with objects on the platform. For example, the electronic device detects that the user is reading a book, working on a laptop computer, participating in a video call, writing in a diary, searching for items on the furniture or floor of the platform, or another similar user activity. In some embodiments, the electronic device receives data (e.g., image data) indicating that the user is interacting with an object via one or more camera sensors of the platform and / or the electronic device. For example, the user's head pose is within a predetermined distance (e.g., 50 cm, 60 cm, 70 cm, 80 cm, 90 cm, 100 cm, or 200 cm) of the object. In another example, the object's pose is within a predetermined distance of the user's head.

[0275] In some embodiments, the electronic device receives data (e.g., image data of the user's eyes) indicating that the user is interacting with an object via a gaze tracking device in communication with the electronic device. In some embodiments, before adjusting the light output within the platform in a third manner, the electronic device detects that the user's gaze is directed at the object for a period of time greater than a threshold time interval (e.g., 0.1 seconds, 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds, 7 seconds, 10 seconds, 20 seconds, 30 seconds, or 60 seconds). In some embodiments, the object is the electronic device or a second electronic device different from the electronic device. In some embodiments, the second electronic device has one or more characteristics of the electronic device described above with reference to method 700. In some embodiments, the object is a non-electronic object, such as a book, magazine, notebook, document, diary, and / or other non-electronic object read by the user. In some embodiments, adjusting the light output within the platform in a third manner is the same as in the first manner. In some embodiments, adjusting the light output within the platform in a third manner is different from the first manner. For example, when an electronic device determines that a user is interacting with an object, the electronic device adjusts the light output within the platform in a third manner (e.g., increasing the amount, intensity, brightness, and / or color temperature of the illumination) different from the first manner (e.g., different from the second and / or other manners described above and / or below). For example, when the electronic device determines that a user is reading a book, the third manner in which the electronic device outputs light includes a color temperature in the range of 3000K (Kelvin) to 6500K. In some embodiments, when the electronic device determines that a user is interacting with an object, it decreases the amount, intensity, brightness, and / or color temperature of the illumination. For example, when the electronic device determines that a user is listening to music, the third manner in which the electronic device outputs light includes a color temperature in the range of 2000K to 3000K to promote a quiet environment for the user. In some embodiments, adjusting the light output within the platform in a third manner has one or more characteristics of adjusting the light output within the platform in a first manner (e.g., in the second and / or other manners).

[0276] In some embodiments, the electronic device adjusts the light output within the platform in a third manner without altering the first, second, and / or other methods, such that the third manner in which the light is output within the platform differs from the first, second, and / or other methods. For example, the amount, magnitude, and / or degree of illumination is changed in a third manner without optionally changing the amount, magnitude, and / or degree of illumination in the first, second, and / or other methods. In some embodiments, the electronic device adjusts the light output within the platform in a third manner while altering the first, second, and / or other methods, such that the third manner in which the light is output within the platform is the same as the first, second, and / or other methods, or is an accumulation of the light output adjusted within the platform in the first, second, and / or other methods. In some embodiments, the electronic device does not adjust the light output within the platform in a third manner based on the determination that the user within the platform is not interacting with an object. For example, before detecting a change in ambient light within the platform, the electronic device outputs light within the platform in the first, second, and / or other methods (e.g., the same as before). In some embodiments, the electronic device adjusts the light output without receiving user input corresponding to a request to adjust the light output. In some embodiments, and as described below, the electronic device adjusts the light output in response to receiving user input corresponding to a request to adjust the light output. In some embodiments, the electronic device determines that the user is interacting with a second object different from the first object. In response to determining that the user is interacting with the second object, the electronic device optionally adjusts the light output within the platform in a fourth manner different from a third-party approach. For example, the second object may optionally be a tablet computer or an e-book reader, and to prevent or reduce glare from the tablet computer's display screen (or other light-projecting element), the fourth manner in which the electronic device outputs light includes a brightness in the range of 50 to 200 lux. In some embodiments, the electronic device adjusts the light generated by the display screen of a table, as described below. The electronic device optionally adjusts the light output from one or more light sources within the platform using a fourth manner (e.g., and / or a third and / or other manner) depending on the object with which the user is interacting, as described below. Based on the determination that a user is interacting with an object within the platform, the light output from one or more light sources within the platform is automatically adjusted, enabling the user to view and manipulate objects in environments with varying lighting conditions. This reduces the need for subsequent user input to manipulate the one or more light sources, which reduces power consumption and extends the battery life of electronic devices by automatically adjusting the light more quickly and efficiently, and saves power from the one or more light sources.

[0277] In some implementations, in response to detecting a change in ambient light within the platform (e.g., a change similar to or corresponding to the change in ambient light within the platform as described in reference method 700 above), and based on determining the current time standard corresponding to a light source in the environment outside the platform (such as... Figure 6M The expected changes in Sunset 648 involve electronic devices adjusting the light output within the platform in a third way, different from the second method, such as... Figure 6M The color temperature indicator 650 is shown in the diagram. In some embodiments, the third mode of adjusting the light output is based on a current time standard corresponding to an expected change in light sources in the external environment. For example, sunset, sunrise, dawn, dusk, twilight, or night. In some embodiments, the electronic device adjusts the light output within the platform to enhance the environment outside the platform (e.g., providing time-dependent lighting adjustments that visually emphasize changes in light sources in the environment outside the platform). For example, a sunset or sunrise is enhanced by adjusting the saturation, contrast, and / or brightness of the light output within the platform, making the sunset or sunrise appear as if it were inside the platform (e.g., outputting light in a third mode to match the sunset or sunrise colors from the environment outside the platform). In another example, when the electronic device determines that the current time standard corresponds to a daytime period shortly after sunrise and before sunset (during which the sun is low on the horizon, producing a warm orange glow (e.g., the "golden hour")), the third mode of the light output by the electronic device includes a color temperature in the range of 1000K to 4000K to match the external environment by providing a warm orange glow within the platform. In another example, based on a second anticipated change in the light source in the environment outside the platform, which determines that the current time standard corresponds to a different expected change in the light source, the electronic device adjusts the light output inside the platform in a fourth manner, different from a third-party approach. For example, when the electronic device determines that the current time standard corresponds to the daytime period shortly before sunrise and after sunset (during which the sun is below the horizon, producing a cool, deep blue glow (e.g., the "blue hour")), the fourth manner in which the electronic device outputs light includes a color temperature in the range of 8000K to 15000K to provide a cool, deep blue glow inside the platform that matches the external environment.

[0278] In some embodiments, adjusting the light output within the platform in a third or fourth manner has one or more characteristics of adjusting the light output within the platform in the first, second, and / or other manner as described above and / or below. In some embodiments, adjusting the light output within the platform includes the intensity, brightness, and / or color temperature of the illumination. In some embodiments, adjusting the light output within the platform includes activating or deactivating a specific light source. In some embodiments, adjusting the light output within the platform includes changing the direction of light emitted from the light source (e.g., pointing towards or away from the user). In some embodiments, adjusting the light output within the platform in a third, fourth, and / or other manner includes one or more of these illumination adjustments or other illumination adjustments described above and / or below. For example, a third and / or other manner in which an electronic device adjusts the light output within its platform optionally includes adjusting one or more first color values ​​(such as yellow, orange, and red values) corresponding to a light source in the environment outside the platform (e.g., increasing the intensity of the one or more first color values) without changing one or more second color values ​​that do not correspond to a light source in the environment outside the platform. In some implementations, adjusting the one or more first color values ​​does not include, for example, activating or deactivating a specific light source, changing the direction of light emitted from the light source, and / or changing one or more other lighting characteristics (such as brightness). In some implementations, the electronic device does not adjust the light output inside the platform in a third and / or other manner based on the determination that the current time standard does not correspond to an expected change in light sources in the environment outside the platform's interior. For example, before detecting a change in ambient light inside the platform, the electronic device outputs light inside the platform in a first, second, and / or other manner (e.g., the same as before). Automatically adjusting the light output from one or more light sources inside the platform based on the determination that the current time standard corresponds to an expected change in light sources in the environment outside the platform's interior allows the user to view the external environment inside the platform, thereby reducing potential confusion for the user who cannot determine the time of day.

[0279] In some implementations, in response to detecting a change in ambient light within the platform (e.g., a change similar to or corresponding to the change in ambient light within the platform as described in reference method 700 above), and based on determining that the platform is operating in a first mode (e.g., a normal (non-privacy) mode, in which an observer outside the platform is allowed to easily view the interior of the platform via a display component, and vice versa, as described with reference to step 800), the electronic device adjusts the light output within the platform in a third manner, such as in... Figure 6MThe privacy mode indicator 636 indicates the color temperature indicator 650 during normal mode operation. In some embodiments, the appropriate manner of adjusting the light output is based on the platform's operating mode. For example, and as will be described herein, the electronic device adjusts the light output to maintain the visibility of one or more environmental factors of the external environment to the interior of the platform. For example, when the electronic device determines that the platform is operating in a non-privacy mode, the third manner of the light output by the electronic device does not include any distorted color and / or light pattern to limit the visibility of the one or more environmental factors of the external environment to the interior of the platform and / or prevent an external viewer from easily viewing the interior of the platform, and vice versa. In some embodiments, adjusting the light output inside the platform in a third manner has one or more characteristics of adjusting the light output inside the platform in a third manner and / or other manner as described above and / or below. In some embodiments, adjusting the light output inside the platform in a third manner and / or other manner is based on determining whether changes in ambient light inside the platform satisfy some or all of the lighting adjustment conditions described herein. For example, the electronic device optionally adjusts the light output inside the platform in a third and / or other manner based on determining that the platform is operating in a first mode and based on determining the expected change in light sources in the environment outside the platform corresponding to the current time standard. In another example, the electronic device optionally adjusts the light output inside the platform in a third and / or other manner based on determining that the platform is operating in a first mode and based on determining that a user inside the platform is interacting with an object. In some embodiments, the lighting adjustment conditions are associated with importance values ​​or rankings. For example, adjusting the light output inside the platform in a third and / or other manner based on determining that a user inside the platform is interacting with an object is optionally preferred over adjusting the light output inside the platform in a third and / or other manner based on determining that the platform is operating in a first mode.

[0280] In some implementations, adjusting the light output in a third and / or other manner based on determining that the platform is operating in a first mode optionally takes precedence over adjusting the light output in a third and / or other manner based on determining that a user within the platform is interacting with an object. Other prioritization schemes are envisioned for other lighting adjustment conditions described above and / or below. In some implementations, the importance value or ranking is determined automatically by the electronic device without user input. In some implementations, the importance value is set by the user of the electronic device and / or by a user within the platform. In some implementations, the lighting adjustment conditions are associated with a predetermined target amount of illumination within the platform. For example, when the electronic device determines that adjusting the light output within the platform in a third and / or other manner results in meeting or reaching a predetermined target amount of illumination, the electronic device does not adjust the light output within the platform in a third and / or other manner (e.g., exceeding the predetermined target amount of illumination).

[0281] In some implementations, in response to detecting a change in ambient light within the platform's interior, and based on determining that the platform is operating in a second mode different from the first mode (e.g., privacy mode, in which an observer outside the platform is not permitted to easily view the interior of the platform via the display component, and vice versa, as described with reference to step 800), the electronic device adjusts the light output within the platform's interior in a fourth manner different from a third-party approach (e.g., to maintain privacy), such as in... Figure 6O The privacy mode indicator 636 indicates that the light sources 608a, 608b, 606, and 624 are adjusted during privacy mode operation. In another example, when the electronic device determines that the platform is operating in privacy mode, a fourth manner in which the electronic device outputs light optionally includes distorting the color and / or light pattern to limit the visibility of one or more environmental factors of the external environment to the interior of the platform and / or prevent an external viewer from easily viewing the interior of the platform, or vice versa (e.g., protecting privacy). In some embodiments, the electronic device displays a visual indication of the one or more environmental factors, as described with reference to step 800. In some embodiments, adjusting the light output in the interior of the platform in a fourth manner has one or more characteristics of adjusting the light output in the interior of the platform in a second manner and / or other manner as described above and / or below. In some embodiments, the electronic device controls the platform to operate from a first mode to a second mode, and from a second mode to a first mode (optionally, without receiving user input for changing the operating mode of the platform). The platform is automatically adjusting the light output from one or more light sources within the platform, either in a second or first mode, to allow users to view the external environment within the platform while also protecting user privacy.

[0282] In some implementations, in response to detecting a change in ambient light within the platform (e.g., a change similar to or corresponding to the change in ambient light within the platform as described in reference method 700 above), and based on determining that a user within the platform is within a first threshold distance (e.g., 5 cm, 10 cm, 12 cm, 14 cm, 16 cm, 18 cm, 20 cm, 40 cm, 60 cm, 80 cm, 100 cm, 120 cm, 140 cm, 160 cm, 180 cm, 200 cm, or 250 cm) from the one or more light sources, the electronic device adjusts the light output within the platform in a third manner, such as... Figure 6C The light source 608b is adjusted. In some embodiments, when the electronic device detects that the user has a first position relative to the interior of the platform, the electronic device adjusts the light output in a third or other manner as described above and / or below. In some embodiments, when the electronic device detects that the user has a second position relative to the interior of the platform different from the first position, the electronic device adjusts the light output in a manner different from that associated with the first position. In some embodiments, and as will be described herein, the electronic device considers whether the user's corresponding position relative to the interior of the platform is within a threshold distance from the one or more light sources or at a greater distance than the threshold distance from the one or more light sources.

[0283] In some implementations, in response to detecting a change in ambient light within the platform, and based on determining that a user within the platform is located at a distance greater than a first threshold distance (e.g., as described above), the electronic device adjusts the light output within the platform in a fourth manner, distinct from a third-party approach, such as... Figure 6BThe light source indicator 614b is shown in the diagram. In some embodiments, a third or fourth mode of adjusting the light output is based on the corresponding position of the one or more light sources relative to the user. For example, when the electronic device determines that the user inside the platform is within a first threshold distance from the one or more light sources, the third mode of the electronic device outputting light optionally includes modifying the illumination amount (and / or illumination intensity, luminance, color temperature, and / or other characteristics) of the one or more light sources located within the first threshold distance from the user by a first amount, degree, or level greater than and / or different from the illumination amount, degree, or level associated with the one or more light sources located further from the user than the first threshold distance. In another example, when the electronic device determines that the user inside the platform is located further from the one or more light sources than the first threshold distance, the fourth mode of the electronic device outputting light optionally includes making little or no adjustment or modification to the light output from the one or more light sources. Thus, the electronic device optionally adjusts a specific light source closer to the user (e.g., within the first threshold distance from the user) more than it adjusts a light source located further from the user than a light source located further from the user. In some implementations, adjusting the light output within the platform in a third, fourth, and / or other manner is based on determining whether changes in ambient light within the platform satisfy some or all of the lighting adjustment conditions and / or priority schemes described above and / or below, including whether the user is within a predetermined distance of the one or more light sources.

[0284] In some embodiments, when the electronic device determines that a user moves from a first location farther than a first threshold distance from the one or more light sources to a second location within a first threshold distance from the one or more light sources, the electronic device adjusts the light output of the one or more light sources located within the first threshold distance from the user (e.g., the one or more light sources located within the first threshold distance from the user at the second location are activated and optionally selected to be adjusted in a third manner or other manner or combination thereof as described herein). For example, when the electronic device detects that the user is within a first threshold distance from a first light source, the electronic device optionally adjusts the light output from the first light source in a first manner (and / or other manners described above and / or below). In another example, when the electronic device detects that the user is farther than a first threshold distance from a second light source different from the first light source, the electronic device optionally adjusts the light output from the second light source in a second manner (and / or other manners as described above and / or below). In some embodiments, the first manner includes an illumination amount (and / or illumination intensity, luminance, color temperature, and / or other characteristics) greater than and / or different from the second manner associated with the second light source. The system automatically adjusts the light output from one or more light sources based on the user's location within the platform and within a first threshold distance from those light sources. This allows the user to view the environment with changing lighting conditions, reducing the need for subsequent user input to manipulate the one or more light sources. This reduces power consumption and extends the battery life of electronic devices by automatically adjusting the lighting associated with specific light sources closer to the user more quickly and efficiently.

[0285] In some implementations, the electronic device is connected to a display component and one or more input devices (such as...) Figure 6N The electronic device communicates with the display 652 in the display unit, and displays user interface elements (such as...) via the display component. Figure 6NThe user interface element 654 can be selected to adjust the light output within the platform in a third manner, different from the first manner. In some embodiments, the electronic device has one or more of the characteristics of the display component of method 800. In some embodiments, the one or more input devices include electronic devices or components capable of receiving user input (e.g., capturing or detecting user input) and sending information associated with the user input to the electronic device. Examples of input devices include physical buttons, knobs, handles and / or switches, touchscreens, mice (e.g., external), trackpads (optionally integrated or external), touchpads (optionally integrated or external), microphones for capturing voice commands or other audio input, remote control devices (e.g., external), another electronic device (e.g., a mobile device detached from the electronic device), handheld devices (e.g., external), controllers (e.g., external), cameras, depth sensors, eye-tracking devices and / or motion sensors (e.g., hand-tracking devices or hand motion sensors).

[0286] In some embodiments, the user interface element is the user interface of an application (such as a platform control application or a lighting application). In some embodiments, the user interface element includes one or more options that, when selected, cause the electronic device to adjust the light output within the platform in a third manner (e.g., in a first, second, and / or other manner) as described above and / or below. In some embodiments, the one or more options include step controls for increasing or decreasing the light output by incremental values ​​(e.g., the amount, intensity, and / or luminance of illumination). In some embodiments, the one or more options include slider controls for adjusting between minimum and maximum values ​​of the light output (e.g., the amount, intensity, and / or luminance of illumination) and / or the color temperature of the light output (e.g., from 1000K (warm color temperature) to 15,000K (cool color temperature)). In some embodiments, the user interface element includes selection elements or options other than the step controls or slider controls described herein. In some embodiments, the user interface element, when selected, causes the electronic device to adjust the overall light output (e.g., color temperature) from all light sources from the one or more light sources, rather than from each individual light source (e.g., rather than manipulating the specific illumination characteristics of each individual light). For example, the electronic device optionally detects a selection of a user interface element, and in response, adjusts the overall color temperature rather than one or more specific lighting characteristics. In some embodiments, the user interface element optionally includes buttons or indicators for increasing or decreasing the color temperature (e.g., warmer or less warm). In some embodiments, when the electronic device adjusts the light output in a first manner or other manner, the electronic device displays a user interface element including one or more lighting values ​​corresponding to the manner of the output light. In some embodiments, details about which specific lighting characteristic is adjusted / manipulated (e.g., an indication of the lighting value corresponding to the manner of the output light) are hidden from the user (e.g., the user interface element does not indicate which specific lighting characteristic is adjusted / manipulated in response to user input directed to the user interface element). In some embodiments, the user interface element includes at least two control options: a first control (e.g., a "+" button), which, when selected, increases one or more corresponding lighting values ​​of one or more lighting characteristics; and a second control (e.g., a "-" button), which, when selected, decreases one or more corresponding lighting values ​​of one or more lighting characteristics or parameters. In some implementations, in response to user input directed to a first control (or a second control), the electronic device increases (or decreases) a lighting characteristic or a combination of lighting characteristics, based on whether the ambient light within the platform satisfies one, a combination of, or all of the lighting adjustment conditions and / or priority schemes described above and / or below.Provide user interface elements to adjust the light output from one or more light sources within the platform, enabling users to quickly and efficiently control the light output from the one or more light sources in the environment. This reduces power consumption and extends the battery life of electronic devices by providing more efficient interaction between the user and the electronic device, and saves power from the one or more light sources.

[0287] In some implementations, in response to detecting a change in ambient light within the platform (e.g., a change similar to or corresponding to the change in ambient light within the platform as described in reference method 700 above), and based on determining that the user within the platform is the first user, the electronic device adjusts the light output within the platform in a third manner, such as... Figure 6B The light source 608a is adjusted at least in part based on user 610. In some embodiments, in response to detecting a change in ambient light within the platform, and based on determining that the user within the platform is a second user, the electronic device adjusts the light output within the platform in a fourth manner different from a third-party adjustment, such as... Figure 6C The light source 608b is adjusted at least in part based on user 620. In some embodiments, adjusting the light output within the platform in a third or fourth manner has one or more characteristics of adjusting the light output within the platform in a first and / or other manner as described above and / or below. In some embodiments, the manner of adjusting the light output is based on the user. For example, a first user optionally prefers (e.g., as indicated by a corresponding first user account) a light output with a first color temperature, such that when the electronic device detects the presence of the first user, the third manner of adjusting the light output includes the first color temperature preferred by the first user. In another example, when the electronic device detects a second user (e.g., lighting preferences indicated by a corresponding user account), the electronic device optionally outputs light with a second color temperature (e.g., a fourth manner) that is warmer or cooler than the first color temperature, according to the lighting preferences of the second user / account.

[0288] In some implementations, when the electronic device detects the presence of both the first user and the second user, the electronic device outputs light in a fifth mode, between the third and fourth modes (e.g., interpolating the third and fourth modes). In some implementations, the second user is associated with a second user account different from the first user's account. In some implementations, the second...

Claims

1. A method, the method comprising: At the electronic devices that communicate with the platform: When one or more light sources are used to output light inside the platform, changes in ambient light inside the platform are detected. In response to detecting a change in ambient light within the interior of the platform: Based on the determination that the change in ambient light inside the platform is caused by a first change in a light source in the environment outside the platform, the light output inside the platform is adjusted in a first manner. as well as Based on the determination that the change in ambient light within the platform is caused by a second change in the light source within the platform, the light output within the platform is adjusted in a second manner, different from the first manner.

2. The method according to claim 1, further comprising: In response to detecting a change in ambient light within the interior of the platform: Based on the determination that a user is interacting with an object within the platform's interior, the light output within the platform's interior is adjusted in a third manner.

3. The method according to any one of claims 1 to 2, further comprising: In response to detecting a change in ambient light within the interior of the platform: Based on the anticipated change in the light source in the environment outside the platform, corresponding to the current time standard, the light output inside the platform is adjusted in a third manner, different from the second manner.

4. The method according to any one of claims 1 to 3, further comprising: In response to detecting a change in ambient light within the interior of the platform: Based on the determination that the platform is operating in a first mode, the light output inside the platform is adjusted in a third manner; as well as Based on the determination that the platform is operating in a second mode different from the first mode, the light output within the platform is adjusted in a fourth manner different from the third-party mode.

5. The method according to any one of claims 1 to 4, further comprising: In response to detecting a change in ambient light within the interior of the platform: Based on determining that a user inside the platform is within a first threshold distance from the one or more light sources, the light output inside the platform is adjusted in a third manner. as well as Based on the determination that a user inside the platform is located at a distance greater than the first threshold from one or more light sources, the light output inside the platform is adjusted in a fourth manner, different from the third-party approach.

6. The method according to any one of claims 1 to 5, wherein the electronic device communicates with a display component and one or more input devices, the method further comprising: User interface elements are displayed via the display component, and these user interface elements can be selected to adjust the light output within the platform in a third manner, different from the first manner.

7. The method according to any one of claims 1 to 6, further comprising: In response to detecting a change in ambient light within the interior of the platform: Based on the determination that the user inside the platform is the first user, the light output inside the platform is adjusted in a third manner; as well as Based on the determination that the user within the platform is a second user, the light output within the platform is adjusted in a fourth manner, different from the third-party approach.

8. The method according to any one of claims 1 to 7, further comprising: In response to detecting a change in ambient light within the interior of the platform: Based on determining that the time of day at the electronic device is the first time of day, the light output inside the platform is adjusted in a third manner; as well as Based on determining that the time of day at the electronic device is the second time of day, the light output within the platform is adjusted in a fourth manner, different from the third-party method.

9. The method according to any one of claims 1 to 8, wherein adjusting the light output within the platform in the first manner comprises: The light output inside the platform is adjusted in the first manner to correspond to the environment outside the platform.

10. The method according to any one of claims 1 to 9, further comprising: In response to detecting a change in ambient light within the interior of the platform: Based on the determination that the platform's architecture is operating in a first mode, the light output within the platform's interior is adjusted in a third manner; as well as Based on the determination that the architecture of the platform is operating in a second mode, the light output within the platform is adjusted in a fourth manner different from the third-party approach.

11. The method according to any one of claims 1 to 10, wherein adjusting the light output within the platform in the first manner comprises: Based on the determination that a first amount of light is leaving the interior of the platform, the light output within the interior of the platform is adjusted in a third manner; as well as Based on the determination that a second amount of light is leaving the interior of the platform, the light output within the platform is adjusted in a fourth manner, different from the third-party method.

12. The method according to any one of claims 1 to 11, wherein adjusting the light output within the platform comprises: Adjust the light generated by the platform.

13. The method according to any one of claims 1 to 12, wherein adjusting the light output within the platform in the first manner comprises: Adjust the light generated by one or more second electronic devices.

14. The method according to any one of claims 1 to 13, wherein adjusting the light output within the platform in the first manner comprises: Adjust the light exiting the interior of the platform.

15. The method according to any one of claims 1 to 14, wherein adjusting the light output within the platform in the first manner comprises: Adjust the light entering the interior of the platform.

16. An electronic device comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: When one or more light sources are used to output light inside the platform, changes in ambient light inside the platform are detected. In response to detecting a change in ambient light within the interior of the platform: Based on the determination that the change in ambient light inside the platform is caused by a first change in a light source in the environment outside the platform, the light output inside the platform is adjusted in a first manner. as well as Based on the determination that the change in ambient light within the platform is caused by a second change in the light source within the platform, the light output within the platform is adjusted in a second manner, different from the first manner.

17. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform a method comprising: When one or more light sources are used to output light inside the platform, changes in ambient light inside the platform are detected. In response to detecting a change in ambient light within the interior of the platform: Based on the determination that the change in ambient light within the platform is caused by a first change in a light source in the environment outside the platform, the light output within the platform is adjusted in a first manner; and Based on the determination that the change in ambient light within the platform is caused by a second change in the light source within the platform, the light output within the platform is adjusted in a second manner, different from the first manner.

18. An electronic device comprising: One or more processors; Memory; and A device for: detecting activation of a second display component while displaying a first region from a first viewpoint via a first display component; and A device for use in response to detecting activation of the second display component: When one or more light sources are used to output light inside the platform, changes in ambient light inside the platform are detected. In response to detecting a change in ambient light within the interior of the platform: Based on the determination that the change in ambient light inside the platform is caused by a first change in a light source in the environment outside the platform, the light output inside the platform is adjusted in a first manner. as well as Based on the determination that the change in ambient light within the platform is caused by a second change in the light source within the platform, the light output within the platform is adjusted in a second manner, different from the first manner.

19. An electronic device comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any one of the methods according to claims 1 to 15.

20. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any one of the methods according to claims 1 to 15.

21. An electronic device, comprising: One or more processors; Memory; and Apparatus for performing any one of the methods according to claims 1 to 15.

22. A method, the method comprising: At the electronic device that communicates with the display components included in the platform: When the display component is operating in the first mode: Facilitating the visibility of the platform's environment into the platform's interior, the environment including one or more environmental factors; The detection indicates data requesting the display component to operate according to a second mode different from the first mode; as well as In response to the detection of data indicating the request to operate the display component according to the second mode: The display component displays visual indications of one or more environmental factors of the environment.

23. The method according to claim 22, further comprising: While displaying the visual indication of one or more environmental factors of the environment via the display component, movement of the one or more environmental factors is detected via one or more sensors; as well as In response to detecting movement of the one or more environmental factors, the visual indication of the one or more environmental factors in the environment is moved according to the movement of the one or more environmental factors.

24. The method according to any one of claims 22 to 23, wherein displaying the visual indication of the one or more environmental factors of the environment comprises: Based on determining that the one or more environmental factors have a first appearance, a second appearance based on the first appearance is used to display the visual indication of the one or more environmental factors; as well as Based on determining that the one or more environmental factors have a third appearance different from the first appearance, a fourth appearance different from the second appearance is displayed to indicate the one or more environmental factors.

25. The method of any one of claims 22 to 24, wherein the visual indication of the one or more environmental factors of the environment is associated with a point of interest or a real-world object.

26. The method according to any one of claims 22 to 25, wherein the visual indication of the one or more environmental factors of the environment includes an indication of a corresponding location in the environment.

27. The method according to any one of claims 22 to 26, wherein the visual indication of the one or more environmental factors of the environment includes a simulation of the sun or a simulation of the moon.

28. The method according to any one of claims 22 to 27, wherein the visual indication of the one or more environmental factors of the environment includes a representation of physical objects in the environment.

29. The method according to any one of claims 22 to 28, further comprising: While displaying the visual indication of one or more environmental factors of the environment via the display component, data indicating a request to operate the display component according to the first mode is detected; as well as In response to the detection of data indicating the request to operate the display component according to the first mode: A representation of the physical object is displayed via the display component based on one or more criteria, including criteria that are met when the physical object included in the environment is not visible to the interior of the platform; and If it is determined that one or more of the criteria are not met, the representation of the physical object displayed via the display component is abandoned.

30. The method of any one of claims 22 to 29, wherein the display component includes a portion integrated with a window of the platform, and displaying the visual indication of the one or more environmental factors comprises: The color is applied to the portion of the display component according to the lighting conditions outside the platform.

31. The method according to any one of claims 22 to 30, wherein displaying the visual indication of the one or more environmental factors of the environment comprises: Based on determining that the environment has a first value for a first visual characteristic, the visual indication is displayed for a second value of the first visual characteristic that is greater than the first value. as well as The visual indication is displayed for a fourth value that is greater than the third value for the first visual characteristic, based on a determination that the environment has a third value different from the first value for the first visual characteristic.

32. The method of claim 31, wherein displaying the visual indication of the one or more environmental factors of the environment comprises: Based on determining that the environment has a first value for a second visual characteristic that is different from the first visual characteristic, the visual indication is displayed for a second value of the second visual characteristic that is less than the first value. as well as Based on the determination that the environment has a third value different from the first value for the second visual characteristic, the visual indication is displayed for a fourth value of the second visual characteristic that is less than the third value.

33. The method according to any one of claims 22 to 32, further comprising: When displaying the visual indications of the one or more environmental factors of the environment: Based on the determination that the color of light entering the platform from outside the platform is outside a predefined range of values, color correction is applied to the light within the platform via the display component; as well as If the color of light entering the platform from outside the platform is determined to be within the predefined range of values, the application of color correction to light within the platform via the display component is abandoned.

34. The method according to any one of claims 22 to 33, wherein the visual indication of the one or more environmental factors of the environment is displayed at a location corresponding to a corresponding location of the one or more environmental factors in the environment.

35. The method according to any one of claims 22 to 34, further comprising: While displaying the visual indication of one or more environmental factors of the environment via the display component, data indicating a request to operate the display component according to the first mode is detected; as well as In response to the detection of data indicating the request to operate the display component according to the first mode: Stop displaying the visual indications of the one or more environmental factors of the environment; and Facilitating the visibility of the platform's environment into the platform's interior, the environment including one or more environmental factors.

36. The method of claim 35, wherein promoting the environment of the platform includes one or more environmental factors of the environment to make the interior of the platform visible, comprising: Adjust the light output inside the platform.

37. An electronic device comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: When the display component is operating in the first mode: Facilitating the visibility of the platform's environment into the platform's interior, the environment including one or more environmental factors; The detection indicates data requesting the display component to operate according to a second mode different from the first mode; as well as In response to the detection of data indicating the request to operate the display component according to the second mode: The display component displays visual indications of one or more environmental factors of the environment.

38. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform a method comprising the following operations: When the display component is operating in the first mode: Facilitating the visibility of the platform's environment into the platform's interior, the environment including one or more environmental factors; The detection indicates data requesting the display component to operate according to a second mode different from the first mode; as well as In response to the detection of data indicating the request to operate the display component according to the second mode: The display component displays visual indications of one or more environmental factors of the environment.

39. An electronic device comprising: One or more processors; Memory; and For use in a device where, when the display component is operating in a first mode: Facilitating the visibility of the platform's environment into the platform's interior, the environment including one or more environmental factors; The detection indicates data requesting the display component to operate according to a second mode different from the first mode; as well as In response to the detection of data indicating the request to operate the display component according to the second mode: The display component displays visual indications of one or more environmental factors of the environment.

40. An electronic device comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any one of the methods according to claims 22 to 36.

41. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any one of the methods according to claims 22 to 36.

42. An electronic device, the electronic device comprising: One or more processors; Memory; and Apparatus for performing any one of the methods according to claims 22 to 36.

Images