Devices, methods, and graphical user interfaces for providing computer-generated experiences

Abstract

The present disclosure relates to devices, methods, and graphical user interfaces for providing computer-generated experiences. A computing system displays, via a first display generation component, a first computer-generated environment and concurrently displays, via a second display generation component: a visual representation of a portion of a user of the computing system in a position to view the first computer-generated environment via the first display generation component, and one or more graphical elements providing visual indications of content in the first computer-generated environment. The computing system changes the visual representation of the portion of the user to represent a change in appearance of the user over a respective time period and changes the one or more graphical elements to represent a change in the first computer-generated environment over the respective time period.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 2021800201704, filed on March 15, 2021, entitled "Apparatus, Method and Graphical User Interface for Providing Computer-Generated Experience". Technical Field

[0002] This disclosure relates in its entirety to a computing system having one or more display generation components and one or more input devices (including, but not limited to, electronic devices that provide virtual reality and mixed reality experiences via one or more displays) for providing computer-generated experiences. Background Technology

[0003] In recent years, the development of computing systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or enhance representations of the physical world. Input devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays) used by computing systems and other electronic computing devices interact with the virtual / augmented reality environment. Exemplary virtual elements include virtual 3D objects, digital images, videos, text, icons, control elements (such as buttons), and other graphics.

[0004] However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems where the manipulation of virtual objects is complex, tedious, and error-prone, impose a significant cognitive burden on users and detract from the immersive experience of virtual / augmented reality environments. Furthermore, these methods take longer than necessary, thus wasting energy. This latter consideration is particularly important in battery-powered devices. In addition, many systems providing virtual reality and / or mixed reality experiences use head-mounted displays that physically isolate the user's face from their surroundings, hindering social interaction and information exchange with the outside world when users engage in virtual reality and mixed reality experiences. Summary of the Invention

[0005] Therefore, there is a need for computing systems with improved methods and interfaces to provide users with computer-generated experiences, making interaction with these systems more efficient and intuitive for users. There is also a need for computing systems with improved methods and interfaces to provide users with computer-generated experiences, thereby facilitating better social interaction, etiquette, and information exchange with the surrounding environment when users participate in various virtual reality and mixed reality experiences. Such methods and interfaces optionally complement or replace conventional methods for providing users with computer-generated reality experiences. These methods and interfaces create more efficient human-computer interfaces by helping users understand the relationship between the inputs provided and the device's responses to those inputs, thereby reducing the quantity, extent, and / or nature of input from the user. Such methods and interfaces also improve the user experience, for example, by reducing errors, interruptions, and time delays caused by the lack of social cues and visual information for the user and others in the same physical environment when users participate in virtual reality and / or mixed reality experiences provided by computing systems.

[0006] The disclosed system reduces or eliminates the aforementioned deficiencies and other problems associated with user interfaces used in computing systems, which have display generation components and one or more input devices. In some embodiments, the computing system is a desktop computer with one or more associated displays. In some embodiments, the computing system is a portable device (e.g., a laptop, tablet, or handheld device). In some embodiments, the computing system is a personal electronic device (e.g., a wearable electronic device, such as a watch or head-mounted device). In some embodiments, the computing system has a touchpad. In some embodiments, the computing system has one or more cameras. In some embodiments, the computing system has a touch-sensitive display (also referred to as a "touchscreen" or "touchscreen display"). In some embodiments, the computing system has one or more eye-tracking components. In some embodiments, the computing system has one or more hand-tracking components. In some embodiments, in addition to one or more display generation components, the computing system also has one or more output devices, including one or more haptic output generators and one or more audio output devices. In some embodiments, the computing system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, and a program or instruction set stored in memory for performing multiple functions. In some implementations, users interact with the GUI through touch and gestures on a touch-sensitive surface using a stylus and / or fingers, movement of the user's eyes and hands in space relative to the GUI or the user's body (as captured by a camera and other motion sensors), and voice input (as captured by one or more audio input devices). In some implementations, functions performed through interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving phone calls, video conferencing, sending and receiving emails, instant messaging, test support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0007] As disclosed herein, the computing system includes at least two display generation components, wherein a first display generation component faces the user and provides the user with a three-dimensional computer-generated experience, and a second display generation component faces away from the user and provides user-related state information (e.g., user eye movements) and / or the computer-generated experience currently viewed by the user (e.g., metadata related to the content viewed by the user and the immersion level associated with that content). Optionally, the first and second display generation components are two displays enclosed in the same housing of a head-mounted display device (HMD) and facing inwards towards the user wearing the HMD and outwards towards the physical environment surrounding the user. The second display generation component optionally provides real-time state information, including a visual representation of the portion of the user obscured by the first display generation component and including metadata related to the content currently shown to the user via the first display generation component and / or the immersion level associated with that content, such that one or more other persons in the user's surrounding physical environment can see the visual information and metadata provided by the second display generation component and act accordingly, such as speaking to the user when appropriate while the user is viewing computer-generated content via the first display generation component, rather than unnecessarily avoiding interaction with the user or inappropriately interrupting the user. In some implementations, the user of the computing system may optionally activate different modes of the computing system to meet his / her expected level of participation and privacy needs when engaging in a computer-generated environment provided via a first display generation component, and the computing system provides status information associated with the various modes to alert people in the surrounding physical environment to such intentions and needs of the user, thereby reducing unexpected, undesirable and / or unnecessary interruptions and interaction avoidance by people in the surrounding physical environment.

[0008] As disclosed herein, when a computing system includes at least two display generation components in the same housing, a second (e.g., outward-facing) display generation component optionally displays contextual information that indicates the availability of a computer-generated experience based on the current context. In response to detecting that a first (e.g., inward-facing) display generation component has been placed in front of a user's eyes (e.g., the user is wearing an HMD with the inward-facing display facing their eyes, or the user is holding an HMD with the inward-facing display in front of their eyes), the computing system provides the computer-generated experience to the user via the first display generation component. (For example, when the user is in a position viewing an outward-facing monitor (when the user is not wearing the HMD on his / her head, when the HMD is placed on a table, etc.)) Based on the current context, automatically alert the user to available computer-generated experiences via an outward-facing monitor and / or automatically display computer-generated experiences corresponding to the current context when an inward-facing monitor is placed in front of the user's eyes (e.g., when the user places the HMD on his / her head or holds the HMD with the internal monitor facing the user's face or eyes), reducing the amount, complexity, and scope of input required for the user to ascertain what computer-generated experiences are available in various contexts and to selectively view the desired computer-generated experience (e.g., not having to wear the HMD all the time, and / or not having to navigate through options to locate the desired CGR content item, and / or activate displayed controls while wearing the HMD to begin the desired CGR experience). In some implementations, depending on whether the first display generating component is actually worn by the user (e.g., strapped to the user's head or body, rather than held in front of the user's eyes by hand), the computing system optionally provides different computer-generated experiences corresponding to the wearing state of the first display generating component (e.g., displaying a preview of the available computer-generated experience (e.g., shortened, two-dimensional or three-dimensional, interactive, etc.) when the first display generating component is not actually worn by the user, and displaying the full version of the available computer-generated experience when the first display generating component is worn by the user). Selectively displaying different versions or different computer-generated experiences not only based on the position of the display generating component relative to the user (e.g., based on whether the position allows the user to see the CGR experience) but also based on whether the display generating component is securely worn by the user (e.g., based on whether the user's hands are free or required to hold the display generating component in its current position) reduces the amount of input required to trigger the desired result and avoids unnecessarily activating the full-on computer-generated experience, thereby saving user time when the user only wants a brief preview of the computer-generated experience and conserving battery power for the display generating component and computing system when they are battery powered.

[0009] As disclosed herein, in some embodiments, the computing system includes a first display generating component and a second display generating component, which are housed in the same enclosure or attached to the same physical support structure. The first and second display generating components are optionally opaque and have their respective display sides facing opposite directions. The display generating components, together with the enclosure or support structure, can be quite bulky and can be worn on and removed from the user's head / body in a cumbersome manner. These display generating components also together form a significant physical barrier between the user and others in the surrounding physical environment. By utilizing an external display (e.g., a second display generation component) to show status information (e.g., name, progress, type, etc.) related to the metadata of the CGR content displayed on the internal display (e.g., a first display generation component), the level of immersion associated with the displayed CGR content (e.g., full passthrough, mixed reality, virtual reality, etc.), and / or the visual characteristics of the displayed CGR content (e.g., changes in color, brightness, etc.), the current display mode of the computing system (e.g., privacy mode, parental control mode, do not disturb mode, etc.), and / or user characteristics (e.g., the appearance of the user's eyes, the user's identifier, etc.), the impact of physical barriers between the user and others in the surrounding environment is reduced, and unnecessary obstacles to desired social interaction and unnecessary interruptions to the user's participation in the computer-generated experience are reduced without requiring the user to remove the display generation component. Furthermore, by utilizing an external display to show contextual information and indications of the context-dependent computer-generated experience, the user is not required to always pick up the HMD and place the internal display in front of their eyes to find out what CGR content is available. Nor is the user required to completely bind the HMD to preview the available CGR experience. The HMD is only required to be fully worn when the user wishes to fully engage with the CGR experience (e.g., interacting with the CGR environment using air gestures and micro-gestures). This reduces the number of times the user needs to place the HMD's internal display in front of their eyes and / or completely strap the HMD to their head without affecting the user's need to know what CGR experiences are available and / or hindering the user's ability to enjoy the desired CGR experience.

[0010] As disclosed herein, a computer-generated experience is provided via a display generation component of a computing system (e.g., a single display generation component of a device, the internal display of an HMD, etc.) in response to a user's physical interaction with a physical object in the real world. Specifically, the computing system displays a visual indication of the availability of the computer-generated experience at a specific location in a three-dimensional environment displayed via the display generation component, the location of which corresponds to the location of a representation of a physical object in the three-dimensional environment. In response to detecting a first-mode physical interaction with a physical object that satisfies preset criteria associated with the physical object, the computing system displays a computer-generated experience associated with the physical object and an optional first-mode physical interaction. For example, the computing system displays a perspective view of the user's hand and the physical object before the user's manipulation of the physical object satisfies the preset criteria, and displays a computer-enhanced representation of the user's hand manipulating the physical object after the preset criteria are met. The automatic initiation of the computer-generated experience in response to detecting a preset physical interaction with a physical object in the real world enhances the user's experience of interacting with the physical object, making the interaction more intuitive and reducing user errors when interacting with the physical object.

[0011] As disclosed herein, the computing system includes a display generating component (e.g., a single display generating component of a device, an internal display of an HMD, etc.) within a housing and provides a user interface (e.g., a button, a touch-sensitive surface, etc.) on the housing of the display generating component. Based on a preset configuration that detects whether a user's hand (e.g., both hands simultaneously) touches the housing of the display generating component when input is detected via the user interface, the computing system determines whether to perform an operation associated with the input detected via the user interface on the housing of the display generating component or to abandon performing that operation. By selecting whether to perform or not perform an operation in response to the detection of input on the housing of the display generating component along with the hand configuration, the system can automatically distinguish between intentional user input and other touches on the housing of the display generating component for purposes other than providing input to trigger certain operations. This helps avoid unintended results, reduces user confusion, and allows users to operate the display generating component faster and easier.

[0012] According to some embodiments, a method is performed at a computing system including a first display generation component, a second display generation component, and one or more input devices. The method includes: displaying a first computer-generated environment via the first display generation component; while displaying the first computer-generated environment via the first display generation component, simultaneously displaying via the second display generation component: a visual representation of a portion of a user of the computing system at a position where the user is viewing the first computer-generated environment via the first display generation component; and one or more graphical elements providing visual indications of content within the first computer-generated environment. Simultaneously displaying the visual representation of the user's portion and the one or more graphical elements includes: changing the visual representation of the user's portion to indicate changes in the user's appearance over a corresponding time period; and changing the one or more graphical elements providing visual indications of content within the first computer-generated environment to indicate changes in the first computer-generated environment over a corresponding time period.

[0013] According to some embodiments, a method is performed at a computing system including a first display generation component, a second display generation component, and one or more input devices. The method includes: displaying a computer-generated environment via the first display generation component; while displaying the computer-generated environment via the first display generation component, displaying state information corresponding to the computing system via the second display generation component, including simultaneously displaying: a visual representation of a portion of the computing system's user at a location viewing the computer-generated environment via the first display generation component, and one or more graphical elements providing visual indications of content within the computer-generated environment; detecting a corresponding event; and in response to detecting the corresponding event: changing the immersion level of the computer-generated environment displayed via the first display generation component; and changing the state information displayed via the second display generation component, including changing the appearance of the visual representation of that portion of the computing system's user.

[0014] According to some embodiments, a method is performed at a computing system including a first display generation component, a second display generation component, and one or more input devices. The method includes: displaying one or more user interface elements via the second display generation component; while displaying the one or more user interface elements via the second display generation component, detecting that the first display generation component has moved to a predetermined orientation relative to a corresponding portion of the user; and in response to detecting that the first display generation component has moved to the predetermined orientation relative to the corresponding portion of the user: upon determining that when the first display generation component moves to the predetermined orientation relative to the corresponding portion of the user, the computing system is in a first state, wherein the first state corresponds to displaying a first user interface element via the second display generation component, thereby displaying a first user interface via the first display generation component when the first display generation component is in the predetermined orientation relative to the corresponding portion of the user; and upon determining that when the first display generation component moves to the predetermined orientation relative to the corresponding portion of the user, the computing system is in a second state, wherein the second state corresponds to displaying a second user interface element via the second display generation component instead of displaying a first user interface element via the second display generation component, thereby displaying a second user interface via the first display generation component when the first display generation component is in the predetermined orientation relative to the corresponding portion of the user, wherein the second user interface is different from the first user interface.

[0015] According to some embodiments, a method is performed at a computing system including a first display generation component and one or more input devices, the method comprising: detecting a first trigger event corresponding to the first display generation component being placed in a first predefined configuration relative to a user; and in response to detecting the first trigger event: providing a first computer-generated experience via the first display generation component based on determining that the computing system including the first display generation component is worn by the user when in the first predefined configuration relative to the user; and providing a second computer-generated experience different from the first computer-generated experience via the first display generation component based on determining that the computing system including the first display generation component is not worn by the user when in the first predefined configuration relative to the user.

[0016] According to some embodiments, a method is performed at a computing system including a first display generation component and one or more input devices, the method comprising: while displaying a representation of a physical object at a location in a three-dimensional environment corresponding to the location of the physical object in a physical environment, displaying a computer-generated experience corresponding to the physical object that can be used for display via the first display generation component; while displaying a computer-generated experience that can be used for display via the first display generation component, detecting an interaction with the physical object in the physical environment; and in response to detecting an interaction with the physical object in the physical environment: based on determining that the interaction with the physical object in the physical environment satisfies a first criterion corresponding to the physical object, displaying the computer-generated experience corresponding to the physical object via the first display generation component; and based on determining that the interaction with the physical object in the physical environment does not satisfy the first criterion, abandoning the display of the computer-generated experience corresponding to the physical object.

[0017] According to some embodiments, a method is performed at a computing system including a housing, a first display generating component housed within the housing, and one or more input devices. The method includes: detecting a first hand on the housing housing the first display generating component; and in response to detecting the first hand on the housing housing the first display generating component: abandoning the execution of an operation associated with the first hand based on determining that the first hand was detected and in conjunction with the detection of a second hand on the housing; and performing an operation associated with the first hand based on determining that the first hand was detected on the housing but the other hand was not detected on the housing.

[0018] According to some embodiments, a computing system includes one or more display generating components (e.g., one or more displays, projectors, head-mounted displays, etc., enclosed in the same or different housings), one or more input devices (e.g., one or more cameras, touch-sensitive surfaces, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), optional one or more haptic output generators, one or more processors, and memory storing one or more programs; the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions therein that, when executed by a computing system having one or more display generating components, one or more input devices (e.g., one or more cameras, touch-sensitive surfaces, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), and optional one or more haptic output generators, cause the devices to perform or cause the operation of any of the methods described herein to be performed. According to some embodiments, a graphical user interface on a computing system having one or more display generating components, one or more input devices (e.g., one or more cameras, touch-sensitive surfaces, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), optional one or more haptic output generators, memory, and one or more processors for executing one or more programs stored in the memory includes one or more elements displayed in any of the methods described herein, which are updated in response to input as described in any of the methods described herein. According to some embodiments, the computing system includes: one or more display generating components, one or more input devices (e.g., one or more cameras, touch-sensitive surfaces, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), and optional one or more haptic output generators; and means for performing or causing the operation of any of the methods described herein. According to some embodiments, information processing means for a computing system having one or more display generating components, one or more input devices (e.g., one or more cameras, touch-sensitive surfaces, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), and optional one or more haptic output generators includes means for performing or causing the operation of any of the methods described herein to be performed.

[0019] Therefore, improved methods and interfaces are provided for computing systems with one or more display generation components to deliver computer-generated experiences to users, making interaction with the computing system more efficient and intuitive for users. Improved methods and interfaces are also provided for these computing systems to deliver computer-generated experiences to users, thereby facilitating better social interaction, etiquette, and information exchange with the surrounding environment when users participate in various virtual reality and mixed reality experiences. Such methods and interfaces optionally complement or replace conventional methods for delivering computer-generated reality experiences to users. These methods and interfaces create more efficient human-computer interfaces by reducing the quantity, extent, and / or nature of user input, for example, by helping users understand the relationship between the input provided and the device's response to those inputs. Such methods and interfaces also improve the user experience, for example, by reducing errors, interruptions, and time delays caused by the lack of social cues and visual information for the user and others in the same physical environment when users participate in virtual reality and / or mixed reality experiences provided by the computing system.

[0020] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not exhaustive; in particular, many additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, description, and claims. Furthermore, it should be pointed out that the language used in this specification has been chosen in principle for readability and instruction purposes, and such choice may not be necessary to depict or define the subject matter of the invention. Attached Figure Description

[0021] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0022] Figure 1 This is a block diagram illustrating the operating environment of a computing system for providing a CGR experience according to some implementation schemes.

[0023] Figure 2 This is a block diagram illustrating a controller of a computing system configured to manage and coordinate the user's CGR experience according to some implementation schemes.

[0024] Figure 3 This is a block diagram illustrating a display generation component of a computing system configured to provide a CGR experience to a user, according to some implementation schemes.

[0025] Figure 4 This is a block diagram illustrating a hand tracking unit of a computing system configured to capture user gesture input according to some embodiments.

[0026] Figure 5 This is a block diagram illustrating an eye-tracking unit of a computing system configured to capture a user's gaze input according to some embodiments.

[0027] Figure 6 This is a flowchart illustrating a flare-assisted gaze tracking pipeline according to some implementation schemes.

[0028] Figures 7A to 7E A computing system is illustrated, comprising a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., back-to-back facing opposite directions, facing different angles, such that they cannot be viewed simultaneously by the same user, etc.)). In some embodiments, the computing system displays content to the user via the first display generating component, while, according to some embodiments, it displays dynamically updated status information associated with the user and / or content via the second display generating component (e.g., representations corresponding to changes in the user's appearance behind the first display generating component, metadata of the content, changes in immersion level associated with content playback, etc.).

[0029] Figures 7F to 7J A computing system is illustrated, comprising a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., back-to-back facing opposite directions, facing different angles, such that they cannot be viewed simultaneously by the same user, etc.)). In some embodiments, the computing system displays indications of different computer-generated experiences via the second display generating component based on contextual information associated with the computing system (e.g., the positions of the first and second display generating components, the user's identity, the current time, etc.), and according to some embodiments, the display of different computer-generated experiences corresponding to the contextual information is triggered via the first display generating component in response to changes in spatial relationships (e.g., from not facing the user's eyes to facing the user's eyes, from being placed on a table or in a bag to being held at the user's eye level, etc.) and / or changes in the wearing state of the first display generating component relative to the user (e.g., from being supported by the user's hand to being supported by the user's head / nose / ear, from never being worn on the user's head / body to being worn on the user's head / body, etc.). In some implementations, the computing system includes only a single display generation component and / or does not display an indication of the available computer-generated experiences before being placed in a preset configuration relative to the user and starting to display different computer-generated experiences based on the wear status of the display generation component.

[0030] Figures 7K to 7MIndications are shown regarding the availability of displaying a computer-generated experience at a location corresponding to a representation of a physical object in a mixed reality environment, according to some implementation schemes, and triggering the display of a computer-generated experience corresponding to a physical object in response to the detection of a preset physical interaction with a physical object in the real world.

[0031] Figures 7N to 7Q It is shown that, according to some embodiments, an operation is performed or not performed in response to the detection of an input on the housing of the display generating component, depending on whether one or two hands are detected on the housing when the input is detected.

[0032] Figure 8 It is a flowchart of a method for displaying a computer-generated environment and state information associated with the computer-generated environment and state information associated with a user in a location viewing the computer-generated environment, according to some implementation schemes.

[0033] Figure 9 It is a flowchart of a method for displaying a computer-generated environment and state information associated with the computer-generated environment and state information associated with a user in a location viewing the computer-generated environment, according to some implementation schemes.

[0034] Figure 10 It is a flowchart of a method for providing a computer-generated experience based on contextual information according to some implementation schemes.

[0035] Figure 11 This is a flowchart of a method for providing a computer-generated experience based on the wearing status of the displayed generated components, according to some implementation schemes.

[0036] Figure 12 It is a flowchart of a method for triggering the display of a computer-generated experience based on the detection of a preset physical interaction with a physical object in the real world, according to some implementation schemes.

[0037] Figure 13 It is a flowchart of a method for performing operations in response to input on the housing of a component to be generated, according to some implementation schemes. Detailed Implementation

[0038] According to some implementations, this disclosure relates to a user interface for providing a computer-generated reality (CGR) experience to a user.

[0039] The systems, methods, and GUIs described in this paper improve user interface interactions with virtual / augmented reality environments in a variety of ways.

[0040] In some implementations, the computing system includes a first display generating component and a second display generating component (e.g., a separate monitor, monitors enclosed in the same housing but facing different directions (e.g., back-to-back facing opposite directions, facing different angles, such that they cannot be viewed simultaneously by the same user, etc.)). The first display generating component displays a computer-generated environment that provides a computer-generated experience to a user in a position viewing content presented via the first display generating component (e.g., the display side of the user facing the display generating component (e.g., the side of the physical environment illuminated by a projector, the side of the monitor emitting light capable of forming an image on the user's retina, etc.)). Optionally, the first display generating component provides a computer-generated experience with different levels of immersion, corresponding to different amounts of visual and audio information from the surrounding physical environment, which can still be perceived via the first display generating component while providing the computer-generated experience. During normal operation (e.g., when a user wears an HMD including a first display generating component and / or faces the display side of the first display generating component), the first display generating component obstructs the user from directly viewing the surrounding physical environment, and also obstructs others from viewing the user's face or eyes when the user is in a position to view content displayed via the first display generating component. In some embodiments, the first display generating component is the internal display of the HMD facing the user's eyes when the HMD is placed on the user's head. Traditionally, when a user is in a position to view content displayed via the display generating component, the user can choose to see or not see the physical environment by switching between computer-generated environments with different levels of immersion (e.g., between a fully transparent mode, a mixed reality mode, or a virtual reality mode). However, others in the surrounding environment facing the back of the display generating component have little or no visual cues regarding the user's attention state, what is being displayed on the display generating component, and / or whether the user can see the surrounding environment and the people within it. This imbalance of visual information (and optional audio information) on both sides of the display generating component makes social interactions between the user and others in the surrounding environment unnatural and inefficient. Many considerations can benefit from the computing system using a second display generating component to display an appropriate amount of visual information that conveys user-related status information and / or content displayed to the user via the first display generating component to people in the surrounding environment. The display of status information by the second display generating component may optionally occur whenever the first display generating component is in use, or may optionally be triggered in response to the detection of the presence of other people in the same physical environment and / or in response to the detection of indications that others may want to engage in social discourse with the user (e.g., by entering the same room, looking in the user's direction, waving at the user, etc.).In some embodiments, displaying state information on the second display generating component includes showing a representation of a portion of the user (e.g., a portion of the user obscured by the first display generating component while the user is in a position viewing content displayed via the first display generating component), which dynamically updates based on changes in the user's appearance (e.g., changes in the portion of the user obscured by the first display generating component). In some embodiments, displaying state information also includes displaying graphical elements that provide visual indications of the content currently displayed via the first display generating component (e.g., concurrently with the representation showing that portion of the user). Methods and systems that use the second display generating component to display updated state information related to a user viewing content displayed via the first display generating component and metadata associated with the state of that content (e.g., name, progress, immersion level, display mode, etc.) allow others in the user's surrounding environment to gain a useful insight into the user's current state as the user engages in the computer-generated experience, without fully revealing the computer-generated experience to the surrounding environment. In some embodiments, the representation of the user's portion obscured by the first display generating component (e.g., the user's eyes or face) and the graphical elements indicating the state of the content displayed via the first display generating component are displayed on different display layers of the second display generating component and updated independently of each other. In some implementations, the updating of graphical elements representing the user's portion and the state of the displayed content on different display layers of the second display generating component provides a more realistic view of the user's state behind the head-mounted display device that houses both the first and second display generating components. The state information displayed on the second display generating component enables the user to maintain social connection with people in their surroundings while engaging in a computer-generated experience through the first display generating component. The dynamically updated state information on the second display generating component, showing the state of the user's eyes and the state of the content displayed to the user, improves the user's engagement in the computer-generated experience when the user is in a public or semi-public environment. This is achieved, for example, by encouraging appropriate social interaction when the user expects such interaction, reducing unnecessary avoidance of social interaction by others in the surrounding environment due to the lack of visual cues indicating the user's consent to participate in social activities, notifying others of the appropriate time to interrupt the user's engagement in the computer-generated experience, and reducing unwelcome interruptions to the user's engagement due to the lack of visual cues indicating the user's desire to remain undisturbed.

[0041] As described above, many considerations can benefit from the computing system using a second display generating component to display an appropriate amount of visual information that conveys user-related state information and content displayed to the user via the first display generating component to others in the surrounding environment. In some embodiments, state information is displayed on the second display generating component as long as the first display generating component is in use. In some embodiments, state information is displayed only in response to the detection of the presence of others in the same physical environment and / or in response to the detection of indications that others in the same physical environment may want to engage in social discourse with the user (e.g., by entering the same room, looking in the user's direction, waving at the user, etc.). Displaying state information on the second display generating component optionally includes displaying a representation of a portion of the user (e.g., a portion of the user obscured by the first display generating component when the user is in a position viewing content displayed via the first display generating component) and displaying graphical elements that provide visual indications of the content currently displayed via the first display generating component. Furthermore, in some embodiments, the representation of that portion of the user is updated in conjunction with changes in the level of immersion of the computer-generated experience displayed via the first display generating component. The method and system of using a second display generation component to display state information related to the user viewing content shown via a first display generation component and to the content viewed by the user, and updating the state information (including updating the appearance of the user's representation of that part according to changes in immersion level associated with the provision of content), allows others in the user's surrounding environment to gain a useful insight into the user's current state as the user engages in a computer-generated experience, without fully revealing the computer-generated experience to the surrounding environment. In some embodiments, updates to the representation of a part of the user (e.g., the user's eyes or face) obscured by the first display generation component and updates to the graphical elements showing the state of the content displayed by the first display generation component are shown on different display layers and updated independently of each other. Displaying the representation of the user's part and the graphical elements showing the state of the content on different display layers provides a more realistic view of the user's state behind a head-mounted display device that houses both the first and second display generation components. In some embodiments, the state information shown via the second display generation component (e.g., including the user's representation and the graphical elements showing the state of the content) optionally provides visual indications of a number of different usage patterns of the computing system corresponding to the different needs of the user and others in the same physical environment as the user. This allows users to maintain social connections with people in their surroundings while engaging in computer-generated experiences.The dynamically updated status information on the second display generation component, which shows the state of the user's eyes and the state of the content shown to the user, improves the user's experience of participating in computer-generated activities when the user is in a public or semi-public environment. For example, it can encourage appropriate social interaction when the user expects such interaction, reduce unnecessary avoidance of social interaction by others in the surrounding environment due to the lack of visual cues for the user's consent to participate in social activities, notify others of the appropriate time to interrupt the user's participation in the computer-generated experience, and reduce unwelcome interruptions to the user's participation experience due to the lack of visual cues for the user to remain undisturbed.

[0042] In some implementations, the computing system includes a first display generating component and a second display generating component facing two different directions (e.g., a separate display, displays enclosed in the same housing but facing different directions (e.g., back-to-back facing opposite directions, facing different angles such that they cannot be viewed simultaneously by the same user, etc.)). The first display generating component displays a computer-generated environment that provides the user with a computer-generated experience when the user enters a position to view content presented via the first display generating component (e.g., facing the side of a physical environment illuminated by a projector, facing the side of a display emitting light capable of forming an image on the user's retina, etc.). The user may be in a position to see content presented on the second display generating component before placing the first display generating component relative to the user's position and orientation to view the content displayed thereon (e.g., by moving the display generating component or the user himself or both). In an exemplary scenario, the first display generating component is an internal display of the HMD facing the user's eyes when the HMD is placed on the user's head, and the second display generating component is an external display of the HMD that the user can see when the HMD is located on a table or in the user's outstretched hand away from the user's face and not placed on the user's head or held near the user's eyes. As disclosed herein, the computing system utilizes a second display generation component to display indications of the availability of different computer-generated experiences based on contextual information (e.g., location, time, user identity, user authorization level, etc.), and in response to detecting that a first display generation component has been moved to a predetermined position and orientation relative to the user, enabling the user to view content displayed via the first display generation component (e.g., as a result of the movement, the first display generation component is facing the user's eyes), thereby triggering the display of the selected computer-generated experience. Optionally, the displayed computer-generated experience is selected based on the state of the second display generation component at the time corresponding to the time the first display generation component is moved to a predetermined position and orientation relative to the user. Indicating the availability of computer-generated experiences on the second display generation component based on contextual information and automatically triggering the display of the selected computer-generated experience on the first display generation component based on the state of the second display generation component (and the state of the contextual information) and changes in the orientation of the first display generation component relative to the user reduces the time and amount of input required to achieve the desired result (e.g., obtaining information about available experiences in relation to the current context and starting the desired computer-generated experience), and reduces user errors and the time spent browsing and starting available computer-generated experiences using a conventional user interface.

[0043] In some implementations, a user may position and orient the first display generating component relative to the user to view the displayed content in different ways, such as impromptu or temporary (e.g., held at a distance in front of the user's eyes or held near the user's eyes by hand) or more formal and established (e.g., strapped to or otherwise worn on the user's head or face without support from the user's hands). The computing system selectively displays different computer-generated experiences (e.g., different versions of a computer-generated experience, different computer-generated experiences corresponding to different characteristics of the user or context, previews of experiences compared to the actual experience, etc.) depending on how the first display generating component is positioned and oriented relative to the user, enabling the user to view the content displayed on the first display generating component. Responding to a triggering event corresponding to the placement of the first display generation component in a predefined configuration relative to the user (e.g., automatically initiating the display of a computer-generated experience via the first display generation component in the absence of additional user input in the user interface provided by the first display generation component), and selectively displaying different computer-generated experiences according to how the first display generation component is held in that position and orientation (e.g., with or without support from the user's hand, with or without support from another mechanism besides the user's hand, etc.), the time and amount of input required to achieve the desired result (e.g., initiating the desired computer-generated experience) are reduced, and user errors and the time spent browsing and initiating available computer-generated experiences using a conventional user interface are reduced.

[0044] In some implementations, displaying information on the second display generation component (e.g., status information related to the user's eyes, the status of content displayed via the first display generation component, the display mode of the computing system, indications of available computer-generated experiences, etc.) helps reduce the number of times the user must put on and take off the HMD, including both the first and second display generation components, and / or activate or deactivate the computer-generated experience (e.g., to deal with others in the surrounding physical environment) and / or search for the desired computer-generated experience. This helps save the user's time, reduce power consumption, and reduce user errors, and improves the user experience when the user uses these display generation components. In some implementations, preset modes of physical manipulation of real-world physical objects are detected, and these preset modes serve as trigger conditions for initiating computer-generated experiences associated with physical objects. In some implementations, prior to initiating a computer-generated experience associated with a physical object, visual indications of available computer-generated experiences and optional visual guidance (e.g., previews and animations) on how to initiate the computer-generated experience are displayed at a location in the mixed reality environment corresponding to the location of the representation of the physical object in the mixed reality environment. In addition to displaying visual cues related to the availability of the computer-generated experience and / or visual guidance related to the physical manipulation required to trigger the computer-generated experience, preset physical manipulations of physical objects are used to trigger the display of the computer-generated experience associated with the physical object. This allows users to achieve the desired result (e.g., enter the desired computer-generated experience) more intuitively, quickly, and with less input. This user interaction heuristic also helps reduce user errors when users interact with physical objects, making the human-computer interface more efficient and saving power for battery-operated computing systems.

[0045] In some implementations, the display generation component is housed within a housing that includes sensors (or additionally associated external sensors) for detecting touch or hover input near or above various portions of the housing. Different types of touch and / or hover inputs (e.g., based on movement patterns (e.g., taps, swipes, etc.), durations (e.g., long, short, etc.), intensities (e.g., light, deep, etc.)) at different locations above or near the exterior of the housing trigger different operations associated with the display generation component or the computer-generated environment displayed by it. Interaction heuristics are used to determine whether an operation should be performed based on whether a single hand or two hands are detected on the housing when input is detected. Using the number of hands detected on the housing as an indicator of whether the user intends to provide input or is merely adjusting the position of the display generation component with their hand helps reduce accidental or unintentional operations of the display generation component, making the human-machine interface more efficient and saving power for battery-operated computing systems.

[0046] Figures 1 to 6 A description of an exemplary computing system for providing a CGR experience to users is provided. Figures 7A to 7E A computing system according to some embodiments is shown, which displays content to a user via a first display generation component, while simultaneously displaying dynamically updated status information associated with the user and / or the content via a second display generation component. Figures 7F to 7J A computing system according to some embodiments is shown, which displays indications of different computer-generated experiences via a second display generation component based on contextual information, and in response to detecting a change in spatial relationship relative to the user, optionally triggers the display of different computer-generated experiences corresponding to the contextual information via a first display generation component based on the wearing state of the first display generation component relative to the user. Figures 7K to 7M Indications are shown for displaying the availability of computer-generated experiences associated with physical objects in an augmented reality environment according to some implementation schemes, and the display of computer-generated experiences corresponding to physical objects is triggered in response to the detection of preset physical interactions with physical objects. Figures 7N to 7Q The illustration shows, according to some embodiments, the selection to perform or not perform the operation associated with the input detected on the housing of the display generating component based on whether one or two hands are detected on the housing when the input is detected on the housing. Figure 8 It is a flowchart of a method for displaying computer-generated environment and status information according to some implementation schemes. Figure 9 It is a flowchart of a method for displaying computer-generated environment and status information according to some implementation schemes. Figure 10 It is a flowchart of a method for providing a computer-generated experience based on contextual information according to some implementation schemes. Figure 11 This is a flowchart of a method for providing a computer-generated experience based on the wearing status of the displayed generated components, according to some implementation schemes. Figure 12 It is a flowchart of a method for triggering the display of a computer-generated experience based on physical interaction with a physical object, according to some implementation schemes. Figure 13 This is a flowchart of a method for performing operations in response to input detected on the housing of a display generating component, according to some implementation schemes. Figures 7A to 7Q The user interface in the document is used to show... Figures 8 to 13 The process in.

[0047] In some implementation schemes, such as Figure 1As shown, a CGR experience is provided to the user via an operating environment 100 including a computing system 101. The computing system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), one or more display generation components 120 (e.g., one or more head-mounted devices (HMDs) enclosed in the same housing and facing different directions or enclosed in separate housings, HMDs with internal and external displays, one or more displays, one or more projectors, one or more touchscreens, etc.), one or more input devices 125 (e.g., eye-tracking devices 130, hand-tracking devices 140, other input devices 150), one or more output devices 155 (e.g., speakers 160, haptic output generators 170, and other output devices 180), one or more sensors 190 (e.g., image sensors, light sensors, depth sensors, haptic sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, speed sensors, etc.), and optional one or more peripheral devices 195 (e.g., home appliances, wearable devices, etc.). In some implementations, one or more of the input device 125, output device 155, sensor 190, and peripheral device 195 are integrated with the display generation component 120 (e.g., in a head-mounted device (e.g., on the housing of an HMD or on the outward-facing display of an HMD) or in a handheld device).

[0048] In describing the CGR experience, various terms are used to distinguish several related but different environments that a user can sense and / or interact with (e.g., interacting with inputs detected by the computing system 101 that generates the CGR experience, causing the computing system to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computing system 101). The following is a subset of these terms:

[0049] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. Physical environments, such as physical parks, include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through senses such as sight, touch, hearing, taste, and smell.

[0050] Computer-Generated Reality: Conversely, a computer-generated reality (CGR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In CGR, a subset of a person's physical motion, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are modulated in a manner consistent with at least one physical law. For example, a CGR system can detect a person's head rotation and, in response, modulate the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the modulation of the characteristics of virtual objects in the CGR environment can be done in response to a representation of physical motion (e.g., a voice command). A person can use any of their senses to sense and / or interact with CGR objects, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects that create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. For example, audio objects can enable audio transparency, which selectively introduces ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, people can sense and / or interact only with audio objects.

[0051] Examples of CGR include virtual reality and mixed reality.

[0052] Virtual Reality: A virtual reality (VR) environment is a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing human avatars are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through the simulation of a person's presence within the computer-generated environment and / or through the simulation of a subgroup of physical movements of a person within the computer-generated environment.

[0053] Mixed Reality: Compared to VR environments, which are designed to be entirely based on computer-generated sensory input, mixed reality (MR) environments refer to simulated environments designed to incorporate sensory input from the physical environment, or its representations, in addition to computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any state between, but not limited to, a purely physical environment as one end and a virtual reality environment as the other. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems used to present an MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical objects or their representations from the physical environment). For example, a system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0054] Examples of mixed reality include augmented reality and augmented virtual reality.

[0055] Augmented Reality (AR): An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation of the physical environment. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or semi-transparent display, allowing a person to perceive the virtual objects overlaid on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects overlaid on the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects onto the physical environment, such as as holograms or on physical surfaces, allowing a person to perceive the virtual objects superimposed on the physical environment. Augmented reality environments also refer to simulated environments in which the representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system can transform images from one or more sensors to apply a selected viewpoint (e.g., viewpoint) different from the viewpoint captured by the imaging sensor. Furthermore, the representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions of it, such that the modified portions are representative but not realistic versions of the original captured image. Still further, the representation of the physical environment can be transformed by graphically removing or blurring portions of it.

[0056] Augmented Virtual: An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. Sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park could have virtual trees and virtual buildings, but a person's face could be realistically reproduced from an image taken of a physical person. Similarly, virtual objects could adopt the shape or color of a physical object imaged by one or more imaging sensors. Furthermore, virtual objects could use shadows that correspond to the sun's position within the physical environment.

[0057] Hardware: Many different types of electronic systems enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may combine one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto a person's retina. Projection systems can also be configured to project virtual objects into a physical environment, such as as holograms, or onto a physical surface. In some embodiments, controller 110 is configured to manage and coordinate the user's CGR experience. In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. References below... Figure 2The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device located locally or remotely relative to scene 105 (e.g., physical setup / environment). For example, the controller 110 is a local server located within scene 105. Alternatively, the controller 110 is a remote server (e.g., a cloud server, central server, etc.) located outside scene 105. In some embodiments, the controller 110 is communicatively coupled to display generation components 120 (e.g., one or more HMDs, displays, projectors, touchscreens, etc., enclosed in the same or different housings) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, controller 110 is included within the housing (e.g., physical enclosure) of display generation component 120 (e.g., HMD or portable electronic device including a display and one or more processors), one or more input devices in input device 125, one or more output devices in output device 155, one or more sensors in sensor 190 and / or one or more peripheral devices in peripheral device 195, or shares the same physical housing or support structure with one or more of the aforementioned devices.

[0058] In some embodiments, at least one of the display generation components 120 is configured to provide a CGR experience to a user (e.g., at least the visual components of the CGR experience). In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. The following is relative to... Figure 3 An example of display generation component 120 is described in more detail. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0059] According to some implementation schemes, when a user is virtually and / or physically present within scene 105, at least one of the display generation components 120 provides the user with a CGR experience.

[0060] In some embodiments, the display generating component is worn on a part of the user's body (e.g., on his / her head, his / her hand, etc.). Therefore, at least one of the display generating components 120 includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, at least one of the display generating components 120 surrounds the user's field of view. In some embodiments, at least one of the display generating components 120 is a handheld device (such as a smartphone or tablet) configured to present CGR content, and the user holds the device with the display facing the user's field of view and the camera facing scene 105. In some embodiments, the handheld device is optionally placed within a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, at least one of the display generating components 120 is a CGR chamber, housing, or room configured to present CGR content, wherein the user does not wear or hold the display generating component 120. Many user interfaces described with reference to one type of hardware (e.g., a handheld device or a tripod-mounted device) for displaying CGR content can be implemented on another type of hardware (e.g., an HMD or other wearable computing device) for displaying CGR content. For example, a user interface illustrating interaction with CGR content triggered by an interaction occurring in the space in front of a handheld device or tripod-mounted device can be similarly implemented using an HMD, where the interaction occurs in the space in front of the HMD and the response to the CGR content is displayed via the HMD. Similarly, a user interface illustrating interaction with CGR content triggered by movement of a handheld device or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)) can be similarly implemented using an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)).

[0061] Despite Figure 1 The relevant features of the operating environment 100 are shown in this disclosure, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure further relevant aspects of the exemplary embodiments disclosed herein.

[0062] Figure 2This is a block diagram of an example controller 110 according to some implementation schemes. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the implementation schemes disclosed herein. Therefore, as a non-limiting example, in some embodiments, controller 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing kernels, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0063] In some embodiments, one or more communication buses 204 include circuitry for interconnecting and communicating between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0064] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 220 optionally includes one or more storage devices located remotely from the one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures or subsets thereof, including optional operating system 230 and CGR experience module 240.

[0065] Operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate single or multiple CGR experiences for one or more users (e.g., single CGR experiences for one or more users, or multiple CGR experiences for corresponding groups of one or more users). To this end, in various embodiments, CGR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0066] In some implementations, the data acquisition unit 241 is configured to acquire data from... Figure 1 The data acquisition unit 241 includes at least one or more display generation components in display generation component 120, and optionally acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.) from one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data acquisition unit 241 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0067] In some implementations, the tracking unit 242 is configured to map scene 105 and track at least one or more display generation components in display generation components 120 relative to the scene. Figure 1 The tracking unit 242 tracks the position / location of one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195 in scenario 105. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more portions of the user's hand, and / or the position / location of one or more portions of the user's hand relative to... Figure 1 The movement of scene 105 relative to at least one of the display generating components 120 and / or relative to a coordinate system (defined relative to the user's hand). The following refers to... Figure 4 The hand tracking unit 244 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the position or movement of the user's gaze (or more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hand)) or relative to CGR content displayed via at least one of the display generation components 120. The following description is relative to... Figure 5 The eye-tracking unit 243 is described in more detail.

[0068] In some implementations, coordination unit 246 is configured to manage and coordinate the CGR experience presented to the user by at least one of the display generation components 120, and optionally by one or more of the output device 155 and / or peripheral device 195. For this purpose, in various implementations, coordination unit 246 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0069] In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, position data, etc.) to at least one of the display generation components 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data transmission unit 248 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0070] Although the data acquisition unit 241, tracking unit 242 (e.g., including eye tracking unit 243 and hand tracking unit 244), coordination unit 246, and data transmission unit 248 are shown residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, tracking unit 242 (e.g., including eye tracking unit 243 and hand tracking unit 244), coordination unit 246, and data transmission unit 248 may reside in a separate computing device.

[0071] also, Figure 2 This is used more as a functional description of various features that can exist in a specific implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 2 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0072] Figure 3This is a block diagram illustrating an example of at least one display generation component 120 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the computing system including the display generating component 120 (e.g., HMD) also includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.) in the same housing, one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional internal and / or external image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0073] In some embodiments, one or more communication buses 304 include circuitry for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 306 include inertial measurement units (IMUs), accelerometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, haptic engines, and / or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0074] In some embodiments, one or more CGR displays 312 are configured to provide a user with a CGR experience and optional status information related to the CGR experience. In some embodiments, one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some embodiments, one or more CGR displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, an HMD includes a single CGR display. In another example, an HMD includes CGR displays for each of the user's eyes. In some embodiments, one or more CGR displays 312 are capable of presenting MR and VR content. In some embodiments, one or more CGR displays 312 are capable of presenting AR or VR content. In some embodiments, the HMD includes one or more CGR displays facing the user's eyes when the HMD is placed on the user's head, and one or more CGR displays facing away from the user's eyes (e.g., towards the external environment). In some embodiments, the computing system is a CGR room or CGR housing, and the CGR room or CGR housing includes internally CGR displays that provide CGR content to the user inside the CGR room or housing, and optionally includes externally one or more peripheral displays that display status information related to the CGR content and the status of the user inside.

[0075] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand and optionally the user's arm (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward in order to acquire image data corresponding to the scene that the user would see in the absence of the display generation component 120 (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0076] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 320 optionally includes one or more storage devices located remotely from the one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures, or subsets thereof, including optional operating system 330 and CGR rendering module 340.

[0077] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. To this end, in various embodiments, CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR mapping generation unit 346, a data transmission unit 348, and optionally other operational units for displaying status information related to the user and the CGR content.

[0078] In some implementations, the data acquisition unit 342 is configured to acquire data from at least... Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For the purposes described, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0079] In some implementations, the CGR rendering unit 344 is configured to render CGR content and associated status information via one or more CGR displays 312. For the aforementioned purposes, in various implementations, the CGR rendering unit 344 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0080] In some implementations, the CGR mapping generation unit 346 is configured to generate a CGR mapping based on media content data (e.g., a 3D mapping of a mixed reality scene or a physical environment in which computer-generated objects may be placed to generate a computer-generated reality mapping). For the aforementioned purposes, in various implementations, the CGR mapping generation unit 346 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0081] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For the stated purpose, in various embodiments, the data transmission unit 348 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0082] Although the data acquisition unit 342, CGR rendering unit 344, CGR mapping generation unit 346, and data transmission unit 348 are shown residing in a single device (e.g., Figure 1 The data acquisition unit 342, the CGR rendering unit 344, the CGR mapping generation unit 346, and the data transmission unit 348 are located on the display generation component 120, but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR rendering unit 344, the CGR mapping generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0083] also, Figure 3 This is used more as a functional description of various features that may exist in a particular implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separate. For example, Figure 3 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0084] Figure 4 This is a schematic illustration of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1 Controlled by hand tracking unit 244 Figure 2 To track the position / location of one or more parts of a user's hand, and / or the position / location of one or more parts of the user's hand relative to... Figure 1The movement is defined in scenario 105 (e.g., relative to a portion of the user's surrounding physical environment, relative to at least one of the display generating components 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system (defined relative to the user's hand)). In some embodiments, the hand tracking device 140 is part of at least one of the display generating components 120 (e.g., embedded in or attached to the same housing as the display generating component (e.g., in a head-mounted device)). In some embodiments, the hand tracking device 140 is separate from the display generating component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0085] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures at least three-dimensional scene information including the human user's hand 406. The image sensor 404 captures images of the hand at sufficient resolution to distinguish the fingers and their corresponding positions. The image sensor 404 typically captures images of other parts of the user's body, or possibly all parts of the body, and may have scaling capabilities or be a dedicated sensor with increased magnification to capture images of the hand at the desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of scene 105, or serves as the image sensor for capturing the physical environment of scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a way that uses the field of view of the image sensor 404 or a portion thereof to define an interaction space in which hand movements captured by the image sensor are considered input to the controller 110.

[0086] In some implementations, image sensor 404 outputs a sequence of frames containing 3D mapping data (and, in addition, possibly color image data) to controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided via an application programming interface (API) to an application running on the controller, which in turn drives display generation component 120. For example, a user can interact with software running on controller 110 by moving his hand 408 and changing his hand gestures.

[0087] In some embodiments, image sensor 404 projects a speckle pattern onto a scene including hand 406 and captures an image of the projected pattern. In some embodiments, controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) via triangulation based on the lateral offset of the specks in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. This method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from image sensor 404. In this disclosure, it is assumed that image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis such that the depth coordinates of points in the scene correspond to the z-component measured by the image sensor. Alternatively, hand tracking device 440 may use other 3D mapping methods, such as stereo imaging or time-of-flight measurement, based on single or multiple cameras or other types of sensors.

[0088] In some implementations, hand tracking device 140 captures and processes time-series depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on a processor in image sensor 404 and / or controller 110 processes the 3D mapping data to extract image block descriptors of the hand from these depth maps. The software may match these descriptors with image block descriptors stored in database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and fingertips.

[0089] The software can also analyze the trajectories of the hand and / or fingers across multiple frames in a sequence to identify gestures. The pose estimation function described herein can be alternated with motion tracking, such that patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to find pose changes occurring in the remaining frames. Pose, motion, and gesture information is provided to an application running on controller 110 via the aforementioned API. This application can, for example, move and modify the image presented on display generation unit 120 in response to the pose and / or gesture information, or perform other functions.

[0090] In some embodiments, the software may be downloaded to controller 110 electronically, for example, via a network, or alternatively, may be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, database 408 is also stored in memory associated with controller 110. Alternatively or additionally, some or all of the described functions of the computer may be implemented in dedicated hardware, such as custom or semi-custom integrated circuits or programmable digital signal processors (DSPs). Although in Figure 4The controller 110 is shown, but for example, as a separate unit from the image sensor 440, some or all of the controller's processing functions may be performed by a suitable microprocessor and software, or by dedicated circuitry within the housing of the hand tracking device 402, or by other devices associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing function of the image sensor 404 may also be integrated into a computer or other computerized device controlled by the sensor output.

[0091] Figure 4 It also includes a schematic diagram of a depth map 410 captured by image sensor 404 according to some embodiments. As described above, the depth map comprises a matrix of pixels with corresponding depth values. Pixel 412 corresponding to hand 406 has been segmented from the background and wrist in this map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z-distance from image sensor 404), where gray shadows become darker as depth increases. Controller 110 processes these depth values ​​to identify and segment components of the image that have human hand features (i.e., a group of adjacent pixels). These features may include, for example, overall size, shape, and frame-to-frame motion from the depth map sequence.

[0092] Figure 4 The diagram also schematically illustrates the hand skeleton 414 ultimately extracted by the controller 110 from the depth map 410 of the hand 406 according to some embodiments. Figure 4 In this configuration, the skeleton 414 is overlaid on the hand background 416, which has already been segmented from the original depth map. In some embodiments, key feature points of the hand, and optionally those on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, fingertips, the center of the palm, the end of the hand that connects to the wrist, etc.), are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points across multiple image frames to determine, according to some embodiments, the gesture performed by the hand or the current state of the hand.

[0093] Figure 5 An eye-tracking device 130 is shown. Figure 1 Exemplary implementations of ). In some implementations, the eye-tracking device 130 comprises an eye-tracking unit 244 ( Figure 2The eye-tracking device 130 is controlled to track the position and movement of the user's gaze relative to scene 105 or relative to CGR content displayed via at least one of the display generation components 120. In some embodiments, the eye-tracking device 130 is integrated with at least one of the display generation components 120. For example, in some embodiments, when the display generation component 120 is part of a head-mounted device (such as a headset, helmet, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating CGR content for the user to view and components for tracking the user's gaze relative to the CGR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, when the display generation component is provided by a handheld device or a CGR chamber, the eye-tracking device 130 is optionally a separate device from the handheld device or CGR chamber. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 is optionally used in conjunction with at least one display generating component that is also head-mounted or at least one display generating component that is not head-mounted. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally used in conjunction with a head-mounted display generating component. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally part of a non-head-mounted display generating component.

[0094] In some embodiments, at least one of the display generating components 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display frames including left and right images in front of the user's eyes, thereby providing the user with a 3D virtual view. For example, a head-mounted display generating component may include a left optical lens and a right optical lens (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, at least one of the display generating components 120 may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generating component may have a transparent or semi-transparent display on which virtual objects are displayed, allowing the user to view the physical environment directly through the transparent or semi-transparent display. In some embodiments, at least one of the display generating components 120 projects virtual objects onto the physical environment. The virtual objects may be projected, for example, onto a physical surface or as holograms, allowing an individual to observe the virtual objects superimposed on the physical environment using the system. In this case, separate display panels and image frames for the left and right eyes may not be necessary.

[0095] like Figure 5 As shown, in some embodiments, gaze tracking device 130 includes at least one eye-tracking camera (e.g., an infrared (IR) or near-infrared (NIR) camera) and an illumination source (e.g., an array or ring of IR or NIR light sources, such as LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be pointed at the user's eye to receive IR or NIR light reflected directly from the eye, or alternatively, it may be pointed at "hot" mirrors located between the user's eye and the display panel, which reflect the IR or NIR light from the eye back to the eye-tracking camera while allowing visible light to pass through. Gaze tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps), analyzes these images to generate gaze tracking information, and transmits the gaze tracking information to controller 110. In some embodiments, the user's two eyes are tracked separately using corresponding eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked using corresponding eye-tracking cameras and illumination sources.

[0096] In some implementations, a device-specific calibration procedure is used to calibrate the eye-tracking device 130 to determine parameters for the eye-tracking device in a specific operating environment 100, such as the 3D geometry and parameters of the LEDs, camera, thermal mirror (if present), eye lenses, and display. The device-specific calibration procedure can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration procedure can be automated or manual. According to some implementations, a user-specific calibration procedure may include estimations of eye parameters for a specific user, such as pupil position, foveal position, optical axis, visual axis, interocular distance, etc. According to some implementations, once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, a flash-assisted method can be used to process the images captured by the eye-tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0097] like Figure 5As shown, the eye-tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system. The gaze tracking system includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-infrared (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye-tracking camera 540 may be pointed toward a mirror 550 located between the user's eye 592 and a display 510 (e.g., the left or right display panel of a head-mounted display, the internal display of a head-mounted device, or the display of a handheld device, a projector, etc.). These mirrors reflect the IR or NIR light from the eye 592 while allowing visible light to pass through. Figure 5 (as shown in the top portion), or alternatively, it can be pointed towards the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion), Figure 5 (As shown in the bottom part).

[0098] In some implementations, controller 110 renders AR or VR frames 562 (e.g., left and right frames for the left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye-tracking camera 540 for various purposes, such as processing frame 562 for display. Controller 110 optionally estimates the user's gaze point on display 510 based on the gaze tracking input 542 obtained from eye-tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated based on gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0099] The following describes several possible use cases for the user's current gaze direction and is not intended to be limiting. As an example use case, controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, controller 110 may generate virtual content at a higher resolution in the concave region determined according to the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content in the view based at least partially on the user's current gaze direction. As yet another example, the controller may display specific virtual content in the view based at least partially on the user's current gaze direction. As another exemplary use case in an AR application, controller 110 may guide an external camera used to capture the physical environment for CGR experiences to focus in the determined direction. The external camera's autofocus mechanism may then focus on an object or surface in the environment that the user is currently looking at on display 510. As another exemplary use case, eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of eye lens 520 so that the virtual object the user is currently looking at has appropriate convergence / divergence to match the convergence of the user's eyes 592. The controller 110 can use gaze tracking information to guide the eye lens 520 to adjust its focus so that the nearby object that the user is looking at appears at the correct distance.

[0100] In some embodiments, the eye-tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lens 520), an eye-tracking camera (e.g., eye-tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) mounted in a wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eyes 592. In some embodiments, the light source may be arranged in a ring or circle around each lens in the head-mounted device, such as... Figure 5 As shown in the diagram. In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0101] In some embodiments, the display 510 emits light in the visible light range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. It should be noted that the position and angle of the eye-tracking camera 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, cameras 540 with a wider field of view (FOV) and cameras 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, cameras 540 operating at one wavelength (e.g., 850 nm) and cameras 540 operating at different wavelengths (e.g., 940 nm) may be used on each side of the user's face.

[0102] like Figure 5 The implementation of the gaze tracking system shown can be used, for example, in computer-generated reality (e.g., including virtual reality and / or mixed reality) applications to provide users with computer-generated reality (e.g., including virtual reality, augmented reality and / or augmented virtual) experiences.

[0103] Figure 6 A flare-assisted gaze tracking pipeline according to some embodiments is illustrated. In some embodiments, the gaze tracking pipeline uses a flare-assisted gaze tracking system (e.g., such as...) Figure 1 and Figure 5 The eye-tracking device 130 shown is used to implement this. The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and flash in the current frame. When not in tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and flash in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in tracking state.

[0104] like Figure 6 As shown, the gaze-tracking camera captures left and right images of the user's left and right eyes. The captured images are then fed into a gaze-tracking pipeline for processing to begin at 610. As indicated by the arrow returning to element 600, the gaze-tracking system can continue capturing images of the user's eyes, for example, at a rate of 60 to 120 frames per second. In some embodiments, each set of captured images can be fed into the pipeline for processing. However, in some embodiments or under certain conditions, not all captured frames are processed by the pipeline.

[0105] At 610, for the currently captured image, if the tracking state is yes, the method proceeds to element 640. At 610, if the tracking state is no, the image is analyzed to detect the user's pupil and flash, as indicated at 620. At 630, if the pupil and flash are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0106] At 640, if proceeding from element 410, the current frame is analyzed to track the pupil and flashes in part based on previous information from the previous frame. At 640, if proceeding from element 630, the tracking state is initialized based on the pupils and flashes detected in the current frame. The processing result at element 640 is checked to verify that the tracking or detection result is credible. For example, the result may be checked to determine whether a sufficient number of pupils and flashes were successfully tracked or detected in the current frame for performing gaze estimation. At 650, if the result is not credible, the tracking state is set to no, and the method returns to element 610 to process the next image of the user's eye. At 650, if the result is credible, the method proceeds to element 670. At 670, the tracking state is set to YES (if not already yes), and the pupil and flash information is passed to element 680 to estimate the user's gaze point.

[0107] Figure 6 This is intended as an example of an eye-tracking technology that can be used in a particular specific implementation. As those skilled in the art will recognize, according to various embodiments, other existing or future eye-tracking technologies can be used to replace or in combination with the flash-assisted eye-tracking technology described herein in the computing system 101 for providing a CGR experience to a user.

[0108] In this disclosure, various input methods are described in relation to interaction with a computing system. When an example is provided using one input device or method, and another example is provided using another input device or method, it should be understood that each example is compatible with and optionally utilizes the input device or method described relative to the other example. Similarly, various output methods are described in relation to interaction with a computing system. When an example is provided using one output device or method, and another example is provided using another output device or method, it should be understood that each example is compatible with and optionally utilizes the output device or method described relative to the other example. Similarly, various methods are described in relation to interaction between a computing system and a virtual or mixed reality environment. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and optionally utilizes the methods described relative to the other example. Therefore, this disclosure discloses embodiments that are combinations of features of a plurality of examples without exhaustively listing all features of the embodiments in the description of each exemplary embodiment.

[0109] User interface and related processes

[0110] Now turn our attention to implementations of user interfaces (“UIs”) and associated processes that can be implemented on computing systems (such as portable multifunction devices or head-mounted devices) having one or more display generation components, one or more input devices, and (optionally) one or more cameras.

[0111] Figures 7A to 7E A computing system according to some implementation schemes is shown (e.g., Figure 1 The computing system 101 or Figure 4 The computing system 140 (e.g.) includes at least a first display generation component (e.g., a display 7100) and a second display generation component (e.g., a display 7102), wherein the computing system displays computer-generated content to a user via the first display generation component (e.g., the display 7100) and displays dynamically updated status information associated with the user and / or the content via the second display generation component (e.g., the display 7102). Figures 7A to 7E Used to illustrate the process described below, including Figures 8 to 13 The process in.

[0112] like Figure 7AAs shown in the left portion, a first display generating component (e.g., display 7100) is present at location A7000-a and displays CGR content (e.g., 3D movies, virtual reality games, videos, 3D environments including user interface objects, etc.). A first user 7202 is also present at location A7000-a.

[0113] like Figure 7A As shown in the right portion, a second display generating component (e.g., display 7102) is present at location B7000-b and displays status information corresponding to the CGR content presented by the first user 7202 and / or via the first display generating component (e.g., display 7100). Figure 7A In the exemplary scenario shown, the second user 7204 also exists at location B7000-b.

[0114] like Figure 7AAs shown, the spatial relationship between the first display generating component (e.g., display 7100) and the first user 7202 places the first user 7202 in a position to view CGR content presented via the first display generating component. For example, the first user 7202 faces the display side of the first display generating component. In some embodiments, the first display generating component is the internal display of the HMD, and the spatial relationship represented in A 7000-a, where the display 7100 and the first user 7202 coexist in the same location, corresponds to the first user wearing or holding the HMD with the internal display of the HMD facing the user's eyes. In some embodiments, the first user is in a position to view CGR content presented via the first display generating component when facing a portion of the physical environment illuminated by the projection system of the first display generating component. For example, virtual content is projected onto a portion of the physical environment, and the user sees the virtual content and that portion of the physical environment through a camera view of that portion of the physical environment or through a transparent portion of the first display generating component when the user faces the display side of the first display generating component. In some embodiments, when a user faces the display side of the first display generating component, the first display generating component emits light that forms an image on the user's retina. For example, virtual content is displayed by an LCD or LED display as part of a view of the physical environment that overlays or replaces the view displayed by the LCD or LED display, and the user facing the display side of the LCD or LED display can see the virtual content along with that portion of the physical environment. In some embodiments, the first display generating component displays a camera view of the physical environment in front of the first user or includes a transparent or semi-transparent portion through which the first user can see a portion of the physical environment in front of the first user. In some embodiments, the portion of the physical environment visible to the first user through the first display generating component is a portion of the physical environment corresponding to the display side of the second display generating component 7102 (e.g., including the display side of the second display generating component and optionally the position B 7000-b of the second user 7204). In some implementations, the display side of the second display generating component is the side of the second display generating component that is away from the first user when the first user is in a position to view the content shown by the first display generating component (e.g., when the first user is facing the display side of the first display generating component), and this side emits light to form an image that can be viewed by another person (e.g., a second user 7204 or another person in a physical environment facing the face or eyes of the first user) facing a predetermined portion of the first user.

[0115] like Figure 7AAs shown, the spatial relationship between the second display generating component (e.g., display 7102) and the second user 7204 places the second user 7204 in a position to view the status information presented by the second display generating component. For example, the second user 7204 is located in front of and / or facing the display side of the second display generating component. In some embodiments, the second display generating component is an external display of an HMD that also includes an internal display (e.g., represented by display 7100) that presents CGR content to the first user 7202. In this embodiment, the spatial relationship represented by the coexistence of display 7102 and the second user 7204 in the same location B 7000-b corresponds to the second user being in a part of the physical environment facing the external display of the HMD (e.g., the physical environment also hosts the first display generating component and the first user 7202). In some embodiments, the first display generating component displays a camera view of the physical environment in front of the first user or includes a transparent or semi-transparent transmission portion through which the first user can see a portion of the physical environment in front of the first user, and this portion of the physical environment included in the camera view or transmission portion is also a portion of the physical environment in front of the display side of the second display generating component. In some embodiments, the second display generating component is positioned back-to-back with the first display generating component such that a portion of the physical environment in front of the display side of the second display generating component 7102 (e.g., including the display side of the second display generating component and the optional position B7000-b of the second user 7204) is also in front of the first user and is within the first user's field of view when neither the first nor the second display generating component obscures the first user's face.

[0116] As explained above and will be reiterated here, although Figure 7A (and Figures 7B to 7JThe first display generating component (e.g., display 7100) and the second display generating component (e.g., display 7102) are shown as being located in two separate and unconnected parts of the physical environment. However, it should be understood that the first display generating component and the second display generating component are optionally housed in the same housing (e.g., the housing of a single HMD) or attached to the same support structure (e.g., attached back to back or attached to opposite sides of a single wall or surface) and facing different (e.g., substantially opposite) directions. Therefore, location A 7000-a represents a first part of the physical environment, from which a first user (e.g., first user 7202) facing the display side of the first display generating component can see the content presented via the first display generating component (e.g., CGR content), and the first user (e.g., first user 7202) cannot see the content presented via the second display generating component (e.g., status information) from the first part; and location B 7000-b represents a second part of the same physical environment, from which another user (e.g., second user 7204) facing the display side of the second display generating component cannot see the content presented via the first display generating component (e.g., CGR content), and the other user (e.g., second user 7204) can see the content presented via the second display generating component (e.g., status information) from the second part. In the disclosure presented herein, the first display generating unit and the second display generating unit are controlled by the same computing system (e.g., HMD, portable electronic device separately housed from the display generating unit, portable electronic device having two displays facing different directions, remote server computer, etc.), and unless otherwise specified, the user of the computing system generally refers to a person who can control at least the first display generating unit to place the first display generating unit or himself / herself in a position that enables him / her to see the CGR content shown via the first display generating unit.

[0117] like Figure 7AAs shown, a computing system controlling the first and second display generating components communicates with a first image sensor (e.g., camera 7104) and a second image sensor (e.g., camera 7106). The first image sensor is configured to capture an image of a portion of a physical environment (e.g., location A 7000-a) that includes at least a portion of a first user (e.g., the face and / or eyes of the first user 7202) facing the display side of the first display generating component (e.g., display 7100) and excludes the second user 7204 (or, in the case where the first display generating component is the internal display of an HMD worn by the first user, excludes any other user). The second image sensor is configured to capture an image of a portion of a physical environment (e.g., location B 7000-b) that excludes that portion of the first user (e.g., the face or eyes of the first user 7202) but includes at least a portion of the second user (e.g., the portion of the second user 7204 within the field of view of the first user 7202 provided by the first display generating component). As previously discussed, in some embodiments, the portion of the physical environment captured by the second image sensor 7106 includes the portion of the physical environment that is in the first user's field of view when the first user's eyes are not physically obscured by the presence of the second display generating component (and optionally the first display generating component). Similarly, in some embodiments, the portion of the physical environment captured by the first image sensor 7104 includes portions of the user (e.g., the user's face or eyes) that are physically obscured by the presence of the first display generating component (and optionally the second display generating component). In some embodiments, the computing system also communicates with the first image sensor, the second image sensor, and / or other image sensors to receive images of the hands and wrists of the first user and / or the second user in order to recognize gesture input provided by the first user and / or the second user. In some embodiments, the first image sensor 7104 is also used to capture gaze input provided by the first user. In some embodiments, the first image sensor and the second image sensor are optionally used as image sensors for capturing gesture input from the first user and / or the second user.

[0118] In some embodiments, the computing system optionally controls one or more audio output devices that provide audio output (e.g., sound of CGR content) to a first user located at location A 7000-a and optionally to a second user located at location B 7000-b (e.g., status indication sound or warning, sound of CGR content, etc.). In some embodiments, the computing system optionally (e.g., by one or more active or passive noise suppression or cancellation components) partially or completely isolates location A and the first user from sound propagating from location B, and optionally partially or completely isolates location B and the second user from sound propagating from location A. In some implementations, the amount of active sound blocking or sound pass-through is determined by the computing system based on the current level of immersion associated with the CGR content shown via the first display generation component (e.g., no sound blocking in pass-through mode, partial sound blocking in mixed reality mode, complete sound blocking in virtual reality mode, etc.) and optionally based on whether another user is present at location B (e.g., no sound blocking when no one is present at location B, sound blocking when a person is present at location B or the noise level exceeds a threshold level, etc.).

[0119] In some implementation schemes, such as Figure 7A As shown, when the first user 7202 is in a position viewing CGR content (e.g., the first user 7202 is positioned alongside the first display generating component in position A and at least partially facing the display side of the first display generating component, the first user wears the HMD on her head and holds the HMD in such a way that the internal display is in front of her eyes, etc.), the computing system displays CGR content 7002 via the first display generating component (e.g., display 7100 or the internal display of the HMD). Figure 7A It is shown as 7002-a). Figure 7AAt the time shown, the computing system is displaying Movie X (e.g., a 3D movie, a 2D movie, an interactive computer-generated experience, etc.). The movie is displayed in a mixed reality mode, where the content of the movie and a representation of the physical environment (e.g., a representation of location B (e.g., a portion of the physical environment in front of the first user and obscured by the presence of the first display generating component)) are simultaneously visible through the first display generating component. In some embodiments, this mixed reality mode corresponds to an intermediate level of immersion associated with the CGR content presented via the first display generating component. In some embodiments, the intermediate level of immersion also corresponds to partial obstruction or partial transmission of sound propagating from the physical environment (e.g., a portion of the physical environment surrounding location B (e.g., the first user)). In this example, the representation of the physical environment includes a representation 7010 of the second user 7204 located at location B 7000-b, in front of the second display generating component 7102 (e.g., also in front of the back side of the first display generating component 7100). Figure 7A(Illustrated as 7010-a). In some embodiments, the representation of the physical environment includes a camera view of a portion of the physical environment located within the first user's field of view when the user's eyes are not obstructed by the presence of the first and second display generating components (e.g., when the first user is not wearing the HMD or holds the HMD in front of his / her eyes). In mixed reality mode, CGR content 7002 (e.g., Movie X, 3D augmented reality environment, user interface, virtual objects, etc.) is displayed as covering or replacing at least a portion, but not all, of the representation of the physical environment. In some embodiments, the first display generating component includes a transparent portion through which the first user can see a portion of the physical environment. In some embodiments, in mixed reality mode, CGR content 7002 (e.g., Movie X, 3D augmented reality environment, user interface, virtual objects, etc.) is projected onto a physical surface or blank space in the physical environment and is visible along with the physical environment through the transparent portion or viewable through the transparent portion of the first display generating component or through the camera view of the physical environment provided by the first display generating component. In some embodiments, CGR content 7002 is displayed as part of an overlay display and obscures at least a portion, but not all, of the view of the physical environment visible through a transparent or semi-transparent portion of the first display generating component. In some embodiments, the first display generating component 7100 does not provide a view of the physical environment, but instead provides a fully virtual environment (e.g., without a camera view or transparent pass-through portion) augmented with a real-time visual representation of the physical environment currently captured by one or more sensors (e.g., a stylized representation or segmented camera image). In mixed reality modes (e.g., augmented reality based on a camera view or transparent display, or augmented virtuality based on a virtualized representation of the physical environment), the first user is not fully immersed in the computer-generated environment and is still receiving sensory information (e.g., visual and audio) directly corresponding to the physical environment surrounding the first user and the first display generating component.

[0120] like Figure 7A As shown, while the computing system displays CGR content 7002-a (e.g., Movie X, 3D augmented reality environment, user interface, virtual object, etc.) in mixed reality mode via the first display generation component 7100, the computing system also displays status information related to the first user and the CGR content via the second display generation component (e.g., display 7102 or an external display of the HMD). Figure 7AAs shown in the right portion, the second display generating component (e.g., display 7102 or an external display of the HMD) displays one or more graphic elements representing the state of the CGR content 7002 displayed via the first display generating component (e.g., display 7100 or an internal display of the HMD), and a representation 7006 of at least a portion of the first user 7202 in front of the display side of the first display generating component (e.g., display 7100 or an internal display of the HMD). Figure 7A (Shown as 7006-a). In this example, one or more graphical elements representing the state of the CGR content displayed via the first display generation component optionally include an identifier of the CGR content (e.g., the name of movie X), a progress bar 7004 indicating the current progress of the CGR content (e.g., in...). Figure 7A The visual representation of CGR content shown in Figure 7004-a) and Figure 7008 (e.g., in...) Figure 7A This is illustrated as 7008-a). In some embodiments, the visual representation of CGR content 7008 makes some CGR content obscured (e.g., through blurring, distortion, etc.) and merely conveys the feeling of variation in the CGR content and its color or hue. Figure 7A As shown in the right portion, the representation 7006 of this portion of the first user 7202 optionally includes a camera view of the first user's face or a graphical representation generated based on the camera view of the first user's face. In some embodiments, the representation of this portion of the first user 7202 optionally includes a camera view of the first user's eyes or a graphical representation generated based on a camera view of the first user's eyes in front of the display side of the first display generating component. In some embodiments, the representation of this portion of the first user 7202 is displayed on a different display layer than the display layer of one or more graphical elements representing the state of CGR content displayed via the first display generating component. In some embodiments, the simultaneous display of the representation of the state of the CGR content and the representation of this portion of the first user by the second display generating component provides an indication of the physical environment and CGR content being displayed via the first display generating component in mixed reality mode. By displaying a visual representation of the first user's face and / or eyes, as well as a representation of the state of the CGR content viewed by the first user, on the second display generating component when the first user's face and / or eyes are obscured by the presence of the first display generating component (and optionally the second display generating component) (e.g., by the presence of an HMD including an internal display and an external display), other users in the first user's surrounding physical environment are provided with more information for initiating or restraining interactions with the first user or for making themselves / herself act appropriately in the presence of the first user.

[0121] Figure 7B follow Figure 7A Furthermore, it is shown that at a later time, the CGR content has undergone further development on the first display generating component (e.g., display 7100 or the internal display of the HMD), and the appearance of the first user has changed. For example, the change in the appearance of the first user is due to the movement of at least a portion of the first user 7202 (e.g., the first user's eyes or face) relative to the first display generating component (e.g., the movement includes lateral movement of the first user's eyeballs, blinking of the user's eyes, closing or opening of the user's eyes, up and down movement of the user's eyeballs, etc.; and / or the movement includes the movement of the user's face or head within position A 7000-a relative to the display side of the first display generating component (e.g., moving away from or towards the first display generating component, etc.)) (e.g., while the first user is still wearing the HMD and / or facing the internal display of the HMD). At this time, the CGR content 7002 is still displayed in mixed reality mode, and the representation 7010 of the physical environment (e.g., including position B of the second user 7204) (e.g., shown as 7010-b) is still displayed simultaneously with the CGR content 7002 via the first display generating component. In some embodiments, any changes in the appearance of the physical environment (e.g., movement of the second user 7204 relative to the first display generating component, the second display generating component, and / or the first user, etc.) are also reflected by the representation of the physical environment 7010 shown by the first display generating component. In some embodiments, based on the movement of that part of the first user relative to the first display generating component (e.g., movement of the first user's eyes or face), the computing system updates the representation 7006 displayed via the second display generating component 7102 (e.g., in...). Figure 7B This is illustrated as 7006-b). For example, when the first user 7202 or a portion thereof (e.g., the user's face or eyes) moves in a direction toward a first edge of the first display generating unit 7100 (e.g., the left edge of the display side of the first display generating unit, the top edge of the display side of the first display generating unit, etc.), the representation 7006 of that portion of the first user shown on the display side of the second display generating unit also moves toward the corresponding second edge of the second display generating unit (e.g., the right edge of the display side of the second display generating unit (e.g., corresponding to the left edge of the display side of the first display generating unit, the top edge of the display side of the second display generating unit (e.g., corresponding to the top edge of the display side of the first display generating unit)). In addition to updating the representation 7006 of that portion of the first user, the computing system also updates the representation of the state of the CGR content on the second display generating unit. For example, updating the progress bar 7004 (e.g., in...). Figure 7B The diagram shows 7004-b) to illustrate the self- Figure 7AThe playback of the CGR content has progressed by a certain amount since the indicated time. In some embodiments, the representation 7008 of the CGR content shown on a second display generation component (e.g., display 7102, external display of the HMD, etc.) is also updated based on the current appearance of the CGR content 7002 shown on a first display generation component (e.g., display 7100, internal display of the HMD, etc.). Figure 7B (Illustrated as 7008-b). In some embodiments, real-time updates of the appearance of the first user's portion are shown (e.g., changes and movements of the first user's face and eyes are shown behind the first display generating component) and real-time or periodic updates of the state of the CGR content shown by the first display generating component are shown, allowing others in the physical environment surrounding the first user (e.g., at location B) to obtain information about the first user's attentional state and whether the current time is suitable to talk to or interrupt the first user. In some embodiments, while changes in the first user's appearance and CGR content are reflected by updates of the state information shown by the second display generating component, any changes in the appearance of the physical environment (e.g., movement of the second user 7204 relative to the first display generating component, the second display generating component, and / or the first user, etc.) are also reflected by the representation of the physical environment shown by the first display generating component 7010.

[0122] Figure 7C follow Figure 7A It also shows that at a later time, the CGR content has made further progress on the first display generation unit 7100, and the second user 7204 has moved relative to the second display generation unit 7204 and is moving from... Figure 7A The scene shown is compared to viewing the second display generating unit from different angles. At this time, the CGR content 7002 is still displayed in mixed reality mode, and the representation 7010 (e.g., shown as 7010-c) of the physical environment (e.g., including the location B of the second user 7204) is still simultaneously displayed via the first display generating unit 7100. Based on the movement of the second user 7204 relative to the second display generating unit (and relative to the first display generating unit when the first and second display generating units have a spatially fixed relationship relative to each other (e.g., fixed back-to-back in the same housing of the HMD), the computing system updates the representation 7010 displayed via the first display generating unit (e.g., display 7100 or the internal display of the HMD). Figure 7C(Shown as 7010-c). For example, when the second user 7204 or a portion thereof moves in a direction toward the third edge (e.g., right edge, top edge, etc.) of the display side of the second display generating unit 7102, the representation 7010 of that portion of the second user shown on the display side of the first display generating unit 7100 also moves toward the corresponding fourth edge of the first display generating unit (e.g., the left edge of the display side of the first display generating unit (e.g., corresponding to the right edge of the display side of the second display generating unit), the top edge of the display side of the first display generating unit (e.g., corresponding to the top edge of the display side of the second display generating unit)). Based on the change in the CGR content shown on the first display generating unit (e.g., display 7100, internal display of the HMD, etc.), the computing system also updates the representation of the state of the CGR content shown on the second display generating unit (e.g., display 7102, external display of the HMD, etc.). For example, updating the progress bar 7004 (e.g., in...). Figure 7C The diagram shows 7004-c) to illustrate the self- Figure 7A The playback of the CGR content has advanced by a second amount since the time shown. Figure 7B The playback of the CGR content has progressed three times since the indicated time. In some embodiments, the representation 7008 of the CGR content 7002 shown on a second display generation component (e.g., display 7102, external display of the HMD, etc.) is also updated based on the current appearance of the CGR content 7002 shown on a first display generation component (e.g., display 7100, internal display of the HMD, etc.). Figure 7C It is shown as 7008-c). Figure 7A and Figure 7BThe contrast in appearance of the status information shown (e.g., including representations 7006, 7008, progress bar 7004, etc.) indicates that, with respect to the same relative spatial position between the first display generating unit and the portion of the first user 7202 represented in the status information shown by the second display generating unit, representation 7006 of the first user 7202 is displayed at a depth different from the depth of representation 7008 of the CGR content, and optionally at a depth different from the depth of other status information (e.g., progress bar 7004). This difference in display depth with respect to the display side of the second display generating unit 7102 or the position of the second user 7204 causes a visual parallax effect. For example, when the second user 7204 moves relative to the second display generating unit (e.g., display 7102, external display of the HMD, etc.), representation 7006 of the first user 7202 and representation 7008 of the CGR content appear to move by different amounts on the display side of the second display generating unit (and appear to move relative to each other). In some embodiments, the representation 7008 of the CGR content is displayed as a diffusion layer between the representation 7006 of the first user's portion and representations of other status information (e.g., the name of the CGR content, progress bar 7004, etc.). In some embodiments, the representation of the first user's portion is displayed in the display layer furthest from the display side of the second display generating component, compared to the display layer of the representation of the CGR content and the representation of other status information shown by the second display generating component.

[0123] In some embodiments, the first display generating component and the second display generating component are positioned back-to-back in an HMD worn on the head of a first user or placed in front of the user's face (e.g., their respective display sides face different directions (e.g., substantially opposite directions)). In some embodiments, the second display generating component displays a visual representation of the first user's eyes, which is generated based on a real image of the first user's eyes using one or more image processing filters. For example, the visual representation of the first user's eyes may optionally be generated by reducing the opacity of the camera image of the first user's eyes, increasing its transparency, reducing its color saturation level, reducing its brightness level, reducing its pixel resolution, reducing its color resolution, etc. In some embodiments, the amount of modification applied to the various display characteristics of the corresponding camera image of the first user's eyes may optionally be specified relative to the values ​​of various display characteristics of the representation 7008 of CGR content simultaneously displayed by the second display generating component 7102. For example, when the representation of CGR content is relatively dark (e.g., having a first brightness value range), the representation for the eyes is also made darker, more translucent, and / or has lower color saturation (e.g., having a second brightness value range, a second transparency value range, and a second color saturation value range selected based on the first brightness value range); and when the representation of CGR content is brighter (e.g., having a second brightness value range larger than the first brightness value range), the representation for the eyes is made brighter, less translucent, and / or has higher color saturation (e.g., having a third brightness value range, a third transparency value range, and a third color saturation value range selected based on the second brightness value range). In some embodiments, other display characteristics (e.g., color saturation, pixel resolution, color resolution, hue, etc.) are used as the basis for selecting the value range of display characteristics for the representation of that part of the user (e.g., the user's face or eyes). In some embodiments, the representation of the first user's eyes is generated by applying one or more preset image filters, such as blur filters, color filters, brightness filters, etc., which change the initial appearance of the first user's eyes when the representation is displayed by a second display generation component.

[0124] In some embodiments, a representation of the CGR content shown by a second display generation component (e.g., representation 7008) is generated by applying a diffusion filter to the CGR content displayed by a first display generation component (e.g., all visible content, media-only content, or visible content that optionally excludes a perspective view of the physical environment). For example, representation 7008 of the CGR content preserves the colors and hues of the scene, but the outlines of objects in the CGR content are blurred and not clearly defined in representation 7008. In some embodiments, the representation of the CGR content is semi-transparent, through which representation 7006 of the first user's portion is visible. In some embodiments, the second display generation component (e.g., in the same or a different display layer as representation 7008 of the CGR content, and / or in the same or a different display layer as representation 7006 of the first user's portion) displays graphical user interface elements representing metadata associated with the CGR content (e.g., a progress bar 7004, the name of the CGR content, etc.). In some implementations, compared to the representation 7008 of the CGR content and / or the representation 7006 of the portion belonging to the first user, the graphical user interface elements representing the metadata associated with the CGR content are displayed with higher pixel resolution, higher color resolution, higher color saturation, greater opacity, greater brightness, and / or better defined outlines.

[0125] In some embodiments, the portion of the first user's body (e.g., the first user's face or eyes) moves relative to the first display generating component (e.g., display 7100, the internal display of the HMD), while the CGR content 7002 presented by the first display generating component remains unchanged. In this case, optionally, the representation 7006 of the user's portion is updated on the second display generating component 7102, without updating the representation 7006 and the progress bar 7004 of the CGR content. In some embodiments, the CGR content is not displayed or paused, and the first user views a perspective view of the physical environment via the first display generating component without simultaneously displaying the CGR content, and the second display generating component optionally updates the representation of the first user's portion according to changes in the appearance of the first user's portion (e.g., due to movement or other changes in the user's portion), without displaying any representation of the CGR content, or showing a representation of the CGR content in a static or paused state.

[0126] In some implementations, the CGR content changes on the first display generating component, while the portion belonging to the first user does not change its appearance (e.g., it does not move or change for other reasons). Therefore, the representation 7006 of the portion belonging to the first user remains unchanged, and the second display generating component updates the representation 7008 of the CGR content and other indicators of the state of the CGR content (e.g., progress bar 7004) only based on the changes in the CGR content shown by the first display generating component.

[0127] In some implementations, when changes in both the CGR content and the appearance of the first user's portion are detected during the same time period (e.g., simultaneously and / or during each other's preset time windows, etc.), the second display generation component updates the visual representation of the user's portion and one or more graphical elements (e.g., the representation of the CGR content 7008 and the progress bar 7004) that indicate the state of the CGR content based on the detected changes.

[0128] Figures 7A to 7C The appearance of a visual representation (e.g., representation 7006) of a first user 7202 (e.g., the first user's eyes or face) is updated via a second display generation unit 7102 according to changes in the first user's appearance (e.g., due to eye movement or changes in facial expression, changes in lighting, etc.), and one or more graphic elements (e.g., progress bar 7004, representation 7008 of CGR content 7002, etc.) providing visual indications of the content within the CGR environment shown via the first display generation unit 7100 are updated via the second display generation unit 7102 according to changes in the CGR environment. Figures 7A to 7C In the exemplary scenario shown, the immersion level associated with the CGR content and the attention state of the first user 7202 remain unchanged and correspond to an intermediate immersion level associated with the presentation of the CGR content. In some embodiments, the immersion level associated with the presentation of the CGR content and the corresponding attention state of the first user may optionally change over a period of time, for example, increasing to a more immersive level and a more engaged user attention state, or decreasing to a less immersive level and a less engaged user attention state. In some embodiments, the state information displayed by the second display generation component is updated based on the change in the immersion level of the CGR content presented by the first display generation component. In some embodiments, updating the state information includes updating the representation of that portion of the first user's content (e.g., updating the visibility of that portion of the first user's content, updating the appearance of that portion of the first user's content, etc.).

[0129] In some embodiments, the computing system is configured to display CGR content 7002 at at least a first immersion level, a second immersion level, and a third immersion level. In some embodiments, the computing system causes the CGR content displayed via the first display generation component to transition between different immersion levels in response to a sequence of one or more events (e.g., the natural termination or progress of an application or experience; starting, stopping, and / or pausing an experience in response to user input; changing the immersion level of the experience in response to user input, changes in the state of the computing device, changes in the external environment, etc.). In some embodiments, the first immersion level, the second immersion level, and the third immersion level correspond to an increment in the virtual content present in the CGR environment and / or a decrement in the representation of the surrounding physical environment present in the CGR environment (e.g., a representation of location B or a portion of the physical environment in front of the display side of the second display generation component 7102). In some implementations, the first immersion level, the second immersion level, and the third immersion level correspond to different modes of content display with increasing image fidelity (e.g., increasing pixel resolution, increasing color resolution, increasing color saturation, increasing brightness, increasing opacity, increasing image detail, etc.) and / or spatial range (e.g., angular range, spatial depth, etc.) and / or representation of the surrounding physical environment (e.g., a representation of the portion of the physical environment in front of the display side of the second display generating component) of the computer-generated content, and decreasing image fidelity and / or spatial range. In some implementations, the first immersion level is a pass-through mode, wherein a first user can fully see the physical environment (e.g., a portion of the physical environment in front of the display side of the second display generating component) through the first display generating component (e.g., as a camera view of the physical environment or through a transparent portion of the first display generating component). In some implementations, the CGR content presented in pass-through mode includes a pass-through view of the physical environment, which has a minimum number of virtual elements that can be seen simultaneously as a view of the physical environment or only has virtual elements at the periphery of the user's view of the physical environment (e.g., indicators and controls displayed in the peripheral area of ​​the display). Figure 7EAn example of a first immersion level associated with CGR content 7002 according to some embodiments is shown. For example, a view of the physical environment (e.g., a portion of the physical environment in front of the display side of the second display generating component (e.g., another portion of the physical environment in front of the first user)) occupies the center and most of the field of view provided by the first display generating component, and some controls (e.g., movie name, progress bar, playback controls (e.g., play button)) are displayed only in the peripheral area of ​​the field of view provided by the first display generating component. In some embodiments, the second immersion level is a mixed reality mode, in which the transparent view of the physical environment is enhanced with virtual elements generated by a computing system, wherein the virtual elements occupy the center and / or most of the user's field of view (e.g., virtual content is integrated with the physical environment in a view of the computer-generated environment). According to some embodiments, in Figures 7A to 7C An example of a second level of immersion associated with CGR content 7002 is shown. In some implementations, the third level of immersion is a virtual reality mode, in which the user's view of the physical environment is completely replaced or obscured by the view of virtual content provided by the first display generating component. Figure 7D An example of a third immersion level associated with CGR content 7002 is shown according to some implementation schemes.

[0130] like Figure 7C After Figure 7D As shown, the computing system has switched from displaying CGR content 7002 in mixed reality mode to displaying CGR content 7002 in virtual reality mode (e.g., movie X shown as 7002-d) without a physical environment (e.g., including the location B of the second user 7204 (e.g., the portion of the physical environment in front of the display side of the second display generation unit 7102)). In some embodiments, the switching performed by the computing system is in response to a request from the first user (e.g., a gesture input that satisfies a preset criterion for changing the immersion level of the CGR content (e.g., lifting the first user's hand away from the HMD)). In conjunction with the switch from displaying CGR content 7002 in mixed reality mode via the first display generation unit 7100 to displaying CGR content 7002 in virtual reality mode, the computing system changes the state information displayed via the second display generation unit 7102. Figure 7D As shown, although one or more graphic elements indicating CGR content 7002 (e.g., name, progress bar 7004 (e.g., in...)) Figure 7D The symbols shown are 7004-d and 7008 (e.g., in...). Figure 7D The text is a jumbled mix of Chinese characters and symbols, making it impossible to translate coherently. It appears to be a collection of fragments from various sources, possibly related to CGR content, CGR, and possibly related to a display component. A proper translation would require the original, uncorrupted text. Figure 7DThe representation of the first user's portion (e.g., the eyes or face of the first user) is updated by changes shown in 7002-d, but the representation 7006 of that portion is no longer displayed by the second display generation component 7100. In some embodiments, the computing system does not completely stop displaying the representation of the first user's portion, but rather reduces the visibility of the representation of the first user's portion on the second display generation component relative to other state information (e.g., reduced brightness, reduced color resolution, reduced opacity, reduced pixel resolution, etc.). In some embodiments, the representation 7006 of the first user's portion is optionally displayed with reduced visibility relative to its previous appearance (e.g., completely invisible, or with reduced brightness, increased transparency, reduced opacity, reduced color saturation, increased blur level, etc.) to indicate an increase in the level of immersion associated with the CGR content shown by the first display generation component. Other status information (e.g., the representation of CGR content 7008 and the progress bar 7004) is continuously or periodically updated according to changes in the CGR content 7002 shown by the first display generating component, while such other status information is still displayed by the second display generating component 7102 (e.g., without reducing visibility relative to its previous appearance, unless the reduction is due to changes in the appearance of the CGR content).

[0131] In some embodiments, the switch from mixed reality mode to virtual reality mode is triggered by the second user 7204 moving away from the estimated field of view that the first user would normally have when their eyes are not obstructed by the presence of the first display generating component and / or the second display generating component. In some embodiments, the switch from mixed reality mode to virtual reality mode is triggered by the second user 7204 moving away from the physical environment surrounding the first user (e.g., leaving the room occupied by the first user). In some embodiments, when no other user is present in the physical environment, the computing system stops displaying a representation of the physical environment (e.g., a representation of location B (e.g., a representation of a portion of the physical environment in front of the first user)). In some embodiments, the second user 7204 moving into the estimated field of view that the first user would normally have when their view of view at location B is not obstructed by the presence of the first display generating component and / or the second display generating component, and / or entering the physical environment surrounding the first user (e.g., entering the room occupied by the first user), a predefined gesture performed by the second user (e.g., the second user waving at the first user), or the second user moving into a threshold distance range of the first user are optionally used as conditions for triggering the switch from virtual reality mode to mixed reality mode. In some implementations, in conjunction with switching the display mode from virtual reality mode to mixed reality mode, the computing system restores the visibility level of the representation 7006 of the first user's portion among the elements of the state information shown by the second display generating component 7102 (e.g., if the representation of the first user's portion is not visible, then restore the display of that representation, or increase the brightness, color saturation, pixel resolution, opacity, and / or color resolution of the user's portion of the representation, etc., and others). Accordingly, in mixed reality mode, in addition to computer-generated virtual content (e.g., Movie X), the first display generating component (e.g., display 7100, the internal display of the HMD, etc.) also displays a portion of the physical environment (e.g., representation 7010) in front of the display side of the second display generating component (and correspondingly, in front of the first user when the first and second display generating components are back-to-back within the same housing of the HMD worn by the first user).

[0132] like Figure 7E (For example, in) Figure 7C or Figure 7D As shown later, computing systems have evolved from mixed reality modes (e.g., such as...) Figures 7A to 7C (as shown) or virtual reality mode (e.g., as shown) Figure 7DThe CGR content is switched to a fully transparent mode or a reality mode (e.g., Movie X is finished and no longer shown) to display the CGR content (e.g., CGR content 7002), and only has a representation of the physical environment (e.g., including the position B of the second user 7204 (e.g., the portion of the physical environment in front of the display side of the second display generating component 7102)). Figure 7E The display is shown as 7010-e) and optionally has a minimal amount of virtual content (e.g., only indicators (e.g., the name of movie X, progress bar, playback controls, etc.) without playback content) on the peripheral areas of the display (e.g., the top and bottom edges of the display). In some embodiments, the computing system performs a switching of display modes in response to the termination or pause of playback of the CGR content and / or a request from the first user (e.g., a gesture input that meets preset criteria for changing the immersion level of the CGR content (e.g., placing a hand on the first user's eyebrow, pulling down with pinched fingers on that side of the HMD, etc.)). In some embodiments, in conjunction with switching from displaying the CGR content in mixed reality mode or virtual reality mode via the first display generation unit 7100 to displaying the CGR content in full pass-through mode or reality mode, the computing system changes the state information displayed via the second display generation unit 7102. Figure 7E As shown, one or more graphic elements (e.g., name, progress bar 7004) indicate the appearance and status of CGR content. Figure 7C It is shown as 7004-c, and in Figure 7D The text shows 7004-d (e.g.) and 7008 (e.g., in...). Figure 7C It is shown as 7008-c, and in Figure 7D The part shown as 7008-d, etc., is no longer displayed by the second display generation component, and the representation of that part by the first user is 7006 (e.g., in...). Figure 7E The representation 7006 (shown as 7006-e) is fully displayed by the second display generating component with increased visibility (e.g., becoming visible if previously invisible, or displayed with increased brightness, decreased transparency, increased opacity, increased color saturation, increased pixel resolution, increased color resolution, decreased blur level, etc.). In some embodiments, the representation 7006 of the first user's portion is optionally displayed with increased visibility relative to its previous appearance to indicate a reduction in the level of immersion associated with the CGR content shown by the first display generating component. In some embodiments, the representation 7006 of the first user's portion is continuously updated according to changes in the appearance of the first user 7202, while the representation 7006 of the first user's portion is displayed by the second display generating component.

[0133] In some embodiments, the switch from mixed reality or virtual reality mode to fully transparent mode or reality mode is triggered by the second user 7204 moving into the estimated field of view that the first user would normally have if their eyes were not obstructed by the presence of the first display generating component and / or the second display generating component. In some embodiments, the switch from mixed reality or virtual reality mode to fully transparent mode or reality mode is triggered by the second user 7204 moving into a personal space within a threshold distance from the first user 7202 (e.g., within arm's length of the first user, within three feet of the first user, etc.). In some embodiments, when preset conditions are met, such as when a preset person (e.g., spouse, teacher, teammate, child, etc.) enters the estimated field of view of the first user 7202, the computing system enters fully transparent mode or reality mode (e.g., stops displaying CGR content via the first display generating component and only displays a representation of the physical environment (e.g., position B, the physical environment in front of the first user, etc.)). In some implementations, the second user 7204 moves away from the estimated field of view of the first user 7202 and / or has personal space and / or other conditions within a threshold distance from the first user 7202 to trigger an automatic switch from fully transparent mode or reality mode back to mixed reality mode or virtual reality mode (e.g., preset mode or previous mode). In some implementations, in conjunction with the switch of the display mode from fully transparent mode to virtual reality mode or mixed reality mode, the computing system recovers the visibility level of the representation 7006 of the first user's portion of the state information shown by the second display generation component 7102 (e.g., stopping full display or reducing its visibility without making it completely invisible, etc.) and the visibility level of the representation 7008 of the CGR content (e.g., increasing its visibility) among the elements of the state information shown by the second display generation component 7102.

[0134] Figures 7C to 7E The illustration shows corresponding changes in information displayed by the first display generation unit 7100 and the second display generation unit 7102, according to some embodiments, from a second immersion level (e.g., mixed reality mode) down to a first immersion level (e.g., pass-through mode) and up to a third immersion level (e.g., virtual reality mode). In some embodiments, a direct transition between any two of these three immersion levels can be achieved in response to different events that satisfy the corresponding criteria for such a direct transition. Accordingly, the information displayed by the first display generation unit and the information displayed by the second display generation unit are updated (e.g., changing the visibility of different components of the information (e.g., CGR content, representation of the physical environment, representation of the CGR content, representation of the first user's portion, representation of static metadata associated with the CGR content, etc.)) to reflect the current immersion level at which the first display generation unit 7100 displays the CGR content.

[0135] In some implementation schemes, such as Figure 7E As shown, as a result of the computing system switching from displaying CGR content using Mixed Reality mode to displaying CGR content using Full Transmission mode or Reality mode, a representation 7006 of the first user's portion is displayed (e.g., a representation of the first user's face or eyes), but a representation of the CGR content is not displayed simultaneously (e.g., an overlay without a diffused version of the CGR content, a name, or a progress bar, etc.) or the visibility relative to the representation of the CGR content is increased (e.g., the visibility of representation 7006 is increased relative to its previous level, the visibility of the representation of the CGR content is decreased relative to its own previous level, and / or some graphic elements used to represent the CGR content are no longer displayed, etc.).

[0136] In some implementations, as a result of the computing system switching from displaying CGR content using virtual reality mode or pass-through mode to displaying CGR content using mixed reality mode, the representation 7006 of the first user's portion (e.g., a representation of the first user's face or eyes) is displayed together with the representation 7008 of the CGR content (e.g., an overlay with a diffused version of the CGR content) (e.g., to a comparable visibility relative to the representation 7008 of the CGR content (e.g., the visibility of representation 7006 and / or the visibility of representation 7008 relative to its respective previous level increases or decreases)).

[0137] In some implementations, when the computing system switches from displaying CGR content using mixed reality mode to displaying CGR content using virtual reality mode, the representation 7006 of that part of the first user (e.g., the representation of the first user's face or eyes) is not displayed together with the representation 7008 of the CGR content (e.g., not displayed together with a diffused version of the CGR content) or is displayed with reduced visibility relative to the representation 7008 of the CGR content.

[0138] In some implementations, the computing system can display CGR content using other special display modes such as privacy mode, Do Not Disturb (DND) mode, parental control mode, etc. When one or more of these special display modes are enabled, the status information is displayed and / or updated on the second display generation component in a manner that would be used when such special modes are not enabled (e.g., as described above relative to...). Figures 7A to 7E The described method) adjusts the way status information is displayed and / or updated on the second display generation component.

[0139] For example, a privacy mode may optionally be activated by the computing system or a first user to hide state information associated with the CGR content currently displayed by the first display generating component and / or state information associated with the first user's attention state. In some embodiments, when the privacy mode is activated (e.g., in response to a request from the first user), the representation 7006 of the first user's portion and / or the representation 7008 of the CGR content on the second display generating component are no longer updated, stop being displayed, and / or are replaced with other placeholder content such that they no longer reflect changes detected in the appearance of the first user's portion and / or changes detected in the CGR content displayed by the first display generating component. In some embodiments, the privacy mode is activated in response to a user request detected by the computing system (e.g., a preset gesture input, preset voice command, etc. by the first user on the HMD) (e.g., when the computing system displays CGR content to the first user using a mixed reality mode or virtual reality mode, and / or before starting CGR content, etc.). In some embodiments, the privacy mode is activated in response to a user accessing certain CGR content associated with a preset privacy level higher than a first threshold privacy level (e.g., the default privacy level, the privacy level associated with the first user, etc.). In some implementations, when privacy mode is enabled, the representation 7006 and / or the representation 7008 of the first user's portion no longer update, stop displaying, and / or are replaced with other placeholder content so that they no longer reflect changes in the immersion level of the CGR content displayed by the first display generating component. Privacy mode allows the first user to enjoy greater privacy and share less information about his / her own attention state, immersion level, and what he / she viewed using the first display generating component through the content displayed by the second display generating component.

[0140] In some implementations, the DND mode is actively activated by a first user and / or automatically activated by the computing system based on preset conditions to indicate to the external environment that the first user does not wish to be interrupted or disturbed by others in the external environment while he / she is engaging with CGR content. In some implementations, the DND mode may optionally be applied to other intrusive events occurring within the computing system and / or in the surrounding environment. For example, in some implementations, in response to the activation of the DND mode, the computing system may optionally activate noise cancellation to block sound from the surrounding environment, stop / pause the presentation of notifications and / or warnings on the first display generating component, reduce the intrusiveness of the way notifications and / or warnings are presented in the CGR environment shown by the first display generating component (e.g., selecting visual warnings instead of audio warnings, selecting short warning sounds instead of voice output, reducing the visual salience of notifications and warnings, etc.), automatically transmit calls to voicemail without notifying the first user, and / or display a Do Not Disturb symbol on the second display generating component, etc. In some embodiments, the computing system uses one or more methods to reduce the intrusiveness of events to the first user, accompanied by changes in how the representation of the physical environment is displayed on the first display generating component (e.g., a representation of 7010, a representation of location B, a representation of a portion of the physical environment in front of the first user, etc.), and / or changes in how status information is displayed by the second display generating component. In some embodiments, DND mode is optionally enabled when the computing system displays CGR content using a mixed reality mode or a virtual reality mode. In some embodiments, in response to enabling DND mode, the computing system optionally displays visual indicators (e.g., a text label “DND” on the external display of the HMD, a red rim illuminated around the external display of the HMD, etc.) via the second display generating component to indicate that DND mode is active. In some implementations, when the DND mode is active on the computing system, the representation of the CGR content is optionally updated according to changes in the CGR content displayed by the first display generation component, but the representation of that part for the first user is no longer updated and is replaced with placeholder content, or is stopped from being displayed by the second display generation component (e.g., regardless of changes in the appearance of that part for the first user (e.g., changes in the first user's eye level) and / or changes in the immersion level of the CGR content displayed by the first display generation component).

[0141] In some implementations, parental control is enabled to override the normal display of status information by the secondary display generator (e.g., as relative to...). Figures 7A to 7E(As described). Parental control mode is enabled, allowing parents, teachers, or supervisors to view and monitor CGR content presented to a first user and optional input provided by the first user for changing and / or interacting with the CGR content. In some embodiments, parental control mode is optionally enabled by a second user (e.g., via preset gesture input, touch input on the shell of the second display generation component or HMD, voice commands, etc.) while the CGR content is presented by the first display generation component. In some embodiments, parental control mode is optionally enabled before specific CGR content begins on the first display generation component (e.g., via interaction with the user interface presented by the first display generation component, interaction with the user interface presented by the second display generation component, interaction with the shell or other input devices of the computing system, etc.) and remains enabled while the specific CGR content is displayed on the first display generation component. In some embodiments, when parental control mode is enabled, the computing system simultaneously displays the same CGR content via both the first and second display generation components, regardless of changes in immersion level and / or whether privacy mode is enabled. In some embodiments, the computing system displays only a virtual portion of the CGR content shown by the first display generation component on the second display generation component. In some implementations, when parental control mode is enabled, the computing system does not display the representation 7006 of the first user's portion as part of the status information displayed using the second display generation component (e.g., if parental control mode is only used to monitor content shown to the first user, and not to monitor the first user themselves). In some implementations, when parental control mode is enabled, the computing system displays the user's representation 7006 and the CGR content with visibility comparable to the CGR content (e.g., enhanced visibility of representation 7006 compared to its previous level of visibility when parental control mode is not enabled) (e.g., if parental control mode is used to monitor content shown to the first user and the first user's attention state and appearance). In some implementations, the second display generation component determines whether to display the user's representation during parental control mode based on how parental control mode is activated (e.g., using a first input type or a second input type, using a first control or a second control, etc.). In some implementations, the second display generation component determines whether to display the user's representation during parental control mode based on whether privacy mode is enabled. For example, if privacy mode is enabled, the second display generation component does not display the user's representation of that part along with the CGR content; and if privacy mode is not enabled, the second display generation component displays the user's representation of that part along with the CGR content.In some implementations, when parental control mode is enabled, changes in the immersion level of the CGR content displayed by the first display generating component do not alter the information displayed by the second display generating component (e.g., the same CGR content is still displayed on both the first and second display generating components, optionally wherein the current visibility level of the representation of that portion by the first user remains unchanged, or the representation of the first user is not displayed).

[0142] In some implementations, the computing system dynamically adjusts the visibility and information density of the status information displayed by the second display generating component based on the distance to a second user located at a position that allows the second user to view the content displayed by the second display generating component (e.g., directly or partially in front of the display side of the second display generating component). For example, when the second user moves closer to the display side of the second display generating component (e.g., within a threshold distance, into a threshold viewing angle, etc.) (e.g., when the first and second display generating components are placed back-to-back in the same housing of the HMD worn by the first user, moving closer to the first user and the first display generating component), the computing system changes (e.g., increases) the amount of information detail provided on the second display generating component (e.g., details of graphic features, number of text characters per unit display area, color resolution, pixel resolution, etc.) to inform the second user of the status of the first user and the status and metadata of the CGR content. Accordingly, when the second user moves further away from the display side of the second display generating component (e.g., moves beyond a threshold distance, moves beyond a threshold viewing angle, etc.), the computing system changes the amount of information detail provided on the second display generating component in the opposite direction (e.g., reduces the amount of information detail).

[0143] In some implementations, the computing system automatically transitions from displaying the computer-generated experience in a fully immersive mode (e.g., displaying a virtual reality environment, or displaying CGR content at a third level of immersion) to displaying the computer-generated experience in a less immersive mode in response to detecting changes in the surrounding physical environment that meet preset criteria (e.g., a person entering a room or reaching a threshold distance from a first user, other users waving or gesturing towards the first user, etc.). This transition is done in a less immersive mode (e.g., displaying indications of the physical environment in the virtual reality environment (e.g., displaying outlines of people and objects in the physical environment as visual distortions, shadows, etc.), or displaying transparent portions (e.g., a camera view of the physical environment) in a view of the computer-generated environment). In some implementations, in conjunction with automatically changing the immersion level of the computer-generated environment displayed via a first display generation component, the computing system also changes the state information displayed via a second display generation component, including increasing the visibility of the visual representation of the user's portion (e.g., increasing the visibility of the user's visual representation includes switching from not displaying the visual representation of the user's portion to displaying the visual representation of the user's portion, or increasing the brightness, sharpness, opacity, and / or resolution of the visual representation of the user's portion). This simultaneously reduces visual barriers that separate the first user from others in the surrounding environment (e.g., the presence of the display-generated component on the first user's face) to facilitate more informed interaction between the first user and surrounding users. In some embodiments, if the computing system reduces the immersion level of the content displayed on the first display-generated component in response to a second user's action (e.g., in response to the second user waving at the first user, and / or in response to the second user moving too close to the first user, etc.), the computing system stops displaying a representation of the CGR content on the second display-generated component or does not display a representation of the CGR content, and only displays a representation of that part of the first user (e.g., the first user's face or eyes) (e.g., to let the second user know that the first user can see him / her through the first display-generated component). In some implementations, when the computing system increases the immersion level of the content displayed on the first display generating component in response to the actions of the second user (e.g., in response to the second user wearing the HMD, and / or in response to the second user leaving the first user, etc.), the computing system re-displays the representation of the CGR content on the second display generating component and stops displaying the representation of that part of the first user (e.g., the first user's face or eyes) (or reduces its brightness, sharpness, opacity, color, and pixel resolution, etc.).

[0144] Compared to Figures 7F to 7Q and Figures 8 to 13 The accompanying notes provide further details regarding the user interface and operating modes of the computing system.

[0145] Figures 7F to 7IA computing system is shown that includes at least a first display generation component (e.g., display 7100, internal display of the HMD, etc.) and a second display generation component (e.g., display 7102, external display of the HMD, etc.). Figure 1 The computing system 101 or Figure 4 The computing system 140 in the computing system is configured to display visual content (e.g., user interface, computer-generated experience, media content, etc.) to a user (e.g., first user 7202) when the computing system determines that the first display generating component is positioned relative to a user (e.g., first user 7202) or a portion of the user (e.g., the first user's face or eyes) in a predetermined spatial relationship (e.g., a predetermined orientation (e.g., to display the side of the first user's face or eyes)). In some embodiments, before displaying visual content to the user via the first display generating component (e.g., display 7100, internal display of the HMD, etc.), the computing system uses a second display generating component (e.g., display 7102, external display of the HMD, etc.) to display one or more user interface elements that prompt the user for visual content (e.g., user interface, computer-generated experience (e.g., AR content, VR content, etc.), media content, etc.) that can be used to display via the first display generating component, if conditions for triggering the display of content are met (e.g., conditions regarding the spatial relationship between the first display generating component and the user). In some implementations, the one or more user interface elements include user interface objects that convey contextual information (e.g., current time, location, external conditions, user identity and / or the current state of the computing system, or other prompts, notifications, etc.) so that available content can be displayed via the first display generation component. Figure 7F To Figure FG and Figures 7H to 7I Two parallel examples are shown to illustrate how different computer-generated experiences can be displayed by the first display generation component based on different states of the computing system reflected by different user interface objects shown by the second display generation component, prior to satisfying the preset spatial relationship between the first display generation component and the user.

[0146] like Figure 7F and Figure 7H As shown in the corresponding left portion, the first display generating component (e.g., display 7100) is located at position A 7000-a, and the user is not facing the display side of the first display generating component. Therefore, the first display generating component does not display any CGR content at this time. Figure 7F and Figure 7HAs shown in the corresponding right portion, a second display generating component (e.g., display 7102) is present at location B 7000-b and displays one or more user interface elements, each including elements related to the user interface elements as considered in the context of the display. Figure 7F Given the current context state of the computing system shown, a first user interface element (e.g., circle 7012) corresponding to the first computer-generated experience 7024 displayed via the first display generation component can be used, and in consideration of... Figure 7H The current context state of the computing system can be used to create a second user interface element (e.g., square 7026) corresponding to the second computer-generated experience 7030 displayed via the first display generation component. The context state of the computing system can be determined, for example, based on the current time, the current position of the first display generation component (in some embodiments, this is also the position of the first user and the second display generation component), the user identity or authorization level of the first user 7202 present in front of the display side of the second display generation component, notifications received or generated by the computing system for corresponding applications, preset events occurring on the computing system, and / or other contextual information.

[0147] exist Figure 7F and Figure 7HIn the illustrated exemplary scenario, a first user 7202 is shown located at position B 7000-b. In some embodiments, the spatial relationship between the second display generating component (e.g., display 7102, an external display of the HMD, etc.) and the first user 7202 places the first user 7202 in a position to view one or more user interface elements (e.g., user interface element 7012 and user interface element 7026, respectively) presented by the second display generating component. For example, when one or more user interface elements are displayed, the first user 7202 faces the display side of the second display generating component. In some embodiments, the second display generating component is an external display of the HMD, which also includes an internal display (e.g., the first display generating component represented by display 7100) configured to present CGR content corresponding to the user interface elements shown on the external display of the HMD. In this implementation, the spatial relationship represented by the coexistence of the display 7102 and the first user 7202 at the same location B 7000-b corresponds to the first user being in a portion of the physical environment facing the external display of the HMD (e.g., also hosting the physical environment of the second display generating component and the first user 7202). In some implementations, the second display generating component is positioned back-to-back with the first display generating component such that a portion of the physical environment in front of the display side of the second display generating component 7102 (e.g., including location B 7000-b of the display side of the second display generating component) is also within the perspective view provided by the first display generating component. For example, if the first user 7102 moves to the display side of the first display generating component (e.g., moves to location A 7000-a, and / or faces the display side of the first display generating component, etc.), then the physical object 7014 (or...) located in the portion of the physical environment in front of the display side of the second display generating component 7102... Figure 7HBoth physical objects 7014 and 7028 will be within the transparent view provided by the first display generating component. In some embodiments, the computing system displays one or more user interface elements only in response to detecting that a first user is in a position to view the content displayed by the second display generating component, and stops using the second display generating component to display one or more user interface elements when no user is in a position relative to the second display generating component (a position that allows the user to view the content presented by the second display generating component). In some embodiments, the computing system displays one or more user interface elements in response to detecting an event indicating the availability of the first computer-generated experience based on the current state or context of the computing system (e.g., generating a time-based alert or location-based notification on the computing system, the user taking the HMD out of a bag, the user turning on the HMD, the user plugging the HMD into a charging station, etc.). In some embodiments, when the first display generating component does not display any CGR content (e.g., the first display generating component is inactive, in a power-saving state, etc.), the computing system exposes one or more user interface objects only on the second display generating component. In some implementations, it is shown that the first user 7202 is not coexisting with the first display generating component at location A or is not in a position where the user is viewing the content displayed by the first display generating component. Figure 7F and Figure 7H The illustration corresponds to a first user not placing the HMD's internal display in front of the user's face or eyes (e.g., by holding the HMD with the internal display facing his / her face, or by wearing the HMD on his / her head). In some embodiments, one or more user interface objects are displayed on a second display generating component, regardless of whether the first user is in a position to view the content displayed on the second display generating component.

[0148] like Figure 7F After Figure 7G and Figure 7H After Figure 7I As shown, when the computing system uses the second display generation component to display one or more user interface objects (e.g., a circle 7012 or a square 7026, respectively), the computing system detects that the first display generation component (e.g., a display 7100 or the internal display of an HMD) is currently in a preset spatial relationship relative to the first user 7202 (e.g., due to movement of the first user 7202, movement of the first display generation component, or both). Figure 7F and Figure 7GIn the example shown, the first user 7202 has moved to position A7000-a in front of the display side of the first display generating component (e.g., display 7100, internal display of the HMD, etc.); and in response to detecting that the first user has moved to position A7000-a in front of the display side of the first display generating component, the computing system displays, via the first display generating component, a corresponding computer-generated experience (e.g., a first computer-generated experience 7024 or a second computer-generated experience 7030) corresponding to one or more user interface objects (e.g., a circle 7012 or a square 7026) previously shown on the second display generating component. For example, as Figure 7G As shown, the computing system responds to an instruction in the second display generation component ( Figure 7F When displaying the first user interface object (e.g., circle 7012), the first display generating component and the first user (or the first user's face or eyes) are placed in a preset spatial relationship or configuration (e.g., the first user faces the display side of the first display generating component, or the HMD is placed in front of the user's eyes, or the HMD is placed on the user's head, etc.). An event of relative movement between the first display generating component and the first user triggers the display of the first computer-generated experience 7024. Figure 7I As shown, the computing system responds to an instruction in the second display generation component ( Figure 7H When a second user interface object (e.g., square 7026) is displayed, the first display generating component and the first user (or the first user's face or eyes) are placed in a preset spatial relationship or configuration (e.g., the first user faces the display side of the first display generating component, or the HMD is placed in front of the user's eyes, or the HMD is placed on the user's head, etc.) and the relative movement of the first display generating component and the first user is displayed, and a second computer-generated experience 7030 is displayed.

[0149] like Figure 7F After Figure 7G As shown, an example of the first computer-generated experience 7024 is an augmented reality experience shown as a portion of the physical environment in front of the display side of the second display generating component (e.g., a portion of the physical environment in front of the external display of the HMD, which is also a portion of the physical environment in front of the first user wearing the HMD and / or the internal display facing the HMD). The first computer-generated experience optionally includes a representation 7014' of the physical object 7014 enhanced with some virtual content (e.g., a virtual sphere 7020 located on top of the representation 7014' of the physical object 7014 and / or some other virtual objects). In some embodiments, the first computer-generated experience is a purely virtual experience and does not include a representation of the physical environment surrounding the first display generating component and / or the second display generating component.

[0150] like Figure 7H After Figure 7I As shown, an example of the second computer-generated experience 7030 is an augmented reality experience shown as a portion of the physical environment in front of the display side of the second display generating component (e.g., a portion of the physical environment in front of the external display of the HMD, which is also a portion of the physical environment in front of the first user wearing the HMD and / or the internal display facing the HMD). The second computer-generated experience optionally includes stacked physical objects 7014 and 7028 enhanced with some virtual content (e.g., a virtual box 7032 located beside representations 7014' and 7028' of physical objects 7014 and 7028, or some other virtual objects). In some embodiments, the second computer-generated experience is a purely virtual experience and does not include a representation of the physical environment surrounding the first display generating component and / or the second display generating component.

[0151] In some embodiments, as previously described in this disclosure, the first display generating component is an internal display of the HMD and the second display generating component is an external display of the HMD, and the spatial relationship represented by the coexistence of display 7100 and the first user 7202 in the same location A 7000-a corresponds to the first user wearing or holding the HMD with the internal display of the HMD facing the user's eyes or face. In some embodiments, the first display generating component displays a camera view of the physical environment in front of the first user or includes a transparent or semi-transparent portion through which the first user can see a portion of the physical environment in front of the first user. In some embodiments, the physical environment visible to the first user through the first display generating component is a portion of the physical environment in front of the display side of the second display generating component (e.g., including the area in front of the display side of the second display generating component and physical object 7014 (and optionally physical object 7028), the area in front of the external display of the HMD, etc., in locations B 7000-b). In some embodiments, the computing system requires a first display generating component to move to a predefined orientation relative to a first user or a corresponding part of the first user (e.g., the HMD's internal display is oriented towards the user's eyes or face, the first user moves to face the display side of the first display generating component, and / or the HMD's internal display is upright relative to the user's face, etc.) to trigger the display of a computer-generated experience via the first display generating component. In some embodiments, the corresponding computer-generated experience is selected based on the current state of the computing system (e.g., one or more states determined based on contextual information (e.g., time, location, what physical objects are in front of the user, the user's identity, new notifications or alerts generated on the computing system, etc.)) and / or when a movement to a predetermined spatial relationship between the user and the first display generating component begins and / or is completed, depending on which user interface element(s) the second display generating component displays (e.g., one or more of user interface elements conveying the identifier and characteristics of the selected computer-generated experience and / or user interface elements conveying contextual information for the selected computer-generated experience). Figure 7F and Figure 7G In the example shown, the current state of the computing system is determined based on the current position of the display generating component and / or the physical object present in front of the display side of the second display generating component (and optionally also based on the current position and the physical object present in front of the external display of the HMD that houses both the first and second display generating components).

[0152] As mentioned above, in contrast to Figures 7A to 7E As stated and will be reiterated here, although the first display generating component (e.g., display 7100) and the second display generating component (e.g., display 7102) are in Figures 7F to 7IThe first and second display generating components are shown as being located in two separate and unconnected parts of a physical environment. However, it should be understood that the first and second display generating components are optionally housed in the same housing (e.g., the housing of a single HMD) or attached to the same support structure (e.g., attached back-to-back or on opposite sides of a single wall or surface) and facing different directions (e.g., facing opposite directions, facing different angles, etc.). According to some embodiments, a user can move (e.g., rotate, turn, or flip vertically or horizontally) the housings or support structures of the first and second display generating components to move the first display generating component into a preset spatial configuration relative to himself or his / her face or eyes. According to some embodiments, a user can insert his / her head into the housing of the display generating component or attach the support structure of the display generating component to a part of his / her body (e.g., head, shoulder, nose, ear, etc.) to move the first display generating component into a preset spatial configuration relative to himself or his / her face or eyes. Therefore, the coexistence of the first user and the second display generation component at position B 7000-b and the coexistence of the first user and the first display generation component at position A 7000-a respectively represent the first time when the preset spatial relationship between the user and the first display generation component for triggering the display of the computer-generated experience is satisfied and the second display generation component is displaying user interface elements showing the available CGR experience, and the second time when the preset spatial relationship is satisfied due to the movement of the user and / or the display generation component and the available CGR experience is displayed via the first display generation component.

[0153] In some implementations, the second display generating component is a low-resolution, smaller, simpler, single-stereo, monochrome, low-power, and / or auxiliary display of the computing system, while the first display generating component is a higher-resolution, larger, more complex, stereoscopic, full-color, full-power, and / or main display. In some implementations, the computing system uses the second display generating component to display status information, event information, and condition information related to the computing system, particularly notifications and prompts that allow the user to position the first display generating component in a preset spatial relationship relative to the user's eyes to view additional available content relevant to the current context. In some implementations, the computing system uses the second display generating component when it is not placed in front of the user's eyes (or more generally, not in a position where the user can fully appreciate the CGR content displayed on the first display generating component), and / or when the computing system's display generating component (e.g., as a component of a single HMD) is placed on a table, in the user's hands, in a container (e.g., a backpack, stand, case, etc.), or in a standby state (e.g., plugged into a charging station, set to low-power mode, etc.). In some embodiments, while information is displayed using the second display generating component, the computing system continues to monitor the spatial relationship between the user (e.g., a first user or any user) and the first display generating component (e.g., using sensors attached to or surrounding the housing of the first display generating component (e.g., motion sensors, orientation sensors, image sensors, touch sensors, etc.) and / or external sensors (e.g., motion sensors, orientation sensors, image sensors, etc.)). In some embodiments, in response to detecting relative movement between the first display generating component and the user (e.g., in response to the user picking up a display generating component surrounded by the same housing or attached to the same support structure and turning the display side of the first display generating component toward the user's eyes or face, and / or in response to the user wearing an HMD including the first and second display generating components on his / her head, etc.), based on determining that a preset spatial relationship between the user and the first display generating component is satisfied, the computing system displays a computer-generated experience corresponding to the state of the computing system when the preset spatial relationship is satisfied (e.g., optionally, this is the same state the computing system had when the second display generating component displayed information (e.g., one or more user interface objects indicating the availability of the computer-generated experience) at the start of the relative movement).

[0154] In some implementation schemes, such as Figure 7G and Figure 7I As shown, when the computing system displays the current context (e.g., as shown) via the first display generation component, Figure 7F and Figure 7HWhen the computing system provides a corresponding computer-generated experience corresponding to the state of the computing system indicated by user interface objects 7012 and 7026, the computing system optionally displays state information (e.g., state information 7022 and 7034, respectively) using a second display generation component. In some embodiments, the displayed state information conveys information related to the computer-generated content displayed via the first display generation component and, optionally, the state of the user viewing the computer-generated content via the first display generation component (e.g., the appearance of the user's face or eyes). Relative to Figures 7A to 7E And the accompanying description of other aspects and details relating to the use of a second display generation component to display status information when a first display generation component is used to display computer-generated content in a computing system, and relative to Figures 8 to 13 The process is described below. In some embodiments, once the first display generating component begins displaying content and / or once a preset spatial relationship between the user and the first display generating component is satisfied, the second display generating component stops displaying any content (e.g., user interface elements 7012 or 7026, etc.). In some embodiments, the computing system does not display status information or any other content while using the first display generating component to display the computer-generated experience. In some embodiments, the computing system uses the second display generating component to display other information (e.g., a digital clock, weather forecast, a countdown timer based on the duration of the computer-generated experience, or the allocated time for the first user to use the first display generating component, etc.) while using the first display generating component to display the computer-generated experience.

[0155] In some embodiments, the corresponding computer-generated experience displayed via the first display generating component is a mixed reality experience, wherein virtual content and a representation of the physical environment (e.g., location B, a portion of the physical environment in front of the first user, etc.) are simultaneously visible. In some embodiments, the representation of the physical environment includes a camera view of a portion of the physical environment that is within the first user's field of view when the user's eyes are not obstructed by the presence of the first and second display generating components (e.g., when the first user is not wearing an HMD or holds the HMD in front of his / her eyes). In mixed reality mode, CGR content (e.g., movies, 3D augmented reality environments, user interfaces, and / or virtual objects, etc.) is displayed as at least a portion, but not all, of the representation of the physical environment, overriding or replacing it. In some embodiments, the first display generating component includes a transparent portion through which the first user can see a portion of the physical environment, and in mixed reality mode, the CGR content (e.g., movies, 3D augmented reality environments, user interfaces, virtual objects, etc.) is projected onto physical surfaces or empty spaces in the physical environment and is visible along with the physical environment through the transparent portion. In some embodiments, CGR content is displayed on a portion of the display and obscures at least a portion, but not all, of the view of the physical environment visible through a transparent or semi-transparent portion of the first display generating component. In some embodiments, the first display generating component 7100 does not provide a view of the physical environment, but instead provides a fully virtual environment (e.g., without a camera view or transparent pass-through portion) augmented with a real-time visual representation of the physical environment currently captured by one or more sensors (e.g., a stylized representation or segmented camera images). In some embodiments, in mixed reality mode (e.g., augmented reality based on a camera view or transparent display, or augmented virtuality based on a virtualized representation of the physical environment), the first user is not fully immersed in the computer-generated environment and is still provided with sensory information (e.g., visual, audio, etc.) directly corresponding to the physical environment surrounding the first user and the first display generating component. In some embodiments, while the first display generating component displays a fully immersive environment, the second display generating component optionally displays state information without information relating to the state of the user's eyes (e.g., displaying only state information relating to the CGR content) or displays no state information at all.

[0156] In some implementations, the computing system optionally has any number of different states, corresponding to the availability of different computer-generated experiences displayed via a first display generation component. Each different state of the computing system optionally has one or more user interface elements of a corresponding group displayed by the second display generation component when the computing system enters and / or remains in that state. Each different state of the computing system optionally is triggered by the satisfaction of a corresponding event or event group and / or one or more preset criteria of the corresponding group. Although in relation to Figures 7F to 7I The described example illustrates only two states of the computing system, two user interface objects corresponding to these two states, and two different computer-generated experiences. However, a third state, a third user interface element, and a third computer-generated experience may optionally be implemented by the computing system in a manner similar to that described relative to the two states, user interface elements, and computer-generated experiences presented in the example. In some implementations, any finite number of states, user interface elements, and computer-generated experiences may optionally be implemented.

[0157] In some implementations, the computer-generated experience provided by the first display generation component is an immersive experience (e.g., an AR or VR experience) and takes into account the actions of a first user in the physical environment (e.g., gestures, movement, voice, and / or gaze). For example, when the user's hand moves in the physical environment or when the user moves in the physical environment (e.g., turning or walking), the user interface and / or user's perspective of the computer-generated 3D environment are updated to reflect the movement of the user's hand (e.g., pushing open a virtual window in the AR environment, activating a user interface element in the main screen or menu presented in the AR environment, etc.) or the user's movement (e.g., the user's viewpoint moves relative to the AR environment or a virtual 3D game world, etc.).

[0158] In some implementations, the different computer-generated experiences (e.g., a first computer-generated experience, a second computer-generated experience, etc.) are AR experiences that include representations of different virtual elements (e.g., indications of one or more user interface elements (e.g., circle 7012, square 7026, etc.) selected based on the state of the computing system, within the same physical environment. For example, in some implementations, the computer-generated experience optionally includes a view of the same room in which the first user is located. Based on a determination that the computing system displays a corresponding event alert from several different event alerts on the second display generator just before the user places the display side of the first display generator in front of his / her eyes, the computing system displays a representation of the room and a corresponding themed virtual wallpaper from several different themed virtual wallpapers on the representation of the room's walls, while simultaneously displaying a corresponding introductory video for the event corresponding to the event alert.

[0159] In some implementations, the different computer-generated experiences are either augmented reality (AR) experiences or virtual experiences, depending on the context (e.g., the state of the computing system determined based on relevant contextual information such as location, time, user identity, received notifications or alerts, etc.) and / or what is displayed on the second display generation component). In some implementations, after a computer-generated experience begins in one of the AR and VR modes, the experience can (e.g., in response to a user request, in response to the fulfillment of other preset conditions, etc.) transition to another mode in the AR and VR modes.

[0160] In some embodiments, the computing system is configured to use a second display generation component to display corresponding user interfaces and / or user interface objects for various applications based on the state of the computing system. For example, in some embodiments, one or more user interface elements displayed on the second display generation component include elements of an electronic calendar (e.g., social calendar, work calendar, daily schedule, weekly schedule, monthly calendar, standard calendar showing dates and weeks by month, etc.) with scheduled events, appointments, holidays, and / or reminders. In some embodiments, when a preset spatial configuration between the first display generation component and the first user is satisfied (e.g., the first user or his / her eyes are facing the display side of the first display generation component, the first user is in a position that allows him / her to view the content displayed by the first display generation component, etc.), the computing system displays different computer-generated experiences via the first display generation component, wherein the specific computer-generated experience displayed is based on what calendar content is displayed on the second display generation component just before the start and / or completion of the movement that places the first display generation component and the first user in the preset spatial configuration. For example, based on determining that one or more user interface elements displayed on the second display generation component correspond to a first calendar event (e.g., the user interface elements display event information, alerts, notifications, calendar data, notes, etc. for the first calendar event), the computing system displays a first computer-generated experience corresponding to the first calendar event (e.g., detailed and / or interactive information (e.g., previews of the venue and attendees, videos, models, etc.)). Based on determining that one or more user interface elements displayed on the second display generation component correspond to a second calendar event (e.g., the user interface elements display event information, alerts, notifications, calendar data, notes, etc. for the second calendar event), the computing system displays a second computer-generated experience corresponding to the second calendar event (e.g., detailed and / or interactive information (e.g., previews of the venue and attendees, videos, models, etc.)). In some embodiments, when the dual-sided HMD is not worn by the user (e.g., placed on a table with the external display facing the user), the external display of the HMD is used to display a calendar, including the current date, time, weather information, geographical location, and / or a list of upcoming tasks or scheduled appointments for the day or within a preset time period (e.g., within the next two hours, within the next five minutes, etc.). When a user picks up the HMD and places its internal display in front of his / her eyes (e.g., by holding the HMD up or by placing the HMD on his / her head, etc.), the HMD's internal display shows calendar details (e.g., showing a more complete calendar including the current week or month, showing all scheduled events for the day, showing more details of upcoming events, etc.).In some implementations, one or more user interface elements corresponding to a first calendar event include a notification of the first calendar event, and the user interface element corresponding to a second calendar event is a notification of the second calendar event.

[0161] In some implementations, the computing system uses a second display generation component to display media objects, such as photographs and / or video clips with two-dimensional images, and uses a first display generation component to display a three-dimensional experience or full-length media content corresponding to the media objects displayed on the second display generation component. For example, user interface elements shown on the second display generation component may optionally include snapshots or clips from a long video, a reduced-resolution or two-dimensional version of a three-dimensional video, a non-interactive user interface corresponding to an interactive computing environment, etc.; and the first display generation component displays long videos, three-dimensional videos, interactive computing environments, etc., when criteria for triggering the display of such extended content are met (e.g., when the first display generation component and the first user are placed in a preset spatial configuration and optional other conditions (e.g., the user is seated, the HMD has sufficient power, etc.)). In some implementations, when the dual-sided HMD is not worn by the user (e.g., placed on a table with the external display facing the user), the external display of the HMD is used to display a visual representation of available media items that can be displayed via the internal display of the HMD. In some implementations, the available media items are changed based on the current location of the HMD and / or the availability of media items specified by the media item provider. When a user picks up the HMD and places the internal display in front of his / her eyes, the first display generation component shows the actual content of the media item (e.g., showing a more complete movie, a more immersive experience, and / or enabling more interactive capabilities of the media item, etc.).

[0162] In some embodiments, the computing system uses a second display generation component to display an alert for an incoming communication request (e.g., an incoming telephone call, audio / video chat request, video conferencing request, etc.), and once the first display generation component is placed in a preset physical configuration relative to a first user (e.g., by movement of the first user, the first display generation component, or both), the computing system uses the first display generation component to display the corresponding communication environment. In some embodiments, the communication environment shown via the first display generation component illustrates a simulated environment in which virtual avatars or images of each participant are present (e.g., these avatars are represented sitting around a table in front of the first user, or are positioned as talking heads on the desktop in front of the first user, etc.). In some embodiments, the computing system accepts the incoming communication request and begins the corresponding communication session (e.g., using the first display generation component and other components of the computing system) in response to detecting that the first display generation component is placed in a preset physical configuration relative to the first user. In some embodiments, the computing system launches the application corresponding to the incoming communication request and (e.g., using the first display generation component) displays the application's user interface in response to detecting that the first display generation component is placed in a preset physical configuration relative to the first user, from which the first user can choose to accept the incoming communication request. In some implementations, when the dual-sided HMD is not worn by the user (e.g., placed on a table with the external display facing the user), the external display of the HMD is used to display a notification of an incoming communication request when the computing system receives such a request. In some implementations, this notification provides an indication of the caller's identity and the type of communication session requested. When the user picks up the HMD and places its internal display in front of his / her eyes (e.g., by holding the HMD up with his / her hand, or by wearing the HMD over his / her head, etc.), the internal display of the HMD displays a communication interface corresponding to the received communication request, and the user can begin using the internal display of the HMD to communicate with the caller. In some implementations, the computing system launches different applications (or different modes of launching the same application) based on the characteristics of the incoming communication request (e.g., the identity of the caller, time of day, subject of the call, etc.). For example, for an incoming request from a colleague, the computing system displays a user interface on a first display generating component awaiting preset input from the first user before initiating the communication session; and for an incoming request from a family member, the computing system initiates the communication session without displaying a user interface and / or without requiring preset input from the first user. In another example, for an incoming request arriving at a user's home, the computing system initiates a communication session with the avatar of the first user dressed casually; and for an incoming request arriving at a user's office, the computing system initiates a communication session with the avatar of the first user dressed professionally.In another example, for an incoming request for a telephone call, the computing system displays a close-up representation of each participant's speaking head; and for an incoming request for a video chat, the computing system displays a full-body representation of each participant, showing their actual physical movements. In some embodiments, one or more user interface elements shown on the second display generation component visually indicate specific characteristics of the incoming communication request, which the computing system uses to determine the characteristics of the computer-generated experience (e.g., characteristics of the user interface or environment of the communication session). In some embodiments, selected characteristics of the computer-generated experience are also visually represented by one or more user interface elements shown on the second display generation component before the computer-generated experience is displayed by the first display generation component. In some embodiments, the computing system modifies the characteristics of the computer-generated experience based on user input received before the computer-generated experience is displayed using the first display generation component (e.g., touch gestures on the second display generation component, touch gestures on the housing of the first and / or second display generation component, air gestures, voice commands, etc.).

[0163] In some implementations, the computing system modifies the content displayed on the second display generating component (e.g., one or more user interface elements) based on various parameters (e.g., user distance, user identity, user gestures, etc.). For example, based on detecting a first user at a first distance from the second display generating component (e.g., the first distance is less than a first threshold distance but greater than a second threshold distance), the computing system displays a first version of one or more user interface elements (e.g., larger, simpler icons or text) to indicate the availability of the computer-generated corresponding experience; and based on detecting the first user at a second distance from the second display generating component (e.g., the second distance is less than a second threshold distance), the computing system displays a second version of one or more user interface elements (e.g., graphics, more details, etc.) to indicate the availability of the computer-generated corresponding experience (e.g., replacing the display of the first version of one or more user interface elements when the first user moves closer to the second display generating component). In another example, based on detecting a user within a threshold distance of the second display-generating component, the computing system displays a generic version of one or more user interface elements (e.g., a larger, simpler icon or text) to indicate the availability of the corresponding computer-generated experience; and based on detecting the user's identity (e.g., in response to detecting the user's fingerprint when picking up the first / second display-generating component (e.g., HMD), or in response to the user moving closer to the second display-generating component, etc.), the computing system displays a user-specific version of one or more user interface elements corresponding to the user's identity (e.g., customized based on user preferences, usage history, demographics, etc.).

[0164] In some embodiments, the computing system displays a user interface including selectable options (e.g., one or more user interface elements and / or user interface objects other than one or more user interface elements) before detecting that a first display generation component is placed in a preset physical configuration relative to a first user, and detects user input selecting one or more of these selectable options, wherein these selectable options include preferences for customizing a computer-generated experience corresponding to one or more user interface elements displayed via the first display generation component. When the first display generation component is placed in the preset physical configuration relative to the first user, the computing system displays a computer-generated experience customized based on the user-selected options. In some embodiments, these selectable options correspond to a set of two or more modes of the computing system (e.g., AR mode, VR mode, 2D mode, privacy mode, parental control mode, DND mode, etc.), wherein a computer-generated experience can be presented via the first display generation component.

[0165] In some embodiments, one or more user interface elements displayed by the second display generation component include previews usable for a 3D experience displayed by the first display generation component. In some embodiments, the preview provided by the second display generation component is a 3D preview simulating a viewport in a 3D experience. The user can move his head relative to the second display generation component and see different portions of the 3D environment represented in the 3D experience. In some embodiments, the preview is initiated when the user picks up the second display generation component (e.g., picks up a dual-sided HMD) and / or places the second display generation component in a preset spatial configuration relative to the first user (e.g., holds the HMD with an external display facing the user's eyes). In some embodiments, after the preview is initiated on the first display generation component, in response to detecting that the user has placed the first display generation component in a preset spatial relationship with the first user (e.g., the user holds the HMD with an internal display facing the user's face or eyes, the user places the HMD on the user's head, etc.), the computing system initiates a computer-generated experience on the first display generation component.

[0166] Figures 7H to 7JThe illustrations illustrate displaying different computer-generated experiences according to some embodiments, based on a manner in which a first display generating component remains in a preset spatial relationship or configuration relative to a first user during the presentation of the computer-generated experience. In some embodiments, the different computer-generated experiences are related to each other. For example, the different computer-generated experiences may be a preview of a 3D computer-generated experience and the 3D computer-generated experience itself; or they may be a fragment or revision of a computer-generated experience and the full version of the computer-generated experience, etc. In some embodiments, the computing system determines that the first display generating component is placed in a preset spatial relationship or configuration relative to the first user when the display side of the first display generating component faces the first user or a preset part of the first user (e.g., the user's eyes or face), and / or when the user is in a position to view content displayed on the first display generating component (e.g., the user is wearing an HMD, holding an HMD, holding an HMD, placing an HMD in a preset viewing station, connecting an HMD to a preset output device, etc.). In some implementations, the computing system determines which computer-generated experience to display on the first display generating component based on whether the first display generating component (or optionally, the entire HMD surrounding the first display generating component (and optionally a second display generating component)) is worn by a first user (e.g., strapped in and held in front of the user's eyes without hand support). In some implementations, the computing system determines whether the first display generating component is worn by the first user based on the state of devices or sensors other than the first display generating component (e.g., straps or buckles on the housing of the first display generating component, touch or position sensors attached to the housing of the first display generating component, etc.). For example, the straps or buckles optionally have an open state and a closed state, and the straps or buckles are in the closed state when the first display generating component is worn by the first user; and in the open state when the first display generating component is only temporarily placed in front of the user (e.g., the user holds it up to eye level) and is not worn by the user. In some implementations, when the housing of the first display generating component is rested on and supported by the user's nose, ears, head, and / or other parts of the user's body excluding the user's hands, a touch or position sensor on the housing of the first display generating component switches to a first state ("Yes" state); and when the housing of the first display generating component is supported by the user's hands, the touch or position sensor switches to a second state ("No" state). In some implementations, by differentiating how the first display generating component (or the HMD including the first display generating component) remains in a preset configuration relative to the first user so that the user can view the computer-generated experience displayed on the first display generating component, the computing system can more appropriately customize the interaction model and depth of the content presented to the user.For example, the computing system only enables a first interaction model that requires user hand movement as input (e.g., air gestures, micro-gestures, inputs provided on a control device separate from the housing of the first display generating component) when the first display generating component is not held in a preset configuration relative to the first user by the user's hand; and when the first display generating component is held in a preset configuration relative to the first user by the user's hand, the computing system only enables other types of interaction models that do not require the user's hand to be removed from the housing of the first display generating component (e.g., voice interaction, gaze interaction, touch interaction on the housing of the first display generating component).

[0167] exist Figure 7I In this configuration, a first user 7202 is positioned juxtaposed with a first display generating component (e.g., a display 7100, an internal display of an HMD, etc.) at location A 7000-a and facing the display side of the first display generating component. According to some embodiments, this is to illustrate an exemplary scenario where the first display generating component is in a preset configuration relative to the first user or a preset portion of the first user (e.g., the first display generating component is placed in front of the user's face or eyes). Figure 7I In the exemplary scenario shown, the first user 7202 holds the sensor object 7016 in her hand. This position of the sensor object 7016 relative to the user's hand corresponds to the state of the first display generating component when it is not worn by the first user 7202. Another exemplary scenario corresponding to the state of the first display generating component when it is not worn by the first user is when the first display generating component is a display of an HMD (e.g., the internal display of an HMD, a single display of an HMD, etc.) and the HMD's display is held or raised by the first user's hand to face the first user's eyes (e.g., rather than supported by the user's head, nose, ears, or other parts of the user's body other than the user's hand). In some embodiments, in the scenario where the sensor object 7016 is held in the first user's hand (which also corresponds to the first display generating component being supported by the user's hand or not worn by the first user when placed in front of the user's face or eyes), the computing system displays a second computer-generated experience 7030 (e.g., with...). Figure 7H The experience corresponding to the second state of the computing system shown.

[0168] exist Figure 7J In this configuration, the first user and the first display generating component are positioned side-by-side at location A 7000-a and facing the display side of the first display generating component 7100. According to some embodiments, this is to illustrate another exemplary scenario where the first display generating component 7100 is in a preset configuration relative to the first user or a preset portion of the first user (e.g., the first display generating component is placed in front of the user's face or eyes). Figure 7J In the illustrated scenario, the first user 7202 no longer holds the sensor object 7016 in her hand, but has placed it on her body (e.g., her back), so that the first user 7202 is now wearing the sensor object 7016. This position of the sensor object 7016 relative to the user's hand and body corresponds to the state of the first display generating component 7100 when it is worn by the first user. Another exemplary scenario corresponding to the state of the first display generating component when it is worn by the first user is when the first display generating component is the display of an HMD (e.g., the internal display of the HMD) and the HMD is normally worn on the first user's head (e.g., strapped, fastened, or completely resting on the user's nose, ears, and / or head, rather than being supported by the user's hands). In some implementations, in scenarios where the sensor object 7016 is worn by a user and not held in the hand of a first user (which also corresponds to the first display generating component being worn by the user and not supported by the user's hand when placed in front of the user's face or eyes), the computing system displays a third computer-generated experience 7036 (e.g., a different experience, which also corresponds to...). Figure 7H The second state of the computing system shown (e.g., square 7026 is shown) differs from the second computer-generated experience (e.g., due to the state of the sensor object 7016 (and correspondingly, the wearing state of the first display generating component (e.g., display 7100 or the internal display of the HMD)). Figure 7I The experience in 7030).

[0169] like Figure 7JAs shown, the computing system (e.g., using camera 7104 and / or other sensors) detects that the first user 7202 moves her hand in the air to provide a gesture, and in response, moves a virtual object 7032 onto top of the representation 7028' of the physical object 7028. In some embodiments, the computing system disables at least some of the input devices (e.g., touch-sensitive surfaces, buttons, switches, etc.) located on the housing of the first display generating component (e.g., display 7100, the internal display of the HMD, a single display of the HMD, etc.). In some embodiments, the computing system enables at least one interaction model (e.g., an interaction model supporting gestures, micro-gestures, and / or input detected via input devices separate from the housing of the first display generating component) that is not enabled when the first display generating component is not worn by the first user 7202 (e.g., determined as based on the state of sensor object 7016 or other sensors, etc.). In some implementations, the third computer-generated experience 7036 and the second computer-generated experience 7030 are related experiences with corresponding content (e.g., the same content or different versions of the same content) but different interaction models (e.g., different interaction models, or overlapping but different groups of interaction models, etc.).

[0170] like Figure 7I and Figure 7J As shown, in some embodiments, the computing system optionally includes a second display generating component (e.g., a display (e.g., display 7102) included in a different housing from the first display generating component, or a display (e.g., an HMD with an internal display and an external display) included in the same housing as the first display generating component (e.g., back-to-back or otherwise facing different directions). In some embodiments, the second display generating component optionally displays status information (e.g., a visual representation of changes in content) related to content displayed via the first display generating component, and optionally displays status information related to the state of the first user (e.g., an image or representation of the user's face or eyes) and / or the operating mode of the computing system (e.g., mixed reality mode, virtual reality mode, full pass-through mode, parental control mode, privacy mode, DND mode, etc.). In some embodiments, the second display generating component also displays user interface elements corresponding to a computer-generated experience that can be used for the experience displayed by the first display generating component. Figures 7A to 7E and Figures 7F to 7I To describe in more detail the operation of the second display generation component and the corresponding operation of the first display generation component, and relative to Figures 8 to 13The process is described below. In some implementations, computing systems that display different computer-generated experiences based on whether the first display generation component is worn by the user when the first display generation component is placed relative to a preset configuration of the user do not have another display generation component other than the first display generation component, and therefore do not display status information and / or user interface elements indicating the availability of the computer-generated experience described herein.

[0171] In some implementations, depending on whether the first display generating component is worn by the first user when the first display generating component is placed in a preset configuration relative to the first user (e.g., the display side of the first display generating component faces the user's eyes or face and / or is within a threshold distance from the user's face, etc.) (e.g., whether the HMD is strapped or fastened to the user's head and remains in front of the user's eyes without the support of the user's hands, or is simply held in front of the user's eyes by the user's hands and would fall off without the support of the user's hands), the computing system optionally uses the first display generating component to display a different type of user interface (e.g., system user interface (e.g., application launch user interface, home screen, multitasking user interface, configuration user interface, etc.) than an application user interface (e.g., camera user interface, infrared scanner user interface (e.g., showing a thermal map of the current physical environment), augmented reality measurement application (e.g., automatically displaying measurements of physical objects in the camera view, etc.)). In some implementations, when the computing system displays an application user interface using a first display generating component, in response to user input detected via an input device (e.g., a touch sensor, a contact strength sensor, a button, a switch, etc.) disposed on the housing of the first display generating component, the computing system captures a photograph or video of the physical environment within a camera view.

[0172] In some embodiments, when determining a response to user input detected when the user holds the first display generating component in front of his / her eyes and is not wearing the first display generating component, the computing system prioritizes touch input detected on a touch-based input device located on the housing of the first display generating component over micro-gesture input and / or air gesture input detected in front of the first user (e.g., micro-gesture input and air gesture input performed by the user's hand that is not holding the housing of the first display generating component). In some embodiments, when determining a response to user input detected when the user is wearing the first display generating component (e.g., when the user's hand is not needed to support the first display generating component), the computing system prioritizes micro-gesture input and / or air gesture input detected in front of the first user over touch input detected on a touch-based input device located on the housing of the first display generating component. In some implementations, in response to the simultaneous detection of multiple types of input (e.g., input performed by a hand away from the first display generating component, input performed by a hand touching the first display generating component or its housing, etc.), based on determining that the first display generating component is worn by the user in a preset configuration relative to the user (e.g., the HMD including the first display generating component is strapped to the user's head, fastened, not supported by the user's hand, etc.), the computing system enables interaction with the displayed computer-generated experience based on a gesture performed by a hand positioned away from the first display generating component and its housing (e.g., an air gesture, a micro-gesture, etc.) (e.g., the gesture is captured by a camera on the HMD, a mechanical or touch-sensitive input device, or a sensor worn on the user's hand, etc.). Based on the determination that the first display generating component is not worn by the user in a preset configuration relative to the user (e.g., not strapped to the user's head, not fastened, supported by the user's hand, etc.), the computing system enables interaction with the displayed computer-generated experience based on gestures performed by the hand on the first display generating component or its housing (e.g., touch gestures, manipulation of physical controls, etc.) (e.g., the gesture is captured by the touch-sensitive surface of the HMD's housing, buttons or switches on the HMD's housing, etc.).

[0173] Figures 7K to 7M A computing system is shown that includes at least a first display generation component (e.g., display 7100, an internal display of the HMD, a single display of the HMD, etc.) and an optional second display generation component (e.g., display 7102, an external display of the HMD, etc.). Figure 1 The computing system 101 or Figure 4The computing system 140 displays computer-generated experiences (e.g., augmented reality experiences, augmented virtual reality experiences, virtual reality experiences, etc.) to the user via a first display generation component (e.g., display 7100, the internal display of the HMD, a single display of the HMD, etc.) based on physical interactions between the user and physical objects in the physical environment (e.g., picking up and playing an instrument, picking up and opening a book, holding and opening a box, etc.). In some embodiments, only certain physical interactions that satisfy preset criteria corresponding to the physical object can trigger the display of the computer-generated experience. In some embodiments, the different computer-generated experiences are optionally displayed based on which set of criteria is satisfied by the physical interaction with the physical object. In some embodiments, the different computer-generated experiences include different augmented reality experiences corresponding to different modes of manipulating the physical object (e.g., tapping, playing, stroking, opening, swinging, etc.). Figures 7K to 7M Used to illustrate the process described below, including Figures 8 to 13 The process in.

[0174] like Figure 7K As shown, a user (e.g., user 7202) exists in scene 105, which includes physical objects (e.g., objects including lid 7042 and box 7040, books, musical instruments, etc.) in a room with walls and a floor. Figure 7K In this embodiment, the user holds a first display generating component (e.g., a monitor 7100, HMD, handheld device, etc.) in their hand 7036. In some embodiments, the first display generating component is not held in the user's hand 7036, but is supported by a housing or support structure resting on the user's body (e.g., head, ear, nose, etc.). In some embodiments, the first display generating component (e.g., a head-up display, projector, etc.) is placed in front of the first user's eyes or face and is supported by other support structures that are not part of the user's body (e.g., a desktop, TV cabinet, etc.).

[0175] In some implementation schemes, such as Figure 7KAs shown, the computing system provides an augmented reality view 105' of the physical environment (e.g., a room including physical objects) via a first display generation component (e.g., a display 7100, a monitor of an HMD, etc.). In the augmented reality view of the physical environment, a portion of the view includes representations of physical objects (e.g., representations 7042' including the lid 7042 and 7040' including the body 7040) and, optionally, representations of the surrounding environment (e.g., representations 7044' of the support structure 7044 supporting the physical objects and representations of the walls and floors of the room). In addition to representations of physical objects in the environment, the computing system also displays virtual content (e.g., user interface objects, visual enhancements of physical objects, etc.), including one or more computer-generated experiences corresponding to physical objects (e.g., a box including the lid 7042 and the body 7040, another physical object in the environment, etc.) that can be used for visual indications (e.g., label 7046 or other visual indications, etc.) displayed via the first display generation component. Figure 7K As shown in (B), in some embodiments, a visual indication (e.g., label 7046) is displayed in a view of the physical environment at a location corresponding to the position of the physical object (e.g., at the location of the lid 7042). For example, in a view of the physical environment displayed via the first display generating component, label 7046 appears to be located on top of the lid 7042.

[0176] In some embodiments, visual cues (e.g., label 7046 or other visual cues, etc.) include descriptive information (e.g., icons, graphics, text, animations, video clips, images, etc.) that can be used for a computer-generated experience displayed by the first display generating component. In some embodiments, the computing system stops displaying (or abandons displaying) visual cues in the view of the augmented reality environment when the first display generating component of the computing system or one or more cameras move in the physical environment, and / or when physical objects move in the physical environment such that the augmented reality view of the physical environment shown by the first display generating component includes only a representation of a portion of the physical object less than a threshold (e.g., less than 50% of the physical object or excluding key parts of the physical object (e.g., the lid of a box 7042, the title text of a book, the sound of a musical instrument, etc.)).

[0177] In some implementations, visual cues include prompts or guidance information (e.g., animated illustrations, indicators pointing to specific portions of a representation of a physical object, etc.) related to the physical interaction required to trigger the display of the computer-generated experience. In some implementations, the computing system displays only the prompts or guidance information related to the physical interaction required to trigger the display of the computer-generated experience in response to detecting user input that meets preset first criteria (e.g., criteria for assessing a user's interest in viewing the computer-generated experience, criteria for detecting the user's presence, criteria for detecting hand contact with a physical object, etc.). Figure 7L As shown, in some embodiments, when the computing system detects that a user touches a physical object with his / her hand without performing the necessary interaction to trigger the display of a computer-generated experience (e.g., hand 7038 touches the lid 7042 or box 7040 of the physical object, but does not open the lid 7042), the computing system displays a prompt or guidance message (e.g., an animated arrow 7048 or other visual effects or virtual objects, etc.) at a location in the augmented reality environment view corresponding to the location of the lid 7042 to prompt the user to open the lid 7042. In some embodiments, the prompt and guidance message is updated according to how the user interacts with the physical object (e.g., the direction of the arrow, the sequence of animations, etc.). In some embodiments, when the user uses his / her hand to manipulate a physical object in the physical environment, a representation of the user's hand (e.g., representation 7038' of hand 7038) is shown in the augmented reality view 105' of the physical environment. It should be noted that these prompts and guidance messages differ from the actual computer-generated experience that can be used to display via the first display generating component after detecting a required physical interaction with the physical object (e.g., opening the lid 7048, some other interaction (e.g., picking up the box 7040 from the support 7044 after removing the lid 7048)). In some embodiments, the computing system does not display any prompts or guidance messages in response to a physical interaction with the physical object not meeting the criteria for triggering the display of a computer-generated experience (e.g., the computing system does not display the computer-generated experience and does not display prompts and guidance messages, but optionally maintains the display of visual indicators (e.g., label 7046) to indicate that the computer-generated experience can be used for display).

[0178] like Figure 7MAs shown, in some embodiments, when the computing system detects that a user has performed a physical interaction with a physical object necessary to trigger the display of a computer-generated experience, the computing system displays the computer-generated experience. For example, in response to detecting that a user's hand 7038 has lifted the lid 7042 from the housing 7040, the computing system determines that the required physical interaction with the physical object has met preset criteria for triggering the display of the computer-generated experience, and displays the computer-generated experience using a first display generation component (e.g., display 7100, the internal display of the HMD, a single display of the HMD, etc.). Figure 7M In this augmented reality experience, the computer-generated experience is an augmented reality experience that shows the user's hand 7038', the lid 7042', and the box 7040' in positions and orientations corresponding to their physical positions and orientations in the physical environment. Furthermore, in some embodiments, the augmented reality experience also simultaneously shows virtual content (e.g., a virtual sphere 7050 that appears to jump out of the box 7040 and projects virtual shadows onto the inside of the box, a virtual platform 7052 that replaces the representation of the physical support 7044 below the box 7040, etc.) with a representation of the physical environment. Additionally, in some embodiments, the representation of walls in the physical environment is replaced by virtual overlays in the augmented reality experience. In some implementations, as the user continues to interact with physical objects after the computer-generated experience has begun, the computing system displays changes in the augmented reality environment based on the user's physical manipulation of the physical objects (e.g., box 7040, lid 7042, etc.) and optionally based on other inputs detected by various input devices of the computing system (e.g., gesture input, touch input, gaze input, voice input, etc.). In some implementations, the computer-generated experience continues in a manner determined based on continued physical interaction with the physical objects. For example, in response to detecting that the user moves the lid 7042 in the physical environment, the computing system moves the representation 7042' of the lid 7042 to push the virtual ball 7050 into the empty space above the representation 7040' of the box 7040; and in response to detecting that the user puts the lid 7042 back on the box 7040, the computing system displays the representation 7042' of the lid 7042 back on the representation 7040' of the box 7040 and stops displaying the virtual ball 7050. In some implementations, the computing system requires the user to make physical contact with a physical object when performing a physical interaction necessary to trigger the display of a computer-generated experience; and the computing system stops displaying the computer-generated experience based on a time thresholded by which the user has ceased physical contact with the physical object for more than a certain amount of time. For example, in some implementations, the computing system stops displaying the computer-generated experience immediately upon detecting that the physical object has been released from the user's hand. In some implementations, the computing system stops displaying the computer-generated experience when it detects that the physical object has fallen and come to rest on another physical surface after being released from the user's hand.

[0179] In some implementations, the visual feedback provided in response to detecting a user's physical interaction with a physical object prior to triggering the display of a computer-generated experience includes a preview of the computer-generated experience and has visual features that are dynamically updated based on the characteristics of the physical interaction when it is detected. For example, optionally, the extent (e.g., size, dimension, angular range, etc.) of animations, visual effects, and / or virtual objects, the amount of detail in the visual feedback, the brightness, color saturation, visual sharpness, etc., of the visual feedback may be adjusted (e.g., dynamically in real-time, periodically, etc.) based on characteristic values ​​of the interaction with physical objects in the physical environment (e.g., characteristic values ​​include distance moved, angular range of movement, speed of movement, type of interaction, distance from a predetermined reference point, etc.). For example, in some implementations, when the physical object is a book, as the user in the physical environment slowly turns the book cover, the colors and light of the computer-generated experience emerge from the gap between the cover and the first page and become increasingly bright and saturated as more of the cover is turned. Optionally, the full computer-generated experience is initiated in a three-dimensional environment when the book cover is turned more than a threshold amount and a first criterion is met. In another example, as the user lifts a corner of the lid 7042 slightly, virtual light is shown emerging from representation 7040' of the box 7040, and a fleeting glimpse of a virtual sphere 7050 is shown. As the user lifts the corner of the lid 7042 higher, more virtual light is shown emerging from representation 7040' of the box 7040, and the virtual sphere 7050 begins to wiggle within representation 7040' of the box 7040. When the user finally removes the lid 7042 from the box 7040, a computer-generated experience is activated, the entire 3D environment changes, replacing the representation of the room with the virtual platform 7052, and the virtual sphere 7050 jumps out of representation 7040' of the box.

[0180] In some implementations, the computer-generated experience is optionally triggered by more than one type of physical interaction. In other words, a first mode of interaction with the physical object and a second mode of interaction with the physical object may satisfy criteria for triggering a computer-generated experience associated with the physical object. For example, a computer-generated experience associated with a book may be initiated in response to a user picking up a book and placing it on a bookshelf with the cover upright relative to the user's face, and in response to the user picking up the book and turning the cover with his / her hand. In some implementations, the computer-generated experience may be initiated from different parts of the computer-generated experience. For example, the criteria for triggering a computer-generated experience associated with a physical object may be satisfied when the same mode of interaction with the physical object is used but with different parameter values ​​(e.g., different pages, different speeds, different times, etc.). For example, a computer-generated experience associated with a book may be initiated from a first part of the computer-generated experience in response to a user picking up a book and turning the first page; and a computer-generated experience associated with a book may be initiated from a second, different part of the computer-generated experience in response to a user picking up a book and turning the page from a previously bookmarked page. In another example, slowly opening a book triggers a computer-generated experience with soothing background music and / or softer colors, while quickly opening the book triggers a computer-generated experience with more upbeat background music and brighter colors. The book example is merely illustrative. The same principle applies to other computer-generated experiences associated with other types of physical objects and other triggering physical interactions. In some implementations, different computer-generated experiences are associated with the same physical object and are triggered by different ways of interacting with the physical object. For example, a box is associated with two different computer-generated experiences, and a first computer-generated experience is triggered when the user opens the lid of the box (e.g., a virtual ball jumps out of the box for the user to push with the lid), and a second computer-generated experience is triggered when the user flips the box upside down (e.g., a virtual bug emerges from the bottom of the box and chases the user's finger moving on the bottom of the box). In some implementations, different ways of interacting with the physical object trigger different versions of computer-generated experiences that enable different input modalities. For example, when holding a book in one hand and turning the pages with the other, enable single-handed air gestures (e.g., air taps, waves, sign language gestures, etc.) and micro-gestures for computer-generated interactive experiences, and disable touch gestures for computer-generated interactive experiences. When holding the book open with both hands, disable air gestures and enable touch gestures (e.g., taps, swipes, etc.) on the back, front, and / or side surfaces of the book for computer-generated interactive experiences.

[0181] Figures 7N to 7QThe diagram illustrates how, according to some embodiments, an operation is performed or not performed in response to the detection of input on the housing of a display generating component, depending on whether one or two hands are detected on the housing at the time of the input. In some embodiments, touch input as an input modality is disabled when two hands are simultaneously detected on the housing of the display generating component (e.g., on the housing of the HMD containing the display generating component, on the frame of the display generating component, etc.). In some embodiments, the computing system responds to touch input performed by either hand of the user on the housing of the display generating component (e.g., on a touch-sensitive surface or other touch sensor disposed on or around the structure of the display generating component) as long as only one hand (e.g., the hand providing the touch input) touches the housing of the display generating component; and if the computing system detects that an additional hand is also touching the housing of the display generating component at the time of detecting the touch input, the computing system ignores the touch input and abandons the operation corresponding to the touch input (e.g., activating a control, interacting with a computer-generated experience, etc.). In some embodiments, the computing system uses the same sensors and input devices used to detect touch input provided by a single hand to detect the presence of two hands. In some embodiments, a camera is used to capture the position of the user's hands, and the computing system uses the image from the camera to determine whether both of the user's hands are touching the housing of the display generating component when one or more touch sensors present on the housing of the display generating component detect touch input. In some embodiments, other means are implemented for detecting touch input and / or whether one or both of the user's hands are touching or supporting the housing of the display generating component. For example, according to various embodiments, position sensors, proximity sensors, mechanical sensors, etc., can be used to detect the presence and posture of the user's hands on the housing of the display generating component.

[0182] exist Figure 7N In this context, a user (e.g., user 7202) exists within a physical scene 105, which includes physical objects (e.g., box 7052), walls 7054 and 7056, and a floor 7058. User 7202 stands in front of a display generation component 7100 (e.g., a flat panel display, projector, HMD, internal display of the HMD, head-up display, etc.) supported by a bracket. The display generation component 7100 displays a view of the physical environment 105. For example, in some embodiments, the view of the physical environment is a camera view of the physical environment. In some embodiments, the view of the physical environment is provided through a transparent portion of the display generation component. In some embodiments, the display generation component 7100 displays an augmented reality environment with a view of the physical environment and virtual objects and content displayed as an overlay on or replacing a portion of the view of the physical environment. Figure 7NAs shown, the view of the physical environment includes representation 7052' of box 7052, representations 7054' and 7056' of walls 7054 and 7056, and representation 7058' of floor 7058. Figure 7N In this embodiment, user 7202 does not touch the housing of the display generating component, and the computing system does not detect any touch input on the housing of the display generating component. In some embodiments, a representation 7038' of the user's hand 7038 shown via the display generating component 7100 is included as part of the representation of the physical environment. In some embodiments, the display generating component 7100 displays a virtual environment or displays nothing until any touch input is detected on its housing.

[0183] In some embodiments, in addition to touch input, the computing system is optionally configured to also detect hover input near the housing of the display generating component. In some embodiments, a proximity sensor located on the housing of the display generating component is configured to detect a user's finger or hand approaching the housing of the display generating component, and to generate an input signal based on the proximity of the finger or hand relative to the housing of the display generating component (e.g., proximity to a portion of the housing configured to detect touch input, other portions of the housing, etc.). In some embodiments, the computing system is configured (e.g., using proximity sensors located at different portions of the housing of the display generating component) to detect corresponding hover inputs at different locations near the housing of the display generating component, and to provide different feedback depending on the location of the hover input. In some embodiments, the computing system adjusts the values ​​of various features of the visual feedback based on the hovering distance of the detected hover input (e.g., the distance of a fingertip from the surface of the housing or a touch-sensitive portion of the housing).

[0184] like Figure 7OAs shown, in some embodiments, the computing system detects that a hand 7038 or a portion thereof (e.g., one or both fingers of hand 7038, etc.) has moved within a threshold distance from a first touch-sensitive portion of the housing of the display generating component 7100 (e.g., the upper portion of the left edge of the housing of the display generating component 7100, the left or top edge portion of the HMD, etc.). In response to detecting that a hand 7038 or a portion thereof has moved within the threshold distance from the first touch-sensitive portion of the housing of the display generating component 7100, the computing system optionally displays one or more user interface objects at a position corresponding to the position of the hand 7038 or a portion thereof. For example, a user interface object 7060 may be displayed near the upper portion of the left edge of the housing of the display generating component 7100 (or the left or top edge portion of the HMD, etc.), next to the hand 7038 or a raised finger of the hand 7038. In some embodiments, the computing system dynamically updates the position of one or more displayed user interface objects (e.g., moving user interface object 7060 closer or further from the left edge of the housing, and / or moving it up and down along the left edge of the housing) based on changes in the position of the hand 7038 or a portion thereof near the housing of the display generating component (e.g., moving closer / further, and / or moving it up and down along the left edge of the housing). In some embodiments, the computing system dynamically updates the appearance of one or more displayed user interface objects (e.g., changing size, shape, color, opacity, resolution, content, etc.) based on changes in the position of the hand or a portion thereof near the housing of the display generating component. In some embodiments, the computing system optionally changes the type of user interface object displayed near the first touch-sensitive portion of the housing of the display generating component based on changes in the posture of the hand 7038 near the housing of the display generating component (e.g., raising two fingers instead of one, raising different fingers, etc.). This can be achieved by changing the display of a representation of a first control (e.g., volume control, power control, etc.) to a representation of a second control (e.g., brightness control, WiFi control, etc.), or by changing the display of a first type of indicator (e.g., scroll bar, button, etc.) to a second type of indicator (e.g., scroll wheel, switch, etc.). In some embodiments, as the hand 7038 moves closer to the first touch-sensitive portion of the housing of the display generating component, one or more user interface objects increase in size from simple indications of the control (e.g., small dots, blurred shadows, etc.) to more specific and clear representations of the control (e.g., buttons or switches with graphics and / or text indicating the state of the computing system or the display generating component). In some embodiments, one or more user interface objects (e.g., user interface object 7060 or other user interface objects, etc.) are displayed simultaneously with a view of the physical environment. In some implementations, for example, if a virtual environment is displayed before hover input of hand 7038 is detected, one or more user interface objects are displayed simultaneously with the virtual environment.In some implementations, for example, if a view of the physical environment or any other virtual content is not displayed before hover input is detected, one or more user interface objects are displayed without simultaneously displaying a view of the physical environment or any other virtual content.

[0185] In some implementation schemes, such as Figure 7O As shown, for example, after detecting hover input near a first touch-sensitive portion on the housing of the display generating component, the computing system detects touch input at the first touch-sensitive portion on the housing of the display generating component. In some embodiments, in response to detecting touch input at a first portion of the touch-sensitive portion of the housing of the display generating component, based on determining that touch input has been detected and no other hand has been detected on the housing of the display generating component, the computing system performs a first operation corresponding to the touch input. For example, as... Figure 7OAs shown, in response to a touch input detected on the upper portion of the left edge of the housing of the display generating component, the computing system activates an augmented reality experience that includes a combination of some virtual content (e.g., a virtual overlay 7062) and a view of the physical environment (e.g., displaying the virtual overlay 7062 at a position corresponding to the position of the box 7052, such that the overlay 7062 appears to be placed on the front surface of the representation 7052' of the box 7052 in the augmented reality environment). In some embodiments, an initial downward touch of the user's hand 7038 or a portion of the hand only causes a touch to be detected on the housing of the display generating component and the touch has not yet met the criteria for initiating any particular operation on the computing system (e.g., a user interface object (e.g., user interface object 7060) is displayed in its full-function state when it is not yet displayed or is only displayed as an indication, the user interface object (e.g., user interface object 7060) changes its appearance to indicate that a touch has been detected, etc.). In some implementations, the computing system evaluates the touch input provided by hand 7038 against criteria for detecting one or more valid touch inputs, and performs the first operation associated with one or more user interface objects based on criteria for triggering a first operation (e.g., lowering display brightness, lowering volume, switching to AR mode, etc.) associated with one or more user interface objects (e.g., criteria for detecting an upward swipe, double tap, single tap, etc.); and performs the second operation associated with one or more user interface objects based on criteria for triggering a second operation (e.g., increasing display brightness, increasing volume, switching to VR mode, etc.) associated with one or more user interface objects (e.g., criteria for detecting a downward swipe, single tap, double tap, etc.).

[0186] In some implementation schemes, such as Figure 7PAs shown, the computing system detects a user's hand 7038 approaching a second touch-sensitive portion of the housing of the display generating component (e.g., the lower left edge of the housing of the display generating component, the lower left edge of the HMD, the upper right edge of the HMD, etc.), and optionally displays one or more user interface objects (e.g., user interface object 7064 or other user interface objects) in response. When the computing system detects touch input on the second touch-sensitive portion of the housing of the display generating component, based on determining that only a single hand (e.g., hand 7038) touches the housing of the display generating component when the touch input is detected, the computing system performs a second operation corresponding to the touch input performed by hand 7038 or a portion thereof. In this example, activation of control 7064 triggers the initiation of an augmented reality experience, which includes some virtual content (e.g., a virtual ball 7066, other virtual objects, etc.) displayed in conjunction with a view of the physical environment (e.g., displaying the virtual ball 7066 at a position corresponding to the position of floor 7058, such that the virtual ball 7066 appears to be resting on the surface of the representation 7058' of floor 7058). Other features related to this exemplary scenario are similar to those relative to... Figure 7O The only difference among the features described will be the detection of a hand 7038 near and / or at the first touch-sensitive portion of the housing of the display generating component, and the specific visual feedback and the performed operation will correspond to the touch input detected on or near the first touch-sensitive portion of the housing of the display generating component. For the sake of brevity, similar features will not be described again herein.

[0187] Figure 7Q This illustration shows another hand (e.g., hand 7035) touching (e.g., holding, supporting, otherwise contacting, etc.) the housing of the display generating component (e.g., display 7100, HMD, etc.) when touch input from hand 7038 is detected on the housing of the display generating component. Figure 7Q In the middle, hand 7038 touches the first touch-sensitive portion and the second touch-sensitive portion of the housing of the display generating component, respectively, and provides the same function as previously triggered to perform the first operation (e.g., display). Figure 7O The virtual overlay 7062 in the middle) and the second operation (e.g., display) Figure 7P The virtual sphere 7066 in the display receives the same touch inputs as those touch inputs. However, when one of the touch inputs from hand 7038 is detected, the computing system abandons the operation corresponding to the detected touch input, based on the determination that another hand (e.g., hand 7036) also touches the housing of the display generating component (e.g., based on the determination that the other hand touches the side of the housing opposite to the side where the touch input was detected, determining that the other hand touches anywhere on the housing, etc.). For example, as Figure 7QAs shown, when another hand 7036 touches the housing of the display generating component, the computing system may still optionally display a user interface object (e.g., user interface object 7060, user interface object 7064, etc.) near the location of the hand 7038 providing valid touch input, but without initiating the corresponding computer-generated experience. In some embodiments, if hover input from both hands is detected simultaneously near the touch-sensitive portion of the housing of the display generating component, and / or when touch input is detected simultaneously on both sides of the housing of the display generating component, the computing system does not display any user interface object. In some embodiments, when both hands are simultaneously detected approaching and / or touching the display generating component (e.g., HMD, other types of display generating components, etc.) are detected and / or touching the display generating component, it is more likely that the user wants to remove the display generating component rather than provide input to it. Therefore, it is more advantageous to ignore such hover or touch input without explicit user instruction (e.g., to reduce user confusion, save power, etc.).

[0188] In some embodiments, the state of the computing system changes based on operations performed on touch input detected on the housing of the display generating component. For example, the computing system switches to a first state based on determining that the touch input meets a first criterion; and switches to a second state different from the first state based on determining that the touch input meets a second criterion different from the first criterion. In some embodiments, the first and second criteria have different location-based criteria, requiring touch input to be detected at different locations on the housing. In some embodiments, the first and second criteria have different intensity-based criteria, requiring touch input to meet different intensity thresholds. In some embodiments, the first and second criteria have different duration-based criteria, requiring touch input to be detected on the housing with movement less than a threshold amount and lasting for a different threshold amount of time. In some embodiments, the first and second criteria have different distance-based criteria, requiring touch input to move beyond different threshold distances. In some embodiments, the first and second criteria have different touch pattern criteria, requiring touch input to include different numbers of contacts and / or contact movement patterns. In some embodiments, the first and second criteria have different movement criteria, requiring touch input to include contact movement that meets different speed thresholds and / or directional requirements. In some implementations, the first standard and the second standard have different posture requirements for the hand providing touch input. In some implementations, the first standard and the second standard have different combinations of two or more of the aforementioned standards (e.g., standards related to the location, intensity, duration, movement and / or quantity of contact, hand posture, etc.).

[0189] In some implementations, the operation performed based on touch input changes the level of immersion associated with the content displayed by the display-generating component. For example, the computing system optionally changes from a first state to a second state in response to a first touch input performed only when a single hand is detected on the housing of the display-generating component. In the first state, the computing system displays a computer-generated experience in a perspective mode with a physical environment (e.g., mixed reality mode), and in the second state, the computing system displays the same computer-generated experience in an immersive mode without a perspective view of the physical environment (e.g., virtual reality mode). In some implementations, the computing system switches back from the second state to the first state when a second touch input is detected on the housing of the display-generating component while only a single hand is touching it. In some implementations, the computing system optionally changes from a third state (e.g., a fully transparent view of the physical environment without displaying any computer-generated experience, such as a transparent view or mixed reality mode) to a first state (e.g., transparent mode or mixed reality mode) or a second state (e.g., immersive mode or virtual reality mode) in response to the detection of a third touch input performed only when a single hand is detected on the housing of the display generating component. In some implementations, when the computing system displays computer-generated content in an immersive mode (e.g., virtual reality mode, transparent view without a physical environment, etc.) via the display generating component, if two hands are detected on the housing of the display generating component, the computing system optionally switches back to a transparent view or special mode that visually indicates the nearest physical object in the virtual content as a special operation. When a user is unsure of their position in the physical environment (e.g., when they are worried about bumping into something), they may instinctively place both hands on the housing of the HMD, and visually indicating the nearest physical object in the virtual content without stopping the virtual content can help the user reorient themselves without completely ending the immersive experience. In some implementations, the computing system modifies the audio functionality of a display generation component or an audio generation component based on touch input. For example, based on a first touch input, the computing system changes from a first audio output mode (e.g., audio pass-through mode or other audio output mode) to a second audio output mode (e.g., audio immersion mode or other audio output mode), and / or from a first audio blocking mode (e.g., noise-free mode or other audio blocking mode) to a second audio blocking mode (e.g., full noise cancellation mode or other audio blocking mode). In some implementations, the audio output mode is changed in conjunction with a change in the level of immersion caused by the touch input.

[0190] In some implementations, when displaying one or more user interface objects (e.g., Figures 70 to 7Q User interface objects 7060 and 7064 in the document) and / or computer-generated experiences (e.g., Figure 7Oand Figure 7P When the computing system detects that the touch input is no longer detected (in the virtual overlay 7062 and virtual ball 7066), the computing system may optionally stop displaying user interface objects and / or computer-generated experiences.

[0191] In some embodiments, more than one display generation component is enclosed within the same housing. Optionally, in addition to the user-facing display generation component, another display generation component is included, and this other display generation component faces away from the user. In some embodiments, relative to... Figures 7A to 7J A computing system comprising two display generation components is described, wherein the second display generation component is optionally used to display state information and / or context information related to the state of the first display generation component and the user looking at the display side of the first display generation component. In some embodiments, in response to touch input, the on / off state of a privacy mode of the first display generation component is changed, and the computing system adjusts the state information displayed on the second display generation component according to the touch input and the privacy mode change. In some embodiments, when a touch input detected on the housing of the first display generation component (and optionally also on the housing of the second display generation component) causes a change in the level of immersion associated with a computer-generated experience, the computing system adjusts the state information displayed on the second display generation component according to the change in the level of immersion associated with the computer-generated experience displayed via the first display generation component. In some embodiments, the state information also includes the state of the user's eyes in front of the display side of the first display generation component, and also updates the representation of the user's eyes in response to touch input. Figures 7A to 7J Other aspects of the changes made to the state of the second display generating component are described, which are combined with the changes made to the state of the first display generating component as a result of touch input detected on the housing of the first display generating component when only a single hand is detected on the housing of the first display generating component.

[0192] In some implementations, the input gestures used in the various examples and implementations described herein optionally include discrete micromotor gestures (which are performed by moving the user’s fingers relative to other fingers or portions of the user’s hand), and optionally, it is not necessary to primarily move the user’s entire hand or arm away from its natural position and posture to perform the operation immediately before or during the gesture for interacting with the virtual or mixed reality environment.

[0193] In some implementations, input gestures are analyzed by a sensor system (e.g., sensor 190, ...). Figure 1 Image sensor 314, Figure 3The sensor system uses captured data or signals for detection. In some implementations, the sensor system includes one or more imaging sensors (e.g., one or more cameras, such as motion RGB cameras, infrared cameras, depth cameras, etc.). For example, the one or more imaging sensors are computing systems (e.g., Figure 1 The computing system 101 (e.g., portable electronic device 7100 or HMD) includes components that provide data to the computing system, which includes display generation components (e.g., Figure 1 , Figure 3 and Figure 4 The display generating component 120 (e.g., a touchscreen display, stereoscopic display, display with a transparent portion, etc., used as a display and touch-sensitive surface) is included. In some embodiments, the one or more imaging sensors include one or more rear-facing cameras on the side of the device opposite to the display of the device. In some embodiments, input gestures are detected by a sensor system of a head-mounted system (e.g., a VR headset including a stereoscopic display that provides a left image for the user's left eye and a right image for the user's right eye). For example, one or more cameras, as components of the head-mounted system, are mounted on the front and / or lower side of the head-mounted system. In some embodiments, one or more imaging sensors are located in the space in which the head-mounted system is used (e.g., arranged around the head-mounted system at various locations in a room), such that the imaging sensors capture images of the head-mounted system and / or the user of the head-mounted system. In some embodiments, input gestures are detected by a sensor system of a heads-up device (e.g., a heads-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics). For example, one or more imaging sensors are attached to the interior surface of a car. In some embodiments, the sensor system includes one or more depth sensors (e.g., a sensor array). For example, the one or more depth sensors may include one or more light-based (e.g., infrared) sensors and / or one or more sound-based (e.g., ultrasonic) sensors. In some embodiments, the sensor system includes one or more signal transmitters, such as light transmitters (e.g., infrared transmitters) and / or sound transmitters (e.g., ultrasonic transmitters). For example, when light (e.g., light from an array of infrared light transmitters with a predetermined pattern) is projected onto a hand (e.g., hand 7200), the one or more cameras capture an image of the hand under illumination, and the captured image is analyzed to determine the position and / or structure of the hand. Using signals from image sensors pointing towards the hand to determine input gestures, rather than using signals from touch-sensitive surfaces or other direct contact mechanisms or proximity-based mechanisms, allows the user the freedom to choose whether to perform large movements or remain relatively still when providing input gestures using his / her hand, without being subject to limitations imposed by a particular input device or input area.

[0194] In some implementations, micro-tap input is a tapping input where the user's thumb is over the index finger (e.g., on the side of the index finger adjacent to the thumb). In some implementations, tapping input is detected without requiring the thumb to be lifted from that side of the index finger. In some implementations, tapping input is detected based on determining that a downward movement of the thumb is followed by an upward movement of the thumb, wherein the thumb is in contact with that side of the index finger for less than a threshold amount of time. In some implementations, tap-hold input is detected based on determining that the thumb moves from the lifted position to the pressed position and remains in the pressed position for at least a first threshold amount of time (e.g., a tap time threshold or another time threshold longer than the tap time threshold). In some implementations, the computing system requires the hand as a whole to remain substantially still in the position for at least a first threshold amount of time in order to detect tap-hold input performed by the thumb on the index finger. In some implementations, touch-hold input is detected without requiring the hand as a whole to remain substantially still (e.g., the hand as a whole can move while the thumb rests on that side of the index finger). In some implementations, tap-hold-drag input is detected when the thumb presses on the side of the index finger and the hand as a whole moves while the thumb rests on the side of the index finger.

[0195] In some embodiments, the micro-flick gesture is a push or flick input performed by moving the thumb across the index finger (e.g., from the palmar side to the back side of the index finger). In some embodiments, the extension movement of the thumb is accompanied by an upward movement on that side away from the index finger, for example, as in an upward flick input performed by the thumb. In some embodiments, the index finger moves in the opposite direction to the thumb during the forward and upward movement of the thumb. In some embodiments, a reverse flick input is performed by moving the thumb from an extended position to a retracted position. In some embodiments, the index finger moves in the opposite direction to the thumb during the backward and downward movement of the thumb.

[0196] In some embodiments, a micro-swipe gesture is a swipe input performed by moving the thumb along the index finger (e.g., along the side of the index finger adjacent to the thumb or on that side of the palm). In some embodiments, the index finger is optionally in an extended (e.g., substantially straight) or curled state. In some embodiments, the index finger moves between an extended and curled state during the movement of the thumb in the swipe input gesture.

[0197] In some implementations, different phalanges of various fingers correspond to different inputs. Micro-tapping inputs by the thumb above the various phalanges of various fingers (e.g., index, middle, ring, and opt...

Claims

1. A method for providing a computer-generated experience, comprising: In a computing system that communicates with a first display generation component, a second display generation component, and one or more input devices, wherein the first display generation component and the second display generation component are enclosed in a first housing: The first computer-generated environment is displayed via the first display generation component; When the first computer-generated environment is displayed via the first display generating component, based on the determination that the first computer-generated environment includes a representation of a physical environment, the representation of the physical environment showing a view of a first portion of the first physical environment in front of a user in a position viewing the first computer-generated environment via the first display generating component, and simultaneously displayed via a second display generating component enclosed in the same first housing as the first display generating component: A visual representation of a portion of the user in a position viewing the first computer-generated environment via the first display generating component, and One or more graphical elements provide visual indications of the content in the first computer-generated environment, wherein simultaneously displaying the visual representation of the user's portion and the one or more graphical elements includes: Change the visual representation of the user's portion to indicate changes in the user's appearance over a corresponding time period; and Changing one or more graphical elements that provide visual indications of the content in the first computer-generated environment to represent changes in the first computer-generated environment within the corresponding time period; and When the first computer-generated environment is displayed via the first display generating component, based on a determination that the first computer-generated environment does not include a representation of the physical environment showing a view of the first portion of the first physical environment in front of the user in a position where the user is viewing the first computer-generated environment via the first display generating component, the one or more graphic elements are displayed via a second display generating component enclosed in the same first housing as the first display generating component, without displaying the visual representation of the portion of the user in a position where the user is viewing the first computer-generated environment via the first display generating component.

2. The method of claim 1, wherein changing the visual representation of the user's portion to represent changes in the user's appearance during the corresponding time period comprises: Based on the movement of the user's portion in a first direction relative to the first display generating component, the visual representation of the user's portion displayed on the second display generating component is moved in the first direction; as well as Based on the movement of the user's portion in a second direction relative to the first display generating component, the second direction being within a threshold angle of a corresponding direction perpendicular to the first direction, the visual representation of the user's portion displayed on the second display generating component is moved in a third direction, the third direction being within the threshold angle of a corresponding direction opposite to the second direction.

3. The method of claim 1, wherein simultaneously displaying the visual representation of the user's portion and the one or more graphic elements comprises generating the visual representation of the user's portion based on predefined feature values ​​specified for a first display characteristic of the visual representation.

4. The method of claim 3, wherein the predefined feature value specified for the first display characteristic of the visual representation includes a predefined reduction in the opacity of the visual representation of the user's portion.

5. The method according to any one of claims 1 to 4, wherein simultaneously displaying the visual representation of the user's portion and the one or more graphic elements includes applying predefined visual effects to the visual representation of the user's portion.

6. The method according to any one of claims 1 to 4, wherein the simultaneous display of the visual representation of the user's portion and the one or more graphic elements via the second display generating component includes a diffused color layer displayed between the visual representation of the user's portion and the one or more graphic elements providing visual indication of the content of the first computer-generated environment.

7. The method according to any one of claims 1 to 4, wherein the visual representation of the user's portion and the one or more graphic elements simultaneously displayed via the second display generating component include the one or more graphic elements that display the visual representation of the user's portion at a first graphic resolution and display the visual indication of the content providing the first computer-generated environment at a second graphic resolution different from the first graphic resolution.

8. The method according to any one of claims 1 to 4, wherein simultaneously displaying the visual representation of the user's portion and the one or more graphical elements comprises: Based on the determination that a first change in the user's appearance has been detected, the visual representation of the user's portion is altered according to the first change in the user's appearance without altering the one or more graphic elements that provide the visual indication of the content in the first computer-generated environment.

9. The method according to any one of claims 1 to 4, wherein simultaneously displaying the visual representation of the user's portion and the one or more graphical elements comprises: Based on the determination that a second change in the content of the first computer-generated environment is detected while the user's appearance remains unchanged, at least one of the one or more graphic elements providing the visual indication of the content in the first computer-generated environment is modified according to the second change in the content of the first computer-generated environment without changing the visual representation of the user's portion.

10. The method according to any one of claims 1 to 4, wherein simultaneously displaying the visual representation of the user's portion and the one or more graphical elements comprises: A third change in the user's appearance is detected based on a fourth change in the content of the first computer-generated environment, and the visual representation of the user's portion is changed based on the third change in the user's appearance, and the one or more graphic elements providing the visual indication of the content of the first computer-generated environment are changed in conjunction with the fourth change in the content of the first computer-generated environment.

11. The method according to any one of claims 1 to 4, wherein: Simultaneously displaying the visual representation of the user's portion and the one or more graphic elements includes simultaneously displaying a modified version of the content and the visual representation of the user's portion, wherein the modified version of the content is generated using one or more image filters, the one or more image filters reducing the image fidelity of the image when applied to the image of the content; as well as Changing the one or more graphic elements that provide visual indications of the content in the first computer-generated environment to represent changes in the first computer-generated environment within the corresponding time period includes updating the modified version of the content by applying the one or more image filters to an image of the content when the content changes.

12. The method according to any one of claims 1 to 4, wherein the visual representation of the portion of the user includes an image of the user's eyes.

13. The method of any one of claims 1 to 4, wherein the visual representation of the user's portion comprises a virtual object generated based on one or more images of the user's eyes, wherein the appearance and movement of the virtual object correspond to the appearance and movement of the user's eyes captured by one or more sensors of the computing system.

14. The method according to any one of claims 1 to 4, wherein changing the visual representation of the user's portion to represent a change in the user's appearance during the corresponding time period comprises: Obtain eye-tracking data corresponding to the user; as well as The visual representation of the user's portion is updated based on the eye-tracking data corresponding to the user.

15. The method according to any one of claims 1 to 4, wherein: The first display generation component is a hardware component of the computing system that is visible to the user when the user is in the corresponding physical environment; and The second display generation component is a hardware component of the computing system that is visible to a corresponding person who is different from the user when the user is in the corresponding physical environment.

16. A non-transitory computer-readable storage medium storing executable instructions, said executable instructions, when executed by a computing system comprising one or more processors and in communication with a first display generation component and a second display generation component, wherein said first display generation component and second display generation component are enclosed in a first housing, such that said computing system performs the method according to any one of claims 1 to 15.

17. A computing system, comprising: One or more processors; First display generation component; A second display generating component, wherein the first display generating component and the second display generating component are enclosed in a first housing; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method according to any one of claims 1 to 15.

18. A computing system having one or more processors, a first display generation component, and a second display generation component, wherein the first display generation component and the second display generation component are enclosed in a first housing, the computing system comprising: Units for performing the method according to any one of claims 1 to 15.

Images