Devices, methods, and graphical user interfaces for providing computer-generated experiences

Abstract

A device, method, and graphical user interface for providing a computer-generated experience are provided. While displaying a first view of a three-dimensional environment that corresponds to a first viewpoint, a computer system detects a change of a grip of a user's hand to a first predefined grip in conjunction with detecting a gaze input directed to a respective location in the three-dimensional environment. If the respective location is a first location, the computer system replaces the first view with a second view of the three-dimensional environment that corresponds to a second viewpoint selected in accordance with a respective movement of the user's hand and the first location in the three-dimensional environment; and if the respective location is a second location, the computer system replaces the first view with a third view of the three-dimensional environment that corresponds to a third viewpoint selected in accordance with the respective movement of the user's hand and the second location in the three-dimensional environment.

Description

[0001] This application is a divisional application of the invention patent application filed on September 15, 2021, with application number 202180062620.6 and entitled "Apparatus, Method and Graphical User Interface for Providing Computer-Generated Experience".

[0002] Related patent applications

[0003] This application is a continuation-in-part of U.S. Patent Application 17 / 475,036, filed September 14, 2021, which claims priority to U.S. Provisional Patent Application 62 / 706,881, filed September 15, 2020, each of which is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure relates in its entirety to a computer system having one or more display generation components and one or more input devices (including, but not limited to, electronic devices that provide virtual reality and mixed reality experiences via one or more displays) for providing computer-generated experiences. Background Technology

[0005] In recent years, the development of computer systems for virtual reality and augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or enhance representations of the physical world. Example virtual reality environments include virtual elements with spatial relationships that simulate spatial relationships in a physical environment. Input devices for computer systems and other electronic computing devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays) are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include virtual 3D objects, digital images, videos, text, icons, control elements (such as buttons), and other graphics.

[0006] However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems where manipulating virtual objects is complex, tedious, and error-prone, impose a significant cognitive burden on users and detract from the user experience of virtual / augmented reality environments. Furthermore, these methods take longer than necessary, thus wasting energy. This latter consideration is particularly important in battery-powered devices. In addition, many systems providing virtual reality and / or mixed reality experiences use head-mounted displays that physically isolate the user's face from their surroundings, hindering social interaction and information exchange with the outside world when users engage in virtual reality and mixed reality experiences. Summary of the Invention

[0007] Therefore, computer systems with improved methods and interfaces are needed to provide users with computer-generated experiences, making user interaction with the computer system more efficient and intuitive. Such methods and interfaces optionally complement or replace conventional methods for providing users with computer-generated reality (CGR) experiences. These methods and interfaces create more efficient human-computer interfaces by helping users understand the relationship between the inputs provided and the device's responses to those inputs, thereby reducing the quantity, extent, and / or nature of user input. Such methods and interfaces also improve the user experience, for example, by reducing errors, interruptions, and time delays when users engage in virtual reality and / or mixed reality experiences provided by the computer system.

[0008] Specifically, when interacting with a 3D environment containing virtual objects and surfaces of different shapes, sizes, spatial extents, and positions, users may wish to navigate within the 3D environment relative to the virtual objects or space to achieve a desired viewing position, viewing angle, and / or interaction distance relative to the selected virtual object. In some scenarios, users may want to navigate towards the virtual object, while in others, they may want to bring the virtual object closer to the current viewpoint. The appropriate viewing distance may vary for different virtual objects, for example, to facilitate inspection, appreciation, manipulation, and / or other purposes corresponding to different virtual objects.

[0009] Therefore, in some embodiments, systems and methods for realizing movement within a three-dimensional environment are disclosed, wherein movement toward the selected destination or anchor location in the three-dimensional environment is performed in response to detecting user input selecting a destination or anchor location and in conjunction with physical movement of the user's hand, while the hand maintains a preset posture or grip (e.g., the formation of a preset posture or grip signals the user's intention to move within the three-dimensional environment, and movement of the user's hand in the preset posture or grip results in the movement toward the selected destination or anchor location). Movement toward the selected destination location is achieved by shifting the viewpoint of the current displayed view of the three-dimensional environment toward the selected destination or anchor location based on the movement of the user's hand in the physical environment. In some embodiments, instead of movement of the user's head or person as a whole, movement within the virtual three-dimensional environment is made faster, easier, and more efficient by using gaze input to select a destination or anchor location for a movement request and moving the viewpoint in response to movement of the user's hand (e.g., visually represented as a changing view of the three-dimensional environment), making movement within the virtual three-dimensional environment faster, easier, and more efficient. Large virtual distances can be covered rapidly in response to a single input or several inputs. Furthermore, the physical characteristics of the input allow users to experience virtual movement in a more intuitive and physical way, without requiring users to expend significant energy when providing input. Additionally, users can easily select a destination or anchoring location using gaze (e.g., by looking at or staring at a specific location in a 3D environment), and use the same physical movements of their hands to travel different distances in different directions within the 3D environment. In some implementations, by changing their hand to preset hand postures or grasping motion characteristics, users are given control over the speed and distance of movement.

[0010] Furthermore, in some implementations, the computer system eliminates ambiguity regarding user hand movement based on the type of grip or hand posture maintained during the user's hand movement. According to various implementations, the type of grip or hand posture is distinguished based on factors such as the position of fingers on the corresponding hand, the orientation of the corresponding hand, the number of hands forming a grip, the relative position and orientation of the hands forming a grip, or a combination of both or more of these. In some implementations, once a destination location or anchor location is selected, if a movable virtual object exists at the destination location or anchor location, and if the user's hand is in a first grip type during hand movement, the computer system recognizes the user's hand movement as a request to move the virtual object relative to the 3D environment (and the viewpoint relative to the user's virtual position or the current display view of the 3D environment); and if the user's hand is in a second grip type different from the first grip type, the computer system recognizes the user's hand movement as a request to move the viewpoint relative to the virtual object (and relative to the 3D environment). In some implementations, if the user's hand is in a third grip type, the computer system reorients the entire 3D environment relative to the anchor location while maintaining a fixed spatial relationship between the anchor location and the viewpoint. In some implementations, if no anchor point is selected, or if there is no movable object at the gaze location, the computer system reorients the entire 3D environment (including all virtual objects contained therein) relative to the user's virtual position or viewpoint. In some implementations, if the selected anchor point is a non-movable virtual object (e.g., a virtual mountain, virtual wall, etc.), the computer system moves and / or reorients the entire 3D environment relative to the user's virtual position or viewpoint. Using the nature of the selected virtual object or position, and the user's hand grip during hand movement, to determine whether to move a virtual object, the viewpoint, or the entire 3D environment reduces the amount of input that the user needs to remember and master, and makes interaction with the 3D environment more efficient and intuitive for the user.

[0011] Furthermore, in some implementations, when a navigation target for a movement request has been selected, the computer system does not simply respond to the movement request by moving the viewpoint to the selected navigation target. Instead, it performs a more detailed determination of where to move the viewpoint based on one or more characteristics of the navigation target in addition to its location (e.g., the center position, surface, or bounding box position of the navigation target). In some implementations, the characteristics of the navigation target include its size, spatial extent, visual content, type, purpose, available interaction modes (e.g., direct manipulation, remote manipulation, etc.). Selecting different distances between the destination target and the viewpoint position at the end of the movement based on the characteristics of the destination target helps to facilitate better observation and / or easier interaction with the destination target after the movement is completed.

[0012] According to some embodiments, a method is performed in a computer system including a display generation component and one or more input devices. The method includes: displaying a first view of at least a portion of a three-dimensional environment corresponding to a first viewpoint by the display generation component; while displaying the first view of at least that portion of the three-dimensional environment, detecting a change in the grip of a user's hand in conjunction with detecting gaze input pointing to a corresponding position in that portion of the three-dimensional environment, wherein the change in the grip of the user's hand results in a first predefined grip; detecting a corresponding movement of the user's hand while the user's hand maintains the first predefined grip; and in response to detecting the corresponding movement of the user's hand while the user's hand maintains the first predefined grip: based on determining that the corresponding position is a first position and the corresponding movement includes a first movement amount, [the method] displays at least [a certain value] of the three-dimensional environment. The first view corresponding to a first viewpoint of this part is replaced with a second view of at least that part of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint, wherein the second viewpoint is selected based on the corresponding movement of the user's hand and a first position in the three-dimensional environment; and the first view corresponding to the first viewpoint of at least that part of the three-dimensional environment is replaced with a third view corresponding to a third viewpoint of at least that part of the three-dimensional environment based on the corresponding movement of the user's hand and a second position in the three-dimensional environment, wherein the third viewpoint is different from the first viewpoint and the second viewpoint.

[0013] According to some embodiments, a method is performed in a computer system including a display generation component and one or more input devices. The method includes: displaying a view of at least a portion of a three-dimensional environment corresponding to a first viewpoint via the display generation component; while displaying the view of at least that portion of the three-dimensional environment corresponding to the first viewpoint, detecting corresponding user input, including detecting movement of one or both of the user's hands in the physical environment; and in response to detecting the corresponding user input: moving a virtual object in the three-dimensional environment relative to the three-dimensional environment based on the movement of one or both hands in the physical environment, according to determining that the corresponding user input meets a first criterion, wherein the first criterion includes a first criterion that is met when one or both of the user's hands maintain a first grasp while a first type of movement of one or both of the user's hands is detected; and replacing the view of at least that portion of the three-dimensional environment corresponding to the first viewpoint with a view of at least that portion of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint, according to determining that the corresponding user input meets a second criterion different from the first criterion, wherein the second criterion includes a second criterion that is met when one or both of the user's hands maintain a second grasp different from the first grasp while a second type of movement of one or both of the user's hands is detected.

[0014] According to some embodiments, a method is performed in a computer system including a display generation component and one or more input devices. The method includes: displaying a first view corresponding to a first viewpoint of at least a portion of a three-dimensional environment by the display generation component; while displaying the first viewpoint corresponding to the first viewpoint of at least that portion of the three-dimensional environment, detecting a set of one or more first user inputs specifying a navigation target located at a corresponding position in the three-dimensional environment and triggering a navigation operation toward the navigation target; and in response to detecting the set of one or more first user inputs, displaying an updated view of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint by the display generation component, wherein the second viewpoint is closer to the corresponding position of the navigation target than the first viewpoint but spaced apart from the corresponding position of the navigation target, and the second viewpoint is selected based on predefined characteristics of the navigation target and in conjunction with the corresponding position of the navigation target.

[0015] According to some embodiments, a computer system includes one or more display generating components (e.g., one or more displays, projectors, head-mounted displays, etc., enclosed in the same or different housings), one or more input devices (e.g., one or more cameras, touch-sensitive surfaces, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), optional one or more haptic output generators, one or more processors, and memory storing one or more programs; the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform operations of any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium has instructions stored therein that, when executed by a computer system having one or more display generating components, one or more input devices, and optional one or more haptic output generators, cause the device to perform or cause to perform operations of any of the methods described herein. According to some embodiments, a graphical user interface on a computer system having one or more display generating components, one or more input devices, optional one or more haptic output generators, memory, and one or more processors for executing one or more programs stored in memory includes one or more elements displayed in any of the methods described herein, which are updated in response to input, as described in any of the methods described herein. According to some embodiments, a computer system includes: one or more display generating components, one or more input devices, and optionally one or more haptic output generators; and means for performing or causing operations to be performed in any of the methods described herein. According to some embodiments, an information processing apparatus for use in a computer system having one or more display generating components, one or more input devices, and optionally one or more haptic output generators includes means for performing or causing operations to be performed in any of the methods described herein.

[0016] Therefore, improved methods and interfaces are provided for computer systems with one or more display generation components to deliver computer-generated experiences to users, making interaction with the computer system more efficient and intuitive for users. Improved methods and interfaces are also provided for these computer systems to deliver computer-generated experiences to users, thereby facilitating better social interaction, etiquette, and information exchange with the surrounding environment when users participate in various virtual reality and mixed reality experiences. Such methods and interfaces optionally complement or replace conventional methods for delivering computer-generated reality (CGR) experiences to users. These methods and interfaces reduce the quantity, extent, and / or nature of input from users by helping users understand the relationship between the input provided and the device's response to those inputs, thus creating a more efficient human-computer interface. Such methods and interfaces also improve the user experience, for example, by reducing errors, interruptions, and time delays caused by the lack of social cues and visual information for the user and others in the same physical environment when users participate in virtual reality and / or mixed reality experiences provided by the computer system.

[0017] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not exhaustive; in particular, many additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, description, and claims. Furthermore, it should be pointed out that the language used in this specification has been chosen in principle for readability and instruction purposes, and such choice may not be necessary to depict or define the subject matter of the invention. Attached Figure Description

[0018] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0019] Figure 1 This is a block diagram illustrating the operating environment of a computer system for providing a CGR experience according to some implementation schemes.

[0020] Figure 2 This is a block diagram illustrating a controller of a computer system configured to manage and coordinate the user's CGR experience according to some implementation schemes.

[0021] Figure 3 This is a block diagram illustrating a display generation component of a computer system configured to provide a CGR experience to a user, according to some embodiments.

[0022] Figure 4 This is a block diagram illustrating a hand tracking unit of a computer system configured to capture user gesture input according to some embodiments.

[0023] Figure 5 This is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input according to some embodiments.

[0024] Figure 6 This is a flowchart illustrating a gaze tracking pipeline with flash assistance according to some implementation schemes.

[0025] Figures 7A to 7D The diagram illustrates the selection of a navigation target according to some implementation schemes and navigation to the navigation target based on physical gestures.

[0026] Figures 7E to 7I The illustration shows how, according to some implementations, virtual objects can be selectively moved relative to a 3D environment (and viewpoint) or the viewpoint can be moved relative to a 3D environment (and all virtual objects within the 3D environment) based on the user's hand grip held during a user input sequence.

[0027] Figures 7J to 7O The diagram illustrates how, in response to a movement request, the viewpoint is moved toward a selected navigation target according to some implementations, wherein the viewpoint's position at the end of the movement is selected based on the location of the navigation target and one or more other characteristics of the navigation target.

[0028] Figure 8 This is a flowchart of method 8000 according to some implementation schemes, which is used to select a navigation target and navigate to the navigation target based on physical gestures.

[0029] Figure 9 This is a flowchart of method 9000 according to some implementation schemes, which is used to selectively move virtual objects relative to a three-dimensional environment (and viewpoint) or move the viewpoint relative to a three-dimensional environment (and all virtual objects within the three-dimensional environment) based on the user's hand grip held during a user input sequence.

[0030] Figure 10 This is a flowchart of method 10000 according to some implementation schemes, which is used to move a viewpoint toward a selected navigation target in response to a move request, wherein the position of the viewpoint at the end of the move is selected based on the location of the navigation target and one or more other features of the navigation target. Detailed Implementation

[0031] According to some implementations, this disclosure relates to a user interface for providing a computer-generated reality (CGR) experience to a user.

[0032] Figures 1 to 6 A description of an exemplary computer system for providing a CGR experience to a user is provided. Figures 7A to 7D The diagram illustrates the selection of a navigation target according to some implementation schemes and navigation to the navigation target based on physical gestures. Figures 7E to 7I The illustration shows how, according to some implementations, virtual objects can be selectively moved relative to a 3D environment (and viewpoint) or the viewpoint can be moved relative to a 3D environment (and all virtual objects within the 3D environment) based on the user's hand grip held during a user input sequence. Figures 7J to 7O The diagram illustrates how, in response to a movement request, the viewpoint is moved toward a selected navigation target according to some implementations, wherein the viewpoint's position at the end of the movement is selected based on the location of the navigation target and one or more other characteristics of the navigation target. Figure 8 This is a flowchart of a method according to some implementation schemes, which is used to select a navigation target and navigate to the navigation target based on physical gestures. Figure 9 This is a flowchart of a method according to some implementation schemes, which is used to selectively move virtual objects relative to a 3D environment (and viewpoint) or move the viewpoint relative to a 3D environment (and all virtual objects within the 3D environment) based on the user's hand grip held during a user input sequence. Figure 10 This is a flowchart of a method according to some implementation schemes for moving a viewpoint toward a selected navigation target in response to a move request, wherein the position of the viewpoint at the end of the move is selected based on the location of the navigation target and one or more other features of the navigation target. Figures 7A to 7O The user interface in the document is used to show... Figures 8 to 10 The process in.

[0033] In some implementation schemes, such as Figure 1As shown, a CGR experience is provided to the user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), one or more display generation components 120 (e.g., one or more head-mounted devices (HMDs) enclosed in the same housing and facing different directions or enclosed in separate housings, HMDs with internal and external displays, one or more displays, one or more projectors, one or more touchscreens, etc.), one or more input devices 125 (e.g., eye-tracking device 130, hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., speakers 160, haptic output generators 170, and other output devices 180), one or more sensors 190 (e.g., image sensors, light sensors, depth sensors, haptic sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, speed sensors, etc.), and optional one or more peripheral devices 195 (e.g., home appliances, wearable devices, etc.). In some implementations, one or more of the input device 125, output device 155, sensor 190, and peripheral device 195 are integrated with the display generation component 120 (e.g., in a head-mounted device (e.g., on the housing of an HMD or on the outward-facing display of an HMD) or in a handheld device).

[0034] In describing the CGR experience, various terms are used to distinguish several related but different environments that a user can sense and / or interact with (e.g., interacting with inputs detected by the computer system 101 that generates the CGR experience, causing the computer system to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computer system 101). The following is a subset of these terms:

[0035] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. Physical environments, such as physical parks, include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through senses such as sight, touch, hearing, taste, and smell.

[0036] Computer-Generated Reality: Conversely, a computer-generated reality (CGR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In CGR, a subset of a person's physical motion, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are modulated in a manner consistent with at least one physical law. For example, a CGR system can detect a person's head rotation and, in response, modulate the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the modulation of characteristics of virtual objects in the CGR environment can be done in response to a representation of physical motion (e.g., a voice command). A person can use any of their senses to sense and / or interact with CGR objects, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects that create (3D) or spatial audio environments, providing the perception of a point audio source in 3D space. For example, audio objects can enable audio transparency, which selectively introduces ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, people can sense and / or interact only with audio objects.

[0037] Examples of CGR include virtual reality and mixed reality.

[0038] Virtual Reality: A virtual reality (VR) environment is a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing human avatars are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through the simulation of a person's presence within the computer-generated environment and / or through the simulation of a subgroup of physical movements of a person within the computer-generated environment.

[0039] Mixed Reality: Compared to VR environments, which are designed to be entirely based on computer-generated sensory input, mixed reality (MR) environments refer to simulated environments designed to incorporate sensory input from the physical environment, or its representations, in addition to computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any state between, but not limited to, a purely physical environment as one end and a virtual reality environment as the other. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems used to present an MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical objects or their representations from the physical environment). For example, a system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0040] Examples of mixed reality environments include augmented reality and augmented virtual environments.

[0041] Augmented Reality (AR): An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation of the physical environment. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or semi-transparent display, allowing a person to perceive the virtual objects overlaid on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects overlaid on the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects onto the physical environment, such as as holograms or on physical surfaces, allowing a person to perceive the virtual objects superimposed on the physical environment. Augmented reality environments also refer to simulated environments in which the representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system can transform images from one or more sensors to apply a selected viewpoint (e.g., viewpoint) different from the viewpoint captured by the imaging sensor. Furthermore, the representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions of it, such that the modified portions are representative but not realistic versions of the original captured image. Still further, the representation of the physical environment can be transformed by graphically removing or blurring portions of it.

[0042] Augmented Virtual: An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. Sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park could have virtual trees and virtual buildings, but a person's face could be realistically reproduced from an image taken of a physical person. Similarly, virtual objects could adopt the shape or color of a physical object imaged by one or more imaging sensors. Furthermore, virtual objects could use shadows that correspond to the sun's position within the physical environment.

[0043] Hardware: Many different types of electronic systems enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may combine one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto a person's retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface. In some embodiments, controller 110 is configured to manage and coordinate the user's CGR experience. In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. References below. Figure 2The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device located locally or remotely relative to scene 105 (e.g., physical setup / environment). For example, the controller 110 is a local server located within scene 105. Alternatively, the controller 110 is a remote server (e.g., a cloud server, central server, etc.) located outside scene 105. In some embodiments, the controller 110 is communicatively coupled to display generation components 120 (e.g., one or more HMDs, displays, projectors, touchscreens, etc., enclosed in the same or different housings) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, controller 110 is included within the housing (e.g., physical enclosure) of display generation component 120 (e.g., HMD or portable electronic device including a display and one or more processors), one or more input devices in input device 125, one or more output devices in output device 155, one or more sensors in sensor 190 and / or one or more peripheral devices in peripheral device 195, or shares the same physical housing or support structure with one or more of the aforementioned devices.

[0044] In some embodiments, at least one of the display generation components 120 is configured to provide a CGR experience to a user (e.g., at least the visual components of the CGR experience). In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. The following is relative to... Figure 3 An example of display generation component 120 is described in more detail. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0045] According to some implementation schemes, when a user is virtually and / or physically present within scene 105, at least one of the display generation components 120 provides the user with a CGR experience.

[0046] In some embodiments, the display generating component is worn on a part of the user's body (e.g., on his / her head, his / her hand, etc.). Therefore, at least one of the display generating components 120 includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, at least one of the display generating components 120 displays the user's field of view (e.g., the field of view from the user's point of view). In some embodiments, at least one of the display generating components 120 is a handheld device (such as a smartphone or tablet) configured to present CGR content, and the user holds the device with the display facing the user's field of view and the camera facing scene 105. In some embodiments, the handheld device is optionally placed within a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, at least one of the display generating components 120 is a CGR chamber, housing, or room configured to present CGR content, wherein the user does not wear or hold the display generating component 120. Many user interfaces described with reference to one type of hardware (e.g., a handheld device or a tripod-mounted device) for displaying CGR content can be implemented on another type of hardware (e.g., an HMD or other wearable computing device) for displaying CGR content. For example, a user interface illustrating interaction with CGR content triggered by an interaction occurring in the space in front of a handheld device or tripod-mounted device can be similarly implemented using an HMD, where the interaction occurs in the space in front of the HMD and the response to the CGR content is displayed via the HMD. Similarly, a user interface illustrating interaction with CGR content triggered by movement of a handheld device or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)) can be similarly implemented using an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)).

[0047] Despite Figure 1 The relevant features of the operating environment 100 are shown in this disclosure, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure further relevant aspects of the exemplary embodiments disclosed herein.

[0048] Figure 2This is a block diagram of an example controller 110 according to some implementation schemes. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the implementation schemes disclosed herein. Therefore, as a non-limiting example, in some embodiments, controller 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing kernels, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0049] In some embodiments, one or more communication buses 204 include circuitry for interconnecting and communicating between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0050] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures or subsets thereof, including optional operating system 230 and CGR experience module 240.

[0051] Operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate single or multiple CGR experiences for one or more users (e.g., single CGR experiences for one or more users, or multiple CGR experiences for corresponding groups of one or more users). To this end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.

[0052] In some implementations, the data acquisition unit 242 is configured to acquire data from... Figure 1 The data acquisition unit 242 includes at least one or more display generation components in display generation component 120, and optionally acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.) from one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data acquisition unit 242 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0053] In some implementations, the tracking unit 244 is configured to map scene 105 and track at least one or more display generation components in display generation components 120 relative to the scene. Figure 1 The tracking unit 244 tracks the position / location of one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195 in scenario 105. For this purpose, in various embodiments, the tracking unit 244 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics. In some embodiments, the tracking unit 244 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more portions of the user's hand, and / or the position / location of one or more portions of the user's hand relative to... Figure 1 The movement of scene 105 relative to at least one of the display generating components 120 and / or relative to a coordinate system (defined relative to the user's hand). The following refers to... Figure 4 The hand tracking unit 244 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the position or movement of the user's gaze (or more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hand)) or relative to CGR content displayed via at least one of the display generation components 120. The following description is relative to... Figure 5 The eye-tracking unit 243 is described in more detail.

[0054] In some implementations, coordination unit 246 is configured to manage and coordinate the CGR experience presented to the user by at least one of the display generation components 120, and optionally by one or more of the output device 155 and / or peripheral device 195. For this purpose, in various implementations, coordination unit 246 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0055] In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, position data, etc.) to at least one of the display generation components 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data transmission unit 248 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0056] Although the data acquisition unit 242, the tracking unit 244 (e.g., including eye tracking unit 243 and hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 are shown residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including eye tracking unit 243 and hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 may reside in a separate computing device.

[0057] also, Figure 2 This is used more as a functional description of various features that can exist in a specific implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 2 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0058] Figure 3This is a block diagram illustrating an example of at least one display generation component 120 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the computer system (e.g., HMD) including the display generating component 120 also includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.) in the same housing, one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional internal and / or external image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0059] In some embodiments, one or more communication buses 304 include circuitry for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 306 include inertial measurement units (IMUs), accelerometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, haptic engines, and / or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0060] In some embodiments, one or more CGR displays 312 are configured to provide a user with a CGR experience and optional status information related to the CGR experience. In some embodiments, one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some embodiments, one or more CGR displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, an HMD includes a single CGR display. In another example, an HMD includes CGR displays for each of the user's eyes. In some embodiments, one or more CGR displays 312 are capable of displaying MR and VR content. In some embodiments, one or more CGR displays 312 are capable of displaying either MR or VR content. In some embodiments, the HMD includes one or more CGR displays facing the user's eyes when the HMD is placed on the user's head, and one or more CGR displays facing away from the user's eyes (e.g., towards the external environment). In some embodiments, the computer system is a CGR room or CGR enclosure, and the CGR room or CGR enclosure includes internally CGR displays that provide CGR content to the user inside the CGR room or enclosure, and optionally includes externally one or more peripheral displays that display status information related to the CGR content and the status of the user inside.

[0061] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand and optionally the user's arm (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward in order to acquire image data corresponding to the scene that the user would see in the absence of the display generation component 120 (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0062] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures, or subsets thereof, including optional operating system 330 and CGR rendering module 340.

[0063] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. To this end, in various embodiments, CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR mapping generation unit 346, a data transmission unit 348, and optionally other operational units for displaying status information related to the user and the CGR content.

[0064] In some implementations, the data acquisition unit 342 is configured to acquire data from at least... Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For the purposes described, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0065] In some embodiments, the CGR rendering unit 344 is configured to render CGR content and associated status information via one or more CGR displays 312. For the aforementioned purposes, in various embodiments, the CGR rendering unit 344 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0066] In some implementations, the CGR mapping generation unit 346 is configured to generate a CGR mapping based on media content data (e.g., a 3D mapping of a mixed reality scene or a physical environment in which computer-generated objects may be placed to generate a computer-generated reality mapping). For the aforementioned purposes, in various implementations, the CGR mapping generation unit 346 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0067] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For the stated purpose, in various embodiments, the data transmission unit 348 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0068] Although the data acquisition unit 342, CGR rendering unit 344, CGR mapping generation unit 346, and data transmission unit 348 are shown residing in a single device (e.g., Figure 1 The data acquisition unit 342, the CGR rendering unit 344, the CGR mapping generation unit 346, and the data transmission unit 348 are located on the display generation component 120, but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR rendering unit 344, the CGR mapping generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0069] also, Figure 3 This is used more as a functional description of various features that may exist in a particular implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separate. For example, Figure 3 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0070] Figure 4 This is a schematic illustration of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1 Controlled by hand tracking unit 244 Figure 2 To track the position / location of one or more parts of a user's hand, and / or the position / location of one or more parts of the user's hand relative to... Figure 1The movement is defined in scenario 105 (e.g., relative to a portion of the user's surrounding physical environment, relative to at least one of the display generating components 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system (defined relative to the user's hand)). In some embodiments, the hand tracking device 140 is part of at least one of the display generating components 120 (e.g., embedded in or attached to the same housing as the display generating component (e.g., in a head-mounted device)). In some embodiments, the hand tracking device 140 is separate from the display generating component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0071] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures at least three-dimensional scene information including the human user's hand 406. The image sensor 404 captures images of the hand at sufficient resolution to distinguish the fingers and their corresponding positions. The image sensor 404 typically captures images of other parts of the user's body, or possibly all parts of the body, and may have scaling capabilities or be a dedicated sensor with increased magnification to capture images of the hand at the desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of scene 105, or serves as the image sensor for capturing the physical environment of scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a way that uses the field of view of the image sensor 404 or a portion thereof to define an interaction space in which hand movements captured by the image sensor are considered input to the controller 110.

[0072] In some implementations, image sensor 404 outputs a sequence of frames containing 3D mapping data (and, in addition, possibly color image data) to controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided via an application programming interface (API) to an application running on the controller, which in turn drives display generation component 120. For example, a user can interact with software running on controller 110 by moving his hand 408 and changing his hand gestures.

[0073] In some embodiments, image sensor 404 projects a speckle pattern onto a scene including hand 406 and captures an image of the projected pattern. In some embodiments, controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) via triangulation based on the lateral offset of the specks in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. This method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from image sensor 404. In this disclosure, it is assumed that image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis such that the depth coordinates of points in the scene correspond to the z-component measured by the image sensor. Alternatively, hand tracking device 440 may use other 3D mapping methods, such as stereo imaging or time-of-flight measurement, based on single or multiple cameras or other types of sensors.

[0074] In some implementations, hand tracking device 140 captures and processes time-series depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on a processor in image sensor 404 and / or controller 110 processes the 3D mapping data to extract image block descriptors of the hand from these depth maps. The software may match these descriptors with image block descriptors stored in database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and fingertips.

[0075] The software can also analyze the trajectories of the hand and / or fingers across multiple frames in a sequence to identify gestures. The pose estimation function described herein can be alternated with motion tracking, such that patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to find pose changes occurring in the remaining frames. Pose, motion, and gesture information is provided to an application running on controller 110 via the aforementioned API. This application can, for example, move and modify the image presented on display generation unit 120 in response to the pose and / or gesture information, or perform other functions.

[0076] In some embodiments, the software may be downloaded to controller 110 electronically, for example, via a network, or alternatively, may be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, database 408 is also stored in memory associated with controller 110. Alternatively or additionally, some or all of the described functions of the computer may be implemented in dedicated hardware, such as custom or semi-custom integrated circuits or programmable digital signal processors (DSPs). Although in Figure 4The controller 110 is shown, but for example, as a separate unit from the image sensor 440, some or all of the controller's processing functions may be performed by a suitable microprocessor and software, or by dedicated circuitry within the housing of the hand tracking device 402, or by other devices associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing function of the image sensor 404 may also be integrated into a computer or other computerized device controlled by the sensor output.

[0077] Figure 4 It also includes a schematic diagram of a depth map 410 captured by image sensor 404 according to some embodiments. As described above, the depth map comprises a matrix of pixels with corresponding depth values. Pixel 412 corresponding to hand 406 has been segmented from the background and wrist in this map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (e.g., a measured z-distance from image sensor 404), where gray shadows become darker as depth increases. Controller 110 processes these depth values ​​to identify and segment components of the image that have human hand features (e.g., a group of adjacent pixels). These features may include, for example, overall size, shape, and frame-to-frame motion from the depth map sequence.

[0078] Figure 4 The diagram also schematically illustrates the hand skeleton 414 ultimately extracted by the controller 110 from the depth map 410 of the hand 406 according to some embodiments. Figure 4 In this configuration, the skeleton 414 is overlaid on the hand background 416, which has already been segmented from the original depth map. In some embodiments, key feature points of the hand, and optionally those on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, fingertips, the center of the palm, the end of the hand that connects to the wrist, etc.), are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points across multiple image frames to determine, according to some embodiments, the gesture performed by the hand or the current state of the hand.

[0079] Figure 5 An eye-tracking device 130 is shown. Figure 1 Exemplary implementations of ). In some implementations, the eye-tracking device 130 comprises an eye-tracking unit 243 ( Figure 2The eye-tracking device 130 is controlled to track the position and movement of the user's gaze relative to scene 105 or relative to CGR content displayed via at least one of the display generation components 120. In some embodiments, the eye-tracking device 130 is integrated with at least one of the display generation components 120. For example, in some embodiments, when the display generation component 120 is part of a head-mounted device (such as a headset, helmet, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating CGR content for the user to view and components for tracking the user's gaze relative to the CGR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, when the display generation component is provided by a handheld device or a CGR chamber, the eye-tracking device 130 is optionally a separate device from the handheld device or CGR chamber. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 is optionally used in conjunction with at least one display generating component that is also head-mounted or at least one display generating component that is not head-mounted. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally used in conjunction with a head-mounted display generating component. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally part of a non-head-mounted display generating component.

[0080] In some embodiments, at least one of the display generating components 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display frames including left and right images in front of the user's eyes, thereby providing the user with a 3D virtual view. For example, a head-mounted display generating component may include a left optical lens and a right optical lens (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, at least one of the display generating components 120 may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generating component may have a transparent or semi-transparent display on which virtual objects are displayed, allowing the user to view the physical environment directly through the transparent or semi-transparent display. In some embodiments, at least one of the display generating components 120 projects virtual objects onto the physical environment. The virtual objects may be projected, for example, onto a physical surface or as holograms, allowing an individual to observe the virtual objects superimposed on the physical environment using the system. In this case, separate display panels and image frames for the left and right eyes may not be necessary.

[0081] like Figure 5 As shown, in some embodiments, gaze tracking device 130 includes at least one eye-tracking camera (e.g., an infrared (IR) or near-infrared (NIR) camera) and an illumination source (e.g., an array or ring of IR or NIR light sources, such as LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be pointed at the user's eye to receive IR or NIR light reflected directly from the eye, or alternatively, it may be pointed at "hot" mirrors located between the user's eye and the display panel, which reflect the IR or NIR light from the eye back to the eye-tracking camera while allowing visible light to pass through. Gaze tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps), analyzes these images to generate gaze tracking information, and transmits the gaze tracking information to controller 110. In some embodiments, the user's two eyes are tracked separately using corresponding eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked using corresponding eye-tracking cameras and illumination sources.

[0082] In some implementations, a device-specific calibration procedure is used to calibrate the eye-tracking device 130 to determine parameters for the eye-tracking device in a specific operating environment 100, such as the 3D geometry and parameters of the LEDs, camera, thermal mirror (if present), eye lenses, and display. The device-specific calibration procedure can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration procedure can be automated or manual. According to some implementations, a user-specific calibration procedure may include estimations of eye parameters for a specific user, such as pupil position, foveal position, optical axis, visual axis, interocular distance, etc. According to some implementations, once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, a flash-assisted method can be used to process the images captured by the eye-tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0083] like Figure 5As shown, the eye-tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system. The gaze tracking system includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-infrared (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye-tracking camera 540 may be pointed toward a mirror 550 located between the user's eye 592 and a display 510 (e.g., the left or right display panel of a head-mounted display, the internal display of a head-mounted device, or the display of a handheld device, a projector, etc.). These mirrors reflect the IR or NIR light from the eye 592 while allowing visible light to pass through. Figure 5 (as shown in the top portion), or alternatively, it can be pointed towards the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion), Figure 5 (As shown in the bottom part).

[0084] In some implementations, controller 110 renders AR or VR frames 562 (e.g., left and right frames for the left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye-tracking camera 540 for various purposes, such as processing frame 562 for display. Controller 110 optionally estimates the user's gaze point on display 510 based on the gaze tracking input 542 obtained from eye-tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated based on gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0085] The following describes several possible use cases for the user's current gaze direction and is not intended to be limiting. As an example use case, controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, controller 110 may generate virtual content at a higher resolution in the concave region determined according to the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content in the view based at least partially on the user's current gaze direction. As yet another example, the controller may display specific virtual content in the view based at least partially on the user's current gaze direction. As another exemplary use case in an AR application, controller 110 may guide an external camera used to capture the physical environment for CGR experiences to focus in the determined direction. The external camera's autofocus mechanism may then focus on an object or surface in the environment that the user is currently looking at on display 510. As another exemplary use case, eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of eye lens 520 so that the virtual object the user is currently looking at has appropriate convergence / divergence to match the convergence of the user's eyes 592. The controller 110 can use gaze tracking information to guide the eye lens 520 to adjust its focus so that the nearby object that the user is looking at appears at the correct distance.

[0086] In some embodiments, the eye-tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lens 520), an eye-tracking camera (e.g., eye-tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) mounted in a wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eyes 592. In some embodiments, the light source may be arranged in a ring or circle around each lens in the head-mounted device, such as... Figure 5 As shown in the diagram. In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0087] In some embodiments, the display 510 emits light in the visible light range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. It should be noted that the position and angle of the eye-tracking camera 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, cameras 540 with a wider field of view (FOV) and cameras 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, cameras 540 operating at one wavelength (e.g., 850 nm) and cameras 540 operating at different wavelengths (e.g., 940 nm) may be used on each side of the user's face.

[0088] like Figure 5 The implementation of the gaze tracking system shown can be used, for example, in computer-generated reality (e.g., including virtual reality and / or mixed reality) applications to provide users with computer-generated reality (e.g., including virtual reality, augmented reality and / or augmented virtual) experiences.

[0089] Figure 6 A flash-assisted gaze tracking pipeline according to some embodiments is illustrated. In some embodiments, the gaze tracking pipeline uses a flash-assisted gaze tracking system (e.g., such as...) Figure 1 and Figure 5 The eye-tracking device 130 shown is used to implement this. The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and flash in the current frame. When not in tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and flash in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in tracking state.

[0090] like Figure 6 As shown, the gaze-tracking camera captures left and right images of the user's left and right eyes. The captured images are then fed into the gaze-tracking pipeline for processing to begin at 610. As indicated by the arrow returning to element 600, the gaze-tracking system can continue capturing images of the user's eyes, for example, at a rate of 60 to 120 frames per second. In some embodiments, each set of captured images can be fed into the pipeline for processing. However, in some embodiments or under certain conditions, not all captured frames are processed by the pipeline.

[0091] At 610, for the currently captured image, if the tracking state is yes, the method proceeds to element 640. At 610, if the tracking state is no, the image is analyzed to detect the user's pupil and flash, as indicated at 620. At 630, if the pupil and flash are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0092] At 640, if proceeding from element 410, the current frame is analyzed to track the pupil and flashes in part based on previous information from the previous frame. At 640, if proceeding from element 630, the tracking state is initialized based on the pupils and flashes detected in the current frame. The processing result at element 640 is checked to verify that the tracking or detection result is credible. For example, the result may be checked to determine whether a sufficient number of pupils and flashes were successfully tracked or detected in the current frame for performing gaze estimation. At 650, if the result is not credible, the tracking state is set to no, and the method returns to element 610 to process the next image of the user's eye. At 650, if the result is credible, the method proceeds to element 670. At 670, the tracking state is set to YES (if not already yes), and the pupil and flash information is passed to element 680 to estimate the user's gaze point.

[0093] Figure 6 This is intended as an example of an eye-tracking technology that can be used in a particular specific implementation. As those skilled in the art will recognize, in various implementations, other existing or future-developed eye-tracking technologies may be used in place of or in combination with the flash-assisted eye-tracking technology described herein in a computer system 101 for providing a CGR experience to a user.

[0094] In this disclosure, various input methods are described in relation to interaction with a computer system. When an example is provided using one input device or method, and another example is provided using another input device or method, it should be understood that each example is compatible with and optionally utilizes the input device or method described with respect to the other example. Similarly, various output methods are described in relation to interaction with a computer system. When an example is provided using one output device or method, and another example is provided using another output device or method, it should be understood that each example is compatible with and optionally utilizes the output device or method described with respect to the other example. Similarly, various methods are described in relation to interaction with a virtual or mixed reality environment via a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and optionally utilizes the methods described with respect to the other example. Therefore, this disclosure discloses embodiments that are combinations of features of a plurality of examples without exhaustively listing all features of the embodiments in the description of each exemplary embodiment.

[0095] User interface and related processes

[0096] Now turn attention to implementations of user interfaces (“UIs”) and associated processes that can be implemented on computer systems (such as portable multifunction devices or head-mounted devices) having one or more display generating components, one or more input devices, and (optionally) one or more cameras.

[0097] Figures 7A to 7D A computer system including at least a first display generation component (e.g., display 7100) according to some embodiments is shown. Figure 1 Computer system 101 or Figure 4 In a computer system 140, etc., the computer system displays computer-generated content to a user via a first display generation component (e.g., a display 7100). In some embodiments, such as Figure 7AAs shown, the first display generating component is a display (e.g., display 7100) placed in front of the user and not supported by or attached to the user's body. In some embodiments, the display generating component has a display side facing the user's face and eyes and displays computer-generated content. In some embodiments, the display generating component has a transparent or semi-transparent portion (e.g., the entire area of ​​the display side, a portion of the display side, etc.) through which the user on the display side facing the display generating component can see the physical environment behind the display generating component. In some embodiments, the display generating component displays a camera view of a portion of the physical environment that is blurred due to the presence of the display generating component, such that the user on the display side facing the display generating component can see the camera view of that portion of the physical environment, optionally together with virtual content overlaid on at least a portion of the camera view of that portion of the physical environment, a view that blocks at least that portion, and / or a display that replaces at least that portion. In some embodiments, the display generating component projects an image of the virtual content into the user's eyes, allowing the user to visually perceive the virtual content. In some embodiments, the display generating component projects an image of virtual content onto a physical surface, and the virtual content, along with the physical environment, is observed through a semi-transparent or transparent portion of the display generating component. In some embodiments, the display generating component is a head-mounted display attached to and / or supported by the user's head, face, ears, shoulders, and / or nose, with the display side of the head-mounted display facing the user's eyes and having a substantially fixed spatial relationship with the user's eyes. In some embodiments, the display generating component has a flat display side. In some embodiments, the display generating component has a curved display side that partially or completely surrounds the user's head or body. In some embodiments, the display generating component displays the complete virtual environment (e.g., a virtual game environment, a virtual office, etc.) without simultaneously displaying a representation of the physical environment (e.g., a see-through view of the user's physical environment (e.g., through a transparent portion of the display, or through a camera view), a recording of a camera view of the user's physical environment, etc.). In some embodiments, the display generating component displays virtual content along with a representation of the physical environment, wherein the virtual content is displayed at various locations relative to the representation of the physical environment, wherein each location has a corresponding physical location in the physical environment. In some implementations, spatial relationships in the physical environment are related to spatial relationships in the mixed reality and / or augmented reality environment displayed via display generation components, such that virtual objects appear to be positioned and / or moved relative to the physical environment in the mixed reality and / or augmented reality environment.While the accompanying drawings illustrate one type of display generation component as an example, it is important to emphasize that the techniques disclosed herein are applicable to other types of display generation components in various embodiments and may prove more advantageous for other types of display generation components (e.g., head-mounted displays, displays that obstruct the user's view of the physical environment, etc.).

[0098] Figures 7A to 7D The illustrations demonstrate how navigation or movement within a virtual 3D environment can be performed and visualized in response to a user request, according to some embodiments. In some embodiments, the virtual 3D environment occupies the entire field of view provided by the display generation components. In some embodiments, the virtual 3D environment occupies a portion of the field of view provided by the display generation components (e.g., a virtual entrance or window displayed in parallel with a representation of the physical environment (e.g., overlaying a portion of the representation of the physical environment, replacing the display of that portion, or obscuring the view of that portion)). Figures 7A to 7D Used to illustrate the process described below, including Figures 8 to 10 The process in.

[0099] like Figure 7A As shown, a first display generation component (e.g., display 7100) displays a virtual 3D environment (e.g., virtual 3D environment 7102) at different locations within the virtual 3D environment, comprising various virtual objects (e.g., a virtual tree with a virtual treetop 7120 connected to a virtual trunk 7128, a virtual hill 7108, etc.) and / or surfaces (e.g., surfaces of virtual objects, virtual walls, virtual ground, etc.). For example, in Figure 7AIn the virtual 3D environment, the virtual treetop 7120 has a first spatial relationship with the virtual tree trunk 7218 (e.g., the virtual treetop 7120 is positioned above the virtual tree trunk 7128, with its bottom surface attached to the top of the virtual tree trunk 7128). The virtual tree has a second spatial relationship with the virtual hill 7108 (e.g., the bottom of the virtual tree trunk 7128 is attached to the sloping front surface of the virtual hill 7108). The virtual tree has a third spatial relationship with a first viewpoint corresponding to the current view 7205 of the virtual 3D environment. The virtual hill 7108 has a fourth spatial relationship with the first viewpoint corresponding to the current view 7205 of the virtual 3D environment. For example, the virtual tree is located at a first distance (e.g., 30 meters) from the first viewpoint corresponding to the current view 7205 of the virtual 3D environment 7102; and, compared to the virtual tree, the virtual hill 7108 spans a range that is closer to the first viewpoint in terms of height and distance at the bottom of the hill and farther away and lower in terms of height at the top of the hill. In some embodiments, the viewpoint corresponding to the current display view of the three-dimensional environment refers to a virtual position, advantageous position, and / or viewing angle in the virtual three-dimensional environment from which a virtual observer (e.g., whose eyes or head are in the virtual position, standing in the virtual position, sitting in the virtual position, etc.) will see the current display view. In some embodiments, the viewpoint corresponding to the current display view of the virtual three-dimensional environment moves in the virtual three-dimensional environment according to the movement (e.g., rotation and / or translation, etc.) of the head of a user at a position observing content displayed via the first display generating component (e.g., head movement relative to the torso, head movement due to torso movement, etc.). In some implementations, the position of a user or a part thereof in the physical environment has a corresponding position in the virtual 3D environment (e.g., a virtual position corresponding to the viewpoint of the current display view in the virtual 3D environment), and movement (e.g., rotation and / or translation, etc.) of the user as a whole or a part thereof (e.g., the user's head and / or torso) in the physical environment optionally results in a corresponding movement (e.g., rotation and / or translation, etc.) of the viewpoint of the current display view in the virtual 3D environment. In some implementations, the correspondence between the movement of the user as a whole or the movement of the user's head in the physical environment and the movement of the viewpoint allows users to experience spatial relationships in the virtual 3D environment in a more physical and realistic way. However, in many cases, physically moving back and forth in the physical environment is time-consuming, cumbersome, and difficult for some users, thus requiring a more efficient and lightweight way to navigate within a virtual 3D environment and experience spatial relationships within it.While input and / or navigation from one scene to another can be instantaneous, discrete, and abrupt in a virtual environment, such abrupt input and transitions, completely detached from the user's physical experience, can confuse the user, cause user errors, diminish the immersive experience offered by the virtual 3D environment, and fail to fully utilize the richness and complexity of the user's control over their physical movement. As disclosed herein, in some embodiments, user physical input (e.g., hand gestures, finger gestures, arm movements, wrist movements, gaze input, various combinations thereof, etc.), in addition to physical movement of the user's head or torso in the physical environment, is recognized as a request for movement in the 3D environment (e.g., moving the viewpoint (e.g., presenting a portion of the 3D environment to the user's virtual location or vantage point)) and causes different views of the 3D environment to be displayed to the user (e.g., changing the view due to movement of the viewpoint relative to the virtual 3D environment (e.g., moving the viewpoint closer to a virtual object, away from a virtual object, around a virtual object, etc.)).

[0100] In some implementation schemes, such as Figure 7A As shown, the first display generating component is a display placed in front of the user and not supported by or attached to the user's body. In some embodiments, the display is a head-mounted display having a display side facing the user's face and eyes, and not moving relative to the user's face or eyes as the user moves their head or body in the physical environment. In some embodiments, when the head-mounted display is used as the display generating component, the virtual position of the viewpoint of the current display view of the physical environment corresponds (e.g., having a preset spatial relationship, having a constant spatial relationship, overlapping with, near, etc.) to the virtual position of the user's eyes or head in the virtual three-dimensional environment. Figures 7A to 7D In the examples shown, according to some embodiments, the user as a whole is stationary relative to the physical environment. In some embodiments, the user as a whole may move within the physical environment, but the viewpoint is not updated based on the user's movement within the physical environment. In some embodiments, the user as a whole or the user's head may move within the physical environment, resulting in a movement of the viewpoint that is independent of and supplements to a movement request made using the movement of the user's hand within the physical environment (e.g., in conjunction with the user's gaze, and when in a first preset hand grip).

[0101] In some implementation schemes, such as Figure 7AAs shown, the computer system detects gaze input (e.g., gaze input 7206) directed at a corresponding portion of the virtual 3D environment. In some embodiments, as the user's eyes move around on the display side facing the first display generating component, the computer system tracks the movement of the user's eyes and determines the position of the user's gaze and focus in the 3D environment. For example, in some embodiments, gaze input is detected based on at least a threshold time amount within a threshold region determining that the user's gaze and focus have been positioned at a first location in the 3D environment; and a virtual object present at the first location is optionally identified as the target of the gaze input. In some embodiments, the computer system displays a visual marker (e.g., visual marker 7122) to indicate the current position of the gaze input (e.g., gaze input 7206). In some embodiments, the visual marker is displayed based on determining that the target of the gaze input has been identified at the location of the gaze input. In some embodiments, the visual marker is displayed based on determining that the gaze input has met preset criteria (e.g., remaining within a threshold region of the virtual location for at least a threshold time amount (e.g., a threshold time amount for detecting the gaze input (e.g., a gaze detection threshold), another threshold time amount after the gaze input is detected (e.g., a dwell time threshold)), etc.). In some implementations, visual markers are displayed solely to indicate movement of the user's gaze and / or focus, and do not necessarily imply that the user's attention is focused on the location of the visual markers. In some implementations, if the visual markers are used solely to indicate movement of the gaze or focus, additional visual feedback is provided when gaze input is detected and / or when a target object of the gaze input is detected (e.g., based on a single gaze input, or based on gaze input detected in combination with another user input, etc.). Figure 7A In this example, a visual marker 7122 is displayed at a location on the virtual treetop 7120 in the first view 7205 of the virtual 3D environment 7102 to indicate that the user is currently viewing the virtual treetop 7120 displayed via a first display generation component (e.g., display 7100). In this example, gaze input 7206 on the virtual treetop 7120 selects the virtual treetop 7120 as the target in conjunction with another user input (e.g., voice input, gesture input, touch input, etc.) detected by gaze input 7206.

[0102] exist Figure 7AIn the three-dimensional environment 7102, a representation (e.g., representation 7202') of the user's hand (e.g., hand 7202) is shown in view 7205. In some embodiments, the representation of the user's hand is a camera view of the hand held in front of the user's body. For example, in some embodiments, the user's hand is identified and segmented from the camera view and then overlaid or displayed in the three-dimensional environment at a location corresponding to the user's virtual position in the three-dimensional environment (e.g., optionally determining the appropriate virtual position of the hand based on the virtual position of the viewpoint of the current display view). In some embodiments, the representation of the user's hand is a stylized graphical representation of the user's hand, which optionally moves and / or changes to show the movement and / or changing posture of the user's hand in the physical environment. In some embodiments, the representation of the user's hand is not displayed in the current display view of the three-dimensional environment. For example, in some embodiments, the user's hand is outside the user's field of view provided by the first display generating component. In some implementations, the user's hand is positioned in front of the user, but this position would otherwise be within the user's field of view if a first display generating component were not present in front of the user (e.g., the view of the user's hand is obstructed by the first display generating component, and the representation of the user's hand is not displayed via the first display generating component). In other words, in some implementations, the user's hand performs gestures or movements in the physical environment, which are captured as input resulting in changes in the 3D environment, even if the gestures and movements of the user's hand are not displayed to the user via a representation of the user's hand in the view of the 3D environment. In some implementations, instead of a single hand, both of the user's hands are used to provide the required user input for requesting movement within the virtual 3D environment, and optionally, both hands are visually represented in the view of the virtual 3D environment.

[0103] exist Figure 7A After Figure 7BIn the system, when gaze input 7206 is detected at a location on a virtual treetop 7120 in the first view 7205 of the 3D environment 7102, the computer system detects movement of the user's hand 7202 in the physical environment (e.g., user hand movement corresponding to a change in hand grip). The movement of the user's hand is determined to correspond to movement of the fingers and / or wrist to form a first preset hand posture (e.g., forming a fist, fully or partially closing the hand, with multiple fingers curled toward the palm and eventually resting on the palm, pinching multiple fingers together, etc.). In some embodiments, the first preset hand posture corresponds to a grasp, grip, hold, and / or multi-finger pinching, etc. In some embodiments, the first preset hand posture requires the hand to have a first preset orientation relative to the user (e.g., palm facing the user's face, facing upwards and / or sideways, etc.). In some embodiments, the movement of the user's hand includes movement of both hands toward each other to form a first preset hand posture or grasp (e.g., fingers of both hands locking together, one hand wrapping around the other in a closed fist, both hands clasped together, etc.). As used herein, the required hand posture optionally includes a required hand grip, a required hand orientation, a required relative spatial relationship between different parts of the hand, a required relative position of two hands or different parts of two hands, and / or a combination of both or more of the above. In some embodiments, the criteria for requesting movement include the required hand grip, without additional requirements on the number of hands forming the required hand grip, the hand orientation, and / or the hand position. In some embodiments, the criteria for requesting movement include the required hand grip and the number of hands forming the required hand grip, the hand orientation, and / or the hand position. In some embodiments, the computer system detects gaze input 7206 on a first part of the virtual 3D environment (e.g., virtual treetop 7120) in conjunction with (e.g., concurrently, within its threshold time window, etc.) movement of the user's hand to form a first preset hand posture or grip; and in response, the computer system selects the virtual treetop 7120 or the entire virtual tree (e.g., including the virtual treetop 7120 and the virtual trunk 7128) as the target location or anchor location for navigation. In some implementations, the computer system displays visual feedback 7124 (e.g., highlighting, outlining, visual differentiation, etc.) to indicate the selection of a target location or anchoring location within the three-dimensional environment for movement (e.g., highlighting the entire virtual tree with an outline or spotlight, darkening the area around the virtual tree, etc.). In some implementations, in response to detecting that gaze input has moved to and / or stopped on a location or object, while the user's hand has formed and maintained a first preset hand posture or grip, the computer system selects that location or object as the target location or anchoring location for movement.

[0104] like Figure 7B After Figure 7C and Figure 7D As shown, when the user's hand maintains a first predefined hand posture or grip (e.g., grasping, pinching, two-handed grip, etc.), the computer system detects movement of the user's hand toward the user (e.g., movement 7130, movement 7132, etc.) (e.g., translation of the user's hand toward the user's body (e.g., due to movement of the user's arm attached to the hand around the elbow or shoulder, rotational movement around the wrist attached to the hand, or both)). Optionally, the start of movement of the user's hand in the first predefined hand posture or grip is detected while gaze input 7206 is held at a selected target position or anchor position for the movement. Figure 7C In response to detecting movement of the user's hand while the user's hand maintains a first predefined hand posture or grip, and based on determining that the selected target position or anchor position for the movement is a first position (e.g., a virtual tree) and the corresponding movement includes (or is) a first amount of movement (e.g., a first portion of the total range of movement the hand can make in this manner 7130 (e.g., rotation around the wrist, rotation around the elbow, rotation around the shoulder, etc.)), the computer system replaces a first view 7205 of the virtual 3D environment corresponding to a first viewpoint (e.g., a viewpoint corresponding to a first virtual position in the virtual 3D environment) with a second view 7207 of the virtual 3D environment corresponding to a second viewpoint (e.g., a viewpoint corresponding to a second virtual position in the 3D environment different from the first virtual position). Figures 7B to 7CAs shown, a second viewpoint is selected based on the user's hand movement and the target or anchoring position selected by gaze input 7206. As a result of the movement of the viewpoint of the current displayed view in the 3D environment (e.g., from a first virtual position to a second virtual position), the virtual tree appears closer to the user, as if the user has moved up the virtual hill 7108 and reached a vantage point in the 3D environment that is higher and closer to the virtual tree. In some embodiments, similar hand movements can be repeated to move the viewpoint closer to the selected target or anchoring position as long as the user's hand maintains a first predefined hand posture or grip. In some embodiments, the movement of the viewpoint in the virtual 3D environment simulates or mimics the movement of the user's virtual position caused by the user's hand gripping and pulling an invisible rubber band or taut rope around a virtual pulley anchored at the selected target or anchoring position in the virtual 3D environment. In some implementations, before repeated movements toward the user's body cause further movement of the viewpoint in the virtual 3D environment, the user's hand relaxes and reforms into a first predefined hand posture or grip and / or the entire hand returns to a neutral starting position (e.g., the arm or wrist extends back to a neutral, relaxed position in front of the user). In some implementations, gaze input is not required to remain at the selected target or anchor position during subsequent repetitions of the hand movement, provided that the movement is repeated at a sufficiently high frequency after the initial hand movement (e.g., each repetition falls within a threshold time window of the previous repetition, there are more than a preset number of repetitions within a preset time window, etc.) to indicate that it is part of a continuous sequence of inputs for the same purpose (e.g., navigation toward the selected target or anchor position, navigation in the same direction, etc.). In some implementations, visual feedback indicating the selected target or anchor position (e.g., visual feedback 7124) is maintained, provided that the time window for repeated hand movements with the first predefined hand posture or grip causing movement toward the selected target or anchor position remains open.

[0105] Figure 7D exist Figure 7C Subsequently, the continuation of the user's hand movement is shown (e.g., movement toward the user's torso or head). In response to detecting the detection of continued user hand movement (e.g., representing a large portion of the entire range of hand movement 7132 in this manner), the viewpoint of the current display view of the 3D environment (e.g., Figure 7DAs the viewpoint (7210) moves closer to the selected target or anchor position (e.g., from a second virtual position to a third virtual position closer to the selected target or anchor position), the virtual tree appears directly in front of the user, as if the user had moved up the virtual hill and reached a vantage point directly in front of the virtual tree. In some implementations, the computer system shifts the viewpoint to the right to the target or anchor position, and the user observes the virtual treetop from its surface. In some implementations, the computer system selects a suitable viewing position and distance for observing the virtual tree that is not directly on its surface (e.g., below the treetop and near the trunk, a few feet above the treetop in the air, etc.). Further details on selecting a suitable viewing position for a virtual object based on its properties are discussed later. Figures 7J to 7O and Figure 10 Describe it.

[0106] In some implementations, the movement of the user's hand does not need to be directed towards the user to trigger movement within the 3D environment, and can occur in other directions relative to the user. For example, in some implementations, the user's hand moves to the left or right of the user or upwards or downwards relative to the user's head while in a first preset hand posture or grasp, which results in a corresponding movement of the viewpoint in the 3D environment. More specifically, in some implementations, when the user's hand moves to the right of the user's body, the viewpoint moves to the left relative to the selected anchor position in the virtual 3D environment; and when the user's hand moves to the left of the user's body, the viewpoint moves to the right relative to the selected anchor position in the virtual 3D environment. In another example, when the user's hand moves downwards relative to the user's body, the viewpoint moves upwards relative to the selected anchor position in the virtual 3D environment; and when the user's hand moves upwards relative to the user's body, the viewpoint moves downwards relative to the selected anchor position in the virtual 3D environment. In yet another example, when the user's hand moves clockwise, the viewpoint rotates counterclockwise relative to the selected anchor position; and when the user's hand moves counterclockwise, the viewpoint rotates clockwise relative to the selected anchor position. In some implementations, the direction of viewpoint movement is reversed relative to the aforementioned directions. More details regarding how users can move the viewpoint of the currently displayed view of the 3D environment relative to the 3D environment are available. Figures 7E to 7I The description includes a preset hand gesture and / or grip used to determine whether to move the viewpoint relative to the 3D environment (and relative to the virtual objects contained therein) or to move the virtual objects relative to the 3D environment (and relative to the viewpoint).

[0107] In some implementations, if a user's hand, forming a first predefined hand posture or grip, is detected as gazing input at another location (e.g., virtual mountaintop 7108), then that other location is selected as the target or anchoring location for movement. For example, based on the determination that the selected target or anchoring location is virtual mountaintop 7180, for the same amount of hand movement while the user's hand maintains the first predefined hand posture or grip, the computer system replaces the first view 7205 with a different viewpoint corresponding to the viewpoint selected between the first viewpoint corresponding to the first viewpoint 7205 and the virtual mountaintop 7108 (e.g., a virtual location higher than the virtual tree and farther from the first viewpoint). For example, for the movement 7130 of the user's hand, the computer system will display a viewpoint that is higher than the first viewpoint. Figure 7C The view shown is closer to the field of view and closer to the virtual tree on the right side of the field of view; and for the movement of the user's hand 7132, the computer system will show the view from the top of the virtual hill, while the virtual tree does not exist in the field of view (for example, the virtual tree will be located behind the virtual position of the viewpoint at this time).

[0108] In some implementation schemes, such as Figures 7B to 7D As shown, in response to a user’s hand moving in a first predefined hand gesture or grip, the computer system maintains the display of visual feedback (e.g., visual feedback 7124) indicating the currently selected target location or anchor location during the viewpoint movement (e.g., applying visual effects that visually emphasize the selected target location or anchor location for the movement (e.g., highlighting or outlining around a virtual tree)).

[0109] In some implementations, before initiating hand movement in a first predefined pose or grip, the user has the option to select different target or anchoring positions by moving gaze input to different locations in the three-dimensional environment. For example, a first target or anchoring position can be selected before the user's hand forms the first predefined pose or grip; and after the user's hand forms the first predefined pose or grip, the user moves their gaze to another position while maintaining the first predefined pose or grip; thus, a new target or anchoring position is selected, and the visual effects applied to the first target or anchoring position cease to be displayed at the first target or anchoring position but are applied to the new target or anchoring position. In some implementations, the target or anchoring position is selected solely by gaze input (e.g., before forming the first predefined pose or grip), and as the user moves their gaze from one position to another, the currently selected target or anchoring position changes with the movement of the gaze input, and the visual effects also move with the movement of the gaze input.

[0110] In some implementations, the selection of a target or anchoring position based on gaze input can be further refined by additional, finer-grained input, such as movement of the user's fingers on a touch-sensitive surface, in the air, or relative to other parts of the user's hand. In some implementations, based on the position of the gaze input (e.g., after selecting a target or anchoring position based on gaze input, before detecting a change in the user's hand grip, and / or when the user's hand maintains a first predefined hand grip, etc.), the computer system detects gestures involving predefined movements of a first portion of the user's hand relative to a second portion of the user's hand (e.g., the user's thumb rubbing against the user's index finger, moving one or more fingers, etc., while maintaining the first predefined hand posture or grip). In response to the detection of a gesture, the computing system moves the selected target or anchoring position by a small preset amount (e.g., relative to the gaze input position) and also offsets the corresponding application position of the visual effect. In some implementations, the direction and amount of movement of the application position of the visual effect are determined based on the direction, speed, and / or amplitude of the gesture. For example, before or while maintaining a predefined hand gesture or grip, the same hand can be used to perform one or more fine-tuning gestures to adjust the target location selected by the gaze input from the top of the virtual treetop 7120 to the middle of the virtual treetop 7120. Then, hand movements in the predefined hand gesture or grip are used to move the viewpoint to a virtual location of a preset viewpoint associated with the target location (e.g., the target location itself, an appropriate viewpoint for observing objects at the target location, etc.). For example, when the gaze input is on the middle portion of the virtual treetop 7120, a spotlight is displayed across the entire tree, and the viewpoint corresponding to the target location (e.g., having the full range of hand movement while maintaining the first predefined hand gesture or grip) is in front of the virtual tree (e.g., below the virtual treetop 7120 and in front of the trunk 7128, such as...). Figure 7D (As shown). If the user, while maintaining a first predefined hand posture or grip (e.g., full-hand grip, two-hand grip, etc.), lightly sweeps their thumb forward (e.g., away from the user) relative to the side of the index finger of the same hand, the spotlight moves to the top of the virtual treetop 7120, and the viewpoint corresponding to the target location (e.g., with the full range of hand movement while maintaining the first predefined hand posture or grip) will be above the top of the virtual tree (e.g., as if the user were standing on the top of the virtual tree observing the 3D environment).

[0111] In some implementations, detecting a user’s hand moving to a first predefined hand posture or grasp includes detecting a hand or one or more fingers of a hand (e.g., index finger, index and middle fingers together, etc.) moving to contact and maintain contact with a predefined portion of another hand (e.g., hand in a closed fist, palm of an open hand, closed wrist, open wrist, etc.).

[0112] In some embodiments, the amount of movement through which the viewpoint travels is determined based on the amount of wrist rotation relative to the maximum range of wrist rotation typical of a human user (e.g., in the case of performing hand movements by wrist rotation). In some embodiments, lateral movement of the hand due to movement of the user's arm or person is not considered when determining the amount of movement resulting from the corresponding movement of the hand. In some embodiments, rotational movement of the hand around the elbow due to movement of the user's arm is not considered when determining the amount of movement resulting from the corresponding movement of the hand. In some embodiments, the amount of movement through which the viewpoint travels is determined based on the amount of elbow rotation relative to the full range of elbow rotation typical of a human user (e.g., in the case of performing hand movements by elbow rotation with a stiff wrist). In some embodiments, the amount of movement through which the viewpoint travels is determined based on the amount of shoulder rotation relative to the full range of shoulder rotation typical of a human user (e.g., in the case of performing hand movements by shoulder rotation with a stiff wrist and elbow).

[0113] In some embodiments, the movement of the viewpoint is triggered in response to a tapping input (e.g., a tapping input performed by the thumb of hand 7202 when hand 7202 is in a closed fist, a tapping input performed by a second hand when the first hand is in a closed fist, etc.). For example, in some embodiments, the first predefined posture or grip is a closed hand grip with a movable thumb, and the movement of the hand that causes the movement includes a tapping movement of the thumb on the surface of the closed hand (e.g., on the side of the index finger of the same hand, on the back side of the other fingers of the same hand, etc.), while the whole hand does not move in the closed hand grip.

[0114] In some embodiments, the movement of the viewpoint is triggered in response to a swipe input (e.g., a swipe performed by the thumb of hand 7202 when hand 7202 is in a grasping posture, a swipe performed by a second hand when the first hand is in a closed posture, etc.). For example, in some embodiments, the first predefined posture or grasp is a closed-hand grasp with a movable thumb, and the movement of the hand causing the movement includes a swipe movement of the thumb on the closed hand (e.g., on the side of the index finger of the same hand, on the back of the other fingers of the same hand, etc.), while the entire hand does not move in the closed-hand grasp. In some embodiments, the swipe direction determines the relative direction of movement between the viewpoint and the anchor position. For example, a leftward sweep of the thumb on a closed hand causes the viewpoint to move to the left (or to the right in some embodiments) relative to the anchor position; a forward sweep of the thumb on a closed hand causes the viewpoint to move forward toward (or backward away from) the anchor position; a rightward sweep of the thumb on a closed hand causes the viewpoint to move to the right (or to the left in some embodiments) relative to the anchor position; and a backward sweep of the thumb on a closed hand causes the viewpoint to move backward away from (or forward toward) the anchor position in some embodiments, and so on.

[0115] In some implementations, when the hand is in a first predefined hand posture or grasp (e.g., hand 7202 is in a grasping posture) and the hand is released (e.g., relaxing the wrist, extending the elbow, releasing the first predefined hand posture, etc.), the movement of the viewpoint is triggered in response to the pullback of the hand (e.g., the hand rotates around the wrist of the same hand, the arm of the same hand rotates around the elbow, etc.).

[0116] In some embodiments, instead of controlling the amount of movement performed by the viewpoint based on the amount of movement performed by the user's hand while maintaining a first predefined hand posture or grip, or as a supplement to it, other movement characteristics of the viewpoint are optionally controlled by movement characteristics of the hand. For example, in some embodiments, the rate of viewpoint movement is determined based on the amount of wrist or elbow rotation performed to cause hand movement. In some embodiments, features (e.g., direction, speed, etc.) of other gestures (e.g., tapping, swiping, etc.) used to cause viewpoint movement are used to control the movement characteristics (e.g., direction, speed, etc.) of the viewpoint. In some embodiments, the computer system displays an intermediate view sequence corresponding to a sequence of intermediate viewpoints between a starting viewpoint (e.g., a viewpoint corresponding to an initial viewpoint before the hand movement begins with the first predefined hand posture or grip) and a final viewpoint (e.g., a viewpoint closest to the selected target location or anchoring location, a viewpoint at the end of the movement, etc.), wherein the intermediate viewpoint sequence corresponds to the changing viewing angles seen through the three-dimensional environment as the user's hand moves from a virtual position according to the movement performed by the user's hand in the first predefined posture or grip. In some implementations, the movement characteristics of the virtual location (e.g., direction, speed, distance, etc.) are based on the user's hand movement characteristics (e.g., the rotation speed, amount of rotation, and combinations thereof around the wrist, around the elbow, etc.; tap speed, tap duration, swipe speed, swipe distance, etc.).

[0117] In some implementations, as the displayed view of the 3D environment changes with the movement of the viewpoint, the outer portion of the view is visually faded (e.g., blurred, displayed at reduced resolution or sharpness, etc.) to simulate motion blur. In some implementations, visually fading the outer portion of the current displayed view of the 3D environment during viewpoint movement helps prevent and alleviate motion sickness experienced by users while moving within a computer-generated environment.

[0118] In some implementations, the computer system determines whether to move the viewpoint of the current display view of the 3D environment or a virtual object within the 3D environment based on the nature of the object present at the gaze input location. For example, if a movable virtual object (e.g., a virtual application window, a virtual ball, a user interface object, etc.) exists at the gaze input location, that virtual object is selected as the target object and moves toward the virtual location of the viewpoint in response to a movement of the user's hand in a first predefined hand gesture or grip. If no movable object exists at the gaze input location (e.g., the location has no virtual objects, or the virtual objects present at the location mimic things that are normally immovable in the physical environment (e.g., virtual trees, virtual houses, virtual walls, virtual hills, virtual clouds, etc.)), the computer system interprets the movement of the user's hand in the first predefined hand gesture or grip as a request to move the viewpoint (e.g., to perform movement in the 3D environment).

[0119] Figures 7E to 7I The illustration shows an interaction with a three-dimensional environment according to some embodiments, which results in the movement of a viewpoint relative to a virtual object or a virtual three-dimensional environment (e.g., simulating the movement of a user relative to a virtual object or a virtual three-dimensional environment), or results in the movement of a virtual object relative to a viewpoint or a three-dimensional environment.

[0120] Figures 7E to 7I A computer system including at least a first display generation component (e.g., display 7100) according to some embodiments is shown. Figure 1 Computer system 101 or Figure 4 In a computer system 140, etc., the computer system displays computer-generated content to a user via a first display generation component (e.g., a display 7100). In some embodiments, such as Figure 7EAs shown, the first display generating component is a display (e.g., display 7100) placed in front of the user and not supported by or attached to the user's body. In some embodiments, the display generating component has a display side facing the user's face and eyes and displays computer-generated content. In some embodiments, the display generating component has a transparent or semi-transparent portion (e.g., the entire area of ​​the display side, a portion of the display side, etc.) through which the user on the display side facing the display generating component can see the physical environment behind the display generating component. In some embodiments, the display generating component displays a camera view of a portion of the physical environment that is blurred due to the presence of the display generating component, such that the user on the display side facing the display generating component can see the camera view of that portion of the physical environment, optionally together with virtual content overlaid on at least a portion of the camera view of that portion of the physical environment, a view that blocks at least that portion, and / or a display that replaces at least that portion. In some embodiments, the display generating component projects an image of the virtual content into the user's eyes, allowing the user to visually perceive the virtual content. In some embodiments, the display generating component projects an image of virtual content onto a physical surface, and the virtual content, along with the physical environment, is observed through a semi-transparent or transparent portion of the display generating component. In some embodiments, the display generating component is a head-mounted display attached to and / or supported by the user's head, face, ears, shoulders, and / or nose, with the display side of the head-mounted display facing the user's eyes and having a substantially fixed spatial relationship with the user's eyes. In some embodiments, the display generating component has a flat display side. In some embodiments, the display generating component has a curved display side that partially or completely surrounds the user's head or body. In some embodiments, the display generating component displays the complete virtual environment (e.g., a virtual game environment, a virtual office, etc.) without simultaneously displaying a representation of the physical environment (e.g., a see-through view of the user's physical environment (e.g., through a transparent portion of the display, or through a camera view), a recording of a camera view of the user's physical environment, etc.). In some embodiments, the display generating component displays virtual content along with a representation of the physical environment, wherein the virtual content is displayed at various locations relative to the representation of the physical environment. These various locations optionally have corresponding locations in the physical environment. In some implementations, spatial relationships in the physical environment are related to spatial relationships in the virtual reality, mixed reality, and / or augmented reality environment displayed via a display generation component. While the accompanying drawings illustrate one type of display generation component as an example, it is to be emphasized that the techniques disclosed herein are applicable to other types of display generation components in various implementations and may prove more advantageous for other types of display generation components (e.g., head-mounted displays, displays that obstruct the user's view of the physical environment, etc.).

[0121] Figures 7E to 7G The diagram illustrates the movement (e.g., rotation of the virtual object, translation of the virtual object, etc.) of a virtual object in response to a first type of user input relative to a three-dimensional environment (e.g., when the viewpoint remains stationary relative to the three-dimensional environment) according to some embodiments. Figure 7E After Figures 7H to 7I This illustrates, according to some embodiments, the movement (e.g., rotation, translation, etc.) of the viewpoint relative to the three-dimensional environment in response to a second type of user input. (e.g., viewpoint rotation, viewpoint translation, etc.) Figures 7E to 7I In the example shown, the computer system eliminates ambiguity in user input based on the hand gestures or grips maintained during user input. Figures 7E to 7I Used to illustrate the process described below, including Figures 8 to 10 The process in.

[0122] In some embodiments, the 3D environment is a virtual environment that does not represent the physical environment surrounding the user. In some embodiments, the 3D environment includes a virtual environment that spatially corresponds to the physical environment (e.g., including a virtual wall at a location corresponding to the physical location of a wall in the physical environment). In some embodiments, portions of the virtual environment are transparent to display portions of the physical environment at locations corresponding to virtual locations of portions of the virtual environment. For example, even if the virtual environment obscures a large portion of the view of the physical environment, a virtual window may be included in the virtual environment displaying a portion of the physical environment at a location corresponding to the location of a virtual window in the virtual environment. In some embodiments, the view change resulting from moving the viewpoint relative to a selected anchor position in the virtual environment is the same as the view change resulting from translating and / or reorienting the entire 3D environment, including all virtual objects contained therein, relative to the virtual position of the current viewpoint in the 3D environment. In some embodiments, the virtual 3D environment occupies the entire field of view provided by the display generation components. In some implementations, the virtual 3D environment occupies a portion of the field of view provided by the display generation component (e.g., the virtual environment can be viewed through a virtual portal or window displayed in parallel with a representation of the physical environment (e.g., overlaying a portion of the representation of the physical environment, replacing the display of that portion, or blocking the view of that portion)).

[0123] like Figure 7EAs shown, a first display generating component (e.g., display 7100) displays a virtual 3D environment (e.g., virtual 3D environment 7104), which includes various virtual objects (e.g., virtual object 7140 on virtual wall 7134, virtual table 7142 on virtual ground 7136, virtual wall 7138 perpendicular to and intersecting with virtual wall 7134 and virtual ground 7136, etc.) and / or surfaces (e.g., virtual objects 7140 and 7142, virtual walls 7134 and 7138, surface of virtual ground 7136, etc.) located at different positions within the virtual 3D environment. In some embodiments, the spatial relationship between virtual objects is defined by their relative positions and relative orientations within the virtual 3D environment. For example, virtual walls 7314 and 7318 are vertical or perpendicular in the 3D environment, while virtual ground 7136 is horizontal in the 3D environment. Virtual object 7140 is flat and parallel to virtual wall 7134, and virtual table 7142 is flat and parallel to virtual ground 7136. Other virtual objects with different positions and orientations relative to the 3D environment are also possible. The corresponding virtual object or surface in the virtual 3D environment has a corresponding position, spatial extent, shape, and / or orientation that can be static or change over time (e.g., in response to user input and / or according to pre-programmed instructions, etc.). In some embodiments, the viewpoint corresponding to the current display view of the virtual 3D environment refers to a virtual position, advantageous position, and / or viewing angle in the virtual 3D environment from which a virtual observer placed in the virtual position, advantageous position, and / or viewing angle (e.g., his / her eyes or head are in the virtual position, standing in the virtual position, sitting in the virtual position, etc.) will see the current display view. In some embodiments, the viewpoint corresponding to the current display view of the virtual 3D environment moves within the virtual 3D environment based on the movement of the user's head (e.g., head movement relative to the torso, head movement due to torso movement, etc.) from the position of the user observing the content displayed via the first display generating component. In some embodiments, the user's or a portion thereof has a corresponding position in the virtual 3D environment (e.g., a virtual position corresponding to the viewpoint of the current display view of the virtual 3D environment), and movement of the user as a whole or movement of a predetermined portion of the user (e.g., the user's head and / or torso) in the physical environment optionally results in a corresponding movement of the viewpoint of the current display view of the virtual 3D environment. In some embodiments, the correspondence between the movement of the user's head or torso in the physical environment and the movement of the viewpoint allows the user to experience spatial relationships in the virtual 3D environment in a more physical and realistic manner.In some embodiments, as disclosed herein, user physical input (e.g., hand gestures, finger gestures, arm movements, wrist movements, gaze input, various combinations thereof, etc.), other than physical movement of the user's head or torso in the physical environment, is used to move the viewpoint relative to virtual objects in the virtual 3D environment (e.g., presenting a portion of the 3D environment to the user's virtual location or vantage point) (e.g., moving the viewpoint closer to the virtual object, away from the virtual object, around the virtual object, etc.). In some embodiments, the movement of the viewpoint is visually represented as replacing a first view corresponding to a first viewpoint at a first virtual location with a second view corresponding to a second viewpoint at a second virtual location, and optionally displaying an intermediate view sequence corresponding to virtual locations between the first and second virtual locations (e.g., virtual locations along a virtual movement path of the viewpoint of the currently displayed view, movement paths of the user's virtual location in the position of observing content displayed via the display generation component, etc.).

[0124] In some implementation schemes, such as Figure 7E As shown, the first display generating component is a display placed in front of the user that is not supported by or attached to the user's body. In some embodiments, the display is a head-mounted display having a display side facing the user's face and eyes, and does not move relative to the user's face or eyes as the user moves their head or body in the physical environment. In some embodiments, when the head-mounted display is used as the display generating component, the virtual position of the viewpoint of the current display view of the physical environment corresponds (e.g., having a preset spatial relationship, having a constant spatial relationship, overlapping with, near, etc.) to the virtual position of the user's eyes or head in the virtual three-dimensional environment. Figures 7E to 7I In the examples shown, according to some implementations, the user as a whole is stationary relative to the physical environment. In some implementations, the user as a whole may move within the physical environment, but the viewpoint is not updated based on the user's movement within the physical environment. In some implementations, the user as a whole may move within the physical environment and cause a movement of the viewpoint, which is independent of and supplements to movement of the viewpoint caused by physical inputs other than the user's movement within the physical environment.

[0125] exist Figure 7EIn the first view of the 3D environment 7104, a representation of the user's hand (e.g., hand 7202) (e.g., representation 7202') and a representation of the user's other hand (e.g., hand 7204) (e.g., representation 7204') are shown. In this example, the first viewpoint is located at a first virtual position in the 3D environment, which is on the side of the virtual table 7142 opposite to the virtual wall 7134, and on the right side of the virtual wall 7138, and above the virtual ground 7136. The virtual table has two parts 7142-1 and 7142-2, and the first virtual position is closer to the shaded part 7142-2 of the virtual table 7142 and further away from the unshaded part 7142-1. In some embodiments, the representation of the user's hand is a camera view of the user holding the hand in front of the user's body. For example, the user's hand is identified and segmented from the camera view, and then overlaid or displayed in the 3D environment at a location corresponding to the user's virtual position in the 3D environment (e.g., optionally determining the appropriate virtual position of the hand based on the virtual position of the viewpoint of the current display view (e.g., a first virtual position)). In some embodiments, the representation of the user's hand is a stylized graphical representation of the user's hand, which optionally moves and / or changes to show the movement and / or changing posture of the user's hand in the physical environment. In some embodiments, the representation of the user's hand is not displayed in the current display view of the 3D environment or is not visible therein. For example, in some embodiments, the user's hand is outside the user's field of view provided by the first display generating component. In some embodiments, the user's hand is located in front of the user, but would otherwise be in the user's field of view if the first display generating component were not present in front of the user (e.g., the view of the user's hand is obstructed by the first display generating component, and the representation of the user's hand is not displayed via the first display generating component). In other words, in some implementations, the user's hand performs gestures or movements in the physical environment, which are captured as input resulting in changes in the 3D environment, even if the gestures and movements of the user's hand are not displayed via a representation of the user's hand in the view of the 3D environment. In some implementations, instead of two hands, a single hand of the user is used to provide the user input required for moving virtual objects and / or moving the viewpoint relative to the 3D environment.

[0126] exist Figures 7E to 7FIn the process of displaying a first view of the 3D environment 7104, the computer system detects movement of the user's hands 7202 and 7204, which forms a first type of hand gesture or grasp (e.g., one or both hands fully closed into a fist, fingers curled towards the palm of one or both hands, etc.). In some embodiments, the first type of hand gesture or grasp requires both of the user's hands to have a first predefined orientation relative to the user or physical environment (e.g., both hands facing each other, both hands facing the same direction (e.g., upward, towards the user, etc.)). In some embodiments, the computer system requires only one of the user's hands to form the first type of hand gesture or grasp for the desired input. In some embodiments, the computer system requires only one of the user's hands to have a first predefined orientation relative to the user or physical environment for the desired input. Figure 7F In the process, after the computer system detects that the user's hand is maintaining a first type of hand posture or grasp, the computer system detects that the user's hand is performing a first type of movement while maintaining the first type of hand posture or grasp (e.g., both hands rotate around a common center, both hands move in opposite directions, both hands move in the same direction, one hand rotates around a selected anchor position, one hand translates relative to a preselected anchor position, etc.). Figure 7F After Figure 7G The diagram illustrates how, in response to the detection of user hand movement, and based on the determination that the user's hand has made a first-type movement while maintaining a first-type hand posture or grasp, the computer system recognizes the user's hand movement as a user request to rotate a virtual object in a three-dimensional environment (e.g., around its own central axis, around the viewpoint, etc.) while keeping the virtual position of the viewpoint stationary in the three-dimensional environment. It should be noted that while in some scenarios the viewpoint movement may be triggered in response to movement of the user's head in the physical environment, the viewpoint movement caused by the user's head movement will be independent of and supplementary to the changes caused by the user's hand movement as described herein.

[0127] like Figure 7GAs shown, a virtual table 7142 is selected as the target of a user-requested rotation and / or translation of a virtual object relative to the 3D environment (e.g., based on the position of the user's hand, the position of the user's gaze, and / or based on which objects in the 3D environment are movable, etc.). In some embodiments, the object selected as the target of the user-requested rotation or translation (e.g., virtual table 7142) is visually emphasized relative to the 3D environment (e.g., through visual effects 7144 (e.g., spotlights, outlines, animations, etc.), or visual effects that soften the surrounding environment of the selected target, etc.). In some embodiments, the computer system detects gaze input pointing towards a corresponding part of the virtual 3D environment and selects the target based on the position of the gaze input at the moment when it approaches the user's hand to form a first type of hand gesture or grasp. In some embodiments, as the user's eyes move around on the display side facing the first display generating component, the computer system tracks the movement of the user's eyes and determines the position of the user's gaze and the user's focus in the 3D environment. For example, in some embodiments, gaze input is detected based on at least a threshold time amount within a threshold region determining that the user's gaze and focus have been positioned at a first location in the 3D environment; and a virtual object present at the first location is optionally identified as the target of the gaze input. In some embodiments, the computer system displays visual markers to indicate the current location of the gaze input. In some embodiments, visual markers are displayed based on determining that the target of the gaze input has been identified at the location of the gaze input. In some embodiments, visual markers are displayed based on determining that the gaze input has met preset criteria (e.g., being held within a threshold region for at least a threshold time amount (e.g., a threshold time amount for detecting the gaze input (e.g., a gaze detection threshold), another threshold time amount after the gaze input is detected (e.g., a dwell time threshold)), etc.). In some embodiments, the visual markers are displayed only to indicate the movement of the user's gaze and / or focus, and do not necessarily mean that the user's attention is focused at the location of the visual markers. In some implementations, if the visual markers are used to indicate only the movement of the gaze or focus, additional visual feedback (e.g., visual effect 7144) is provided when gaze input is detected and / or when a target object of the gaze input is detected (e.g., based on a single gaze input, or based on gaze input detected in combination with a user's hand forming a first type of hand gesture or grasp). In some implementations, when gaze is used to select a target of a virtual object requested by the user to rotate or translate, the user can use their hand to rotate or translate the virtual object at a virtual location away from the viewpoint.

[0128] In some implementations, as the user's hand moves in the physical environment, the computer system determines the corresponding amount of movement to be performed by the virtual object relative to the three-dimensional environment (e.g., rotation relative to the virtual object's axis or center or another anchor axis or position, translation relative to the virtual object's initial position, etc.) based on the amount of movement performed by the user's hand (e.g., rotational movement relative to an anchor position (e.g., the center of both hands), translational movement relative to the user's torso, etc.). In some implementations, the computer system determines the direction of movement to be performed by the virtual object (e.g., clockwise, counterclockwise, forward, backward, up, down, left, right, etc.) based on the direction of movement performed by the user's hand while maintaining a first type of hand posture or grasp (e.g., clockwise, counterclockwise, forward, backward, up, down, left, right, etc.). In some implementations, the computer system determines the movement characteristics of the virtual object (e.g., speed, acceleration, distance, direction, path, etc.) based on the movement characteristics of the user's hand while maintaining a first type of hand posture or grasp. In some implementations, after the user's hand has stopped moving and / or no longer maintains the first type of hand posture or grip, the computer system continues the movement of the virtual object (e.g., rotation, translation, etc.) to simulate the behavior of the virtual object in terms of moment of inertia, conservation of angular momentum, and other physical laws.

[0129] exist Figure 7F and Figure 7G In the example shown, virtual table 7142 is selected as the target of the user's request to rotate a virtual object relative to the 3D environment. In another example, if a different virtual object (e.g., virtual object 7140 on virtual wall 7134) is selected as the target of the user's request to rotate a virtual object relative to the 3D environment (e.g., based on the location of the user's gaze input at the moment when the user's hand forms a first type of hand gesture or grasp, based on the location of a pointing gesture detected before the user's hand forms the first type of hand gesture or grasp, etc.), then the same movement of the user's hand while in the first type of hand gesture or grasp will result in different virtual objects moving in the 3D environment. In some embodiments, the movement of virtual objects is constrained by the type and / or position of the virtual object in the 3D environment. For example, virtual table 7142 is optionally constrained to rotate about a vertical axis and move in a horizontal plane above virtual ground 7136, while virtual object 7140 is optionally constrained to move in a vertical plane of virtual wall 7134 and rotate about a horizontal axis perpendicular to virtual wall 7134. In some implementations, the same type of movement of the user's hand may optionally result in different types of movement of the virtual object (e.g., different rotation directions, different movement directions, different orientations, etc. during movement), depending on the type of the virtual object and the movement restrictions imposed on the virtual object.

[0130] exist Figures 7H to 7I In, with Figures 7F to 7G The scenario shown contrasts with the first view of the 3D environment 7104, where the computer system detects movements of user hands 7202 and 7204 forming a second type of hand gesture or grasp that differs from the first type (e.g., multiple fingers of one or both hands forming a pinch gesture, fingers extending from the palm of one or both hands, thumbs placed on the index finger side of one or both hands, etc.). In some embodiments, the second type of hand gesture or grasp requires both of the user's hands to have a second predefined orientation relative to the user or physical environment (e.g., the same as the first predefined orientation, different from the first predefined orientation, etc.) (e.g., both hands facing each other, both hands facing the same direction (e.g., upward, towards the user, etc.)). In some embodiments, the computer system requires only one of the user's hands to form the second type of hand gesture or grasp for the desired input. In some embodiments, the computer system requires only one of the user's hands to have a second predefined orientation relative to the user or physical environment for the desired input. Figure 7H In this context, after the computer system detects that the user's hand is maintaining a second type of hand posture or grasp, the computer system detects that the user's hand is performing a second type of movement while maintaining the second type of hand posture or grasp (e.g., both hands rotating around a common center, both hands moving in opposite directions, both hands moving in the same direction, one hand rotating around a selected anchor point, one hand translating relative to a pre-selected anchor point, etc.). Figure 7H After Figure 7I The diagram illustrates how, in response to the detection of user hand movement, the computer system recognizes the user hand movement as a user request to rotate and / or translate the viewpoint relative to the 3D environment while keeping the virtual object stationary, based on the determination that the user's hand has performed a second type of movement (e.g., the same as the first type of movement, different from the first type of movement, etc.) while maintaining a second type of hand posture or grasp. It should be noted that while viewpoint movement may be triggered in some scenarios in response to movement of the user's head in the physical environment, viewpoint movement due to movement of the user's head will be independent of and supplementary to the changes caused by user hand movement as described herein.

[0131] like Figure 7HAs shown, the virtual table 7142 is not selected as the target of the user's request because the request pertains to moving the viewpoint relative to the 3D environment. In some embodiments, the movement of the viewpoint is a rotational movement (e.g., pure rotation, without translation) around a selected anchor point (e.g., the virtual table 7142) or around a virtual position anchored to the viewpoint. In some embodiments, when an anchor position other than the viewpoint's virtual position is selected, the computer system selects the anchor position based on the user's hand position, the user's gaze position, and / or other earlier input. In some embodiments, the object selected as the anchor for the rotation or translation of the viewpoint is visually emphasized relative to the 3D environment. In some embodiments, the entire environment is visually varied to indicate that the viewpoint relative to a particular virtual object will be rotated or translated in response to movement of the user's hand in a second type of hand gesture or grasp.

[0132] In some implementations, as the user's hand moves in the physical environment, the computer system determines the corresponding amount of movement to be performed by the viewpoint relative to the three-dimensional environment (e.g., rotational movement relative to the anchor point (e.g., the center of both hands), translational movement relative to the user's torso, etc.) based on the amount of movement performed by the user's hand (e.g., rotational movement relative to the anchor point or axis (e.g., a position or axis passing through the current virtual position of the selected virtual object or viewpoint, etc.)). In some implementations, the computer system determines the direction of movement to be performed by the viewpoint of the currently displayed view (e.g., clockwise, counterclockwise, forward, backward, up, down, left, right, etc.) based on the direction of movement performed by the user's hand while maintaining a second type of hand gesture or grasp. In some implementations, the computer system determines the viewpoint's movement characteristics (e.g., velocity, acceleration, distance, direction, path, etc.) based on the user's hand movement characteristics (e.g., velocity, acceleration, distance, direction, path, etc.) while maintaining the second type of hand posture or grip. In some implementations, after the user's hand has stopped moving and / or no longer maintains the second type of hand posture or grip, the computer system continues the viewpoint movement (e.g., rotation, translation, etc.) to simulate the user's behavior regarding moment of inertia, conservation of angular momentum, and other physical laws in a three-dimensional environment.

[0133] exist Figure 7H and Figure 7IIn the example shown, the virtual table 7142 is rotated as an anchor for the viewpoint of the currently displayed view of the 3D environment requested by the user to rotate relative to the 3D environment. In response to a user hand movement, the viewpoint moves from a virtual position on the virtual table 7142 opposite to the virtual wall 7134 and closer to the shadow portion 7142-2 of the virtual table to another virtual position on the virtual table 7142 opposite to another virtual wall 7150 (adjacent to the virtual wall 7134). This is as if the user's virtual position has moved (e.g., rotated) around the virtual table in a direction corresponding to the user's hand movement direction (e.g., clockwise, counterclockwise, etc.). Therefore, as... Figure 7I As shown, the computer system now displays a view of the three-dimensional environment corresponding to a second viewpoint, which is on the side of the virtual table opposite to the virtual wall 7150, on the same side of the virtual wall 7134 as the virtual table 7142, and above the virtual ground 7136. Figure 7H The views shown create a contrast. In another example, if (e.g., based on the location of the user's gaze input at the moment the second type of hand gesture or grasp is formed near the user's hand, based on the location of a pointing gesture detected before the second type of hand gesture or grasp is formed, etc.) different virtual objects or locations are selected as anchors for the rotational or translational movement of the viewpoint, then the same movement of the user's hand while in the second type of hand gesture or grasp will result in the viewpoint rotating about different locations in the 3D environment or translating relative to different locations in the 3D environment. In some embodiments, the rotational movement of the viewpoint is restricted, for example, to only about the vertical axis. For example, in some embodiments, the user cannot flip or invert the 3D environment to one side.

[0134] In some implementations, the first type of hand gesture or grip is a full-hand grip that moves a virtual object in the 3D environment relative to the 3D environment and viewpoint (e.g., all fingers curled towards the palm of the same hand). In some implementations, the second type of hand gesture or grip is a multi-finger grip that moves the viewpoint in the 3D environment (e.g., two or more fingers touching each other, rather than the palm of the same hand), allowing the user to view the 3D environment from different viewing angles. In some implementations, a full-hand grip involves the user's corresponding hand (e.g., each hand, first hand, second hand, first hand and second hand, etc.) having a first set of fingers (e.g., the last three or four fingers, all fingers, etc.) kept in contact with the palm of the corresponding hand. A full-hand grip may involve gripping by closing the entire hand (e.g., similar to a closed fist of one or both hands), and the second type of movement of the hand while maintaining the full-hand grip causes the object in the 3D environment to move relative to the 3D environment. In some implementations, multi-finger grasping involves a set of fingers (e.g., the thumb) of a corresponding hand (e.g., each hand, a first hand, a second hand, a first hand and a second hand, etc.) maintaining contact with another set of fingers of the corresponding hand (e.g., the index finger and / or the middle finger, etc.). Multi-finger grasping may involve grasping by closing the thumb and another finger (e.g., the fingers of one or both hands), and a first type of hand movement (e.g., rotation) about a common center of rotation while holding the multi-finger grasp causes the user's virtual position (e.g., viewpoint) in the three-dimensional environment to move relative to the three-dimensional environment. In some implementations, the functions of the first type of hand gesture or grasping and the second type of hand gesture or grasping are reversed, and movement of one or both hands while holding the second type of hand gesture or grasping causes objects in the three-dimensional environment to move relative to the three-dimensional environment, and movement of one or both hands while holding the first type of hand gesture or grasping causes the user's virtual position (e.g., viewpoint) in the three-dimensional environment to move relative to the three-dimensional environment.

[0135] In some embodiments, the first type of hand gesture or grasp is a single-hand gesture or grasp, while the second type of hand gesture or grasp is a two-hand gesture or grasp.

[0136] In some implementations, the first type of hand gesture or grip is a single-hand gesture or grip, and movement of the user's hand while maintaining the single-hand gesture or grip causes the selected virtual object to move relative to the 3D environment (e.g., also relative to the viewpoint of the currently displayed view of the 3D environment) (e.g., as if the virtual object moved relative to the user's virtual position in the 3D environment based on the movement of the user's hand). In some implementations, the second type of hand gesture or grip is a two-hand gesture or grip, and movement of the user's hand while maintaining the two-hand gesture or grip causes the viewpoint to move relative to the 3D environment (e.g., as if the user moved relative to the 3D environment based on the movement of the user's hand).

[0137] In some implementations, when switching from displaying a view corresponding to a first viewpoint to displaying a view corresponding to a second viewpoint, the computer system displays an intermediate view sequence corresponding to the intermediate viewpoint sequence along the movement path of the viewpoint, wherein the path is determined based on the movement path of the user's hand while maintaining a second type of hand posture or grip.

[0138] In some implementations, movement of a user's hand while holding a single-handed grip causes the selected virtual object to move relative to the viewpoint in the 3D environment, while movement of a user's hand while holding a two-handed grip causes the viewpoint to move in the 3D environment (e.g., relative to the selected virtual object). For example, in some implementations, if a user moves their hand toward themselves while holding a single-handed grip, the computer system moves the selected virtual object toward itself in a first view of the 3D environment corresponding to a first viewpoint, and if a user moves both hands toward themselves while holding a two-handed grip, the computer system moves the viewpoint of the current display view of the 3D environment toward a selected anchoring position in the 3D environment (e.g., to the selected virtual object, to the nearest suitable viewpoint for observing the selected virtual object, etc.).

[0139] In some embodiments, moving a virtual object in a three-dimensional environment includes rotating the virtual object in the three-dimensional environment about a virtual position corresponding to a first viewpoint (e.g., such that the virtual object rotates in the three-dimensional environment about the user's virtual position without moving closer to the user's virtual position, and optionally, during the rotation of the virtual object, the same side of the virtual object is shown to the user in a first view of the three-dimensional environment). In some embodiments, moving a virtual object in a three-dimensional environment includes rotating the virtual object in the three-dimensional environment about a virtual axis passing through the virtual object (e.g., such that different sides of the virtual object face the user without moving closer to or further away from the user's virtual position). In some embodiments, moving the viewpoint of the current display view of the three-dimensional environment from a first viewpoint to a second viewpoint includes rotating the viewpoint about an anchor position (e.g., the position of the virtual object) with a predetermined spatial relationship relative to the first viewpoint (e.g., in front of the position of the first viewpoint) (e.g., such that the user can see the virtual object at the target position from another angle without moving the virtual object in the three-dimensional environment). For example, in some implementations, when a user moves their hand toward themselves in a clockwise or counterclockwise direction around a horizontal circle, if the hand maintains a first type of grip during the movement, the computer system rotates a virtual object located at a selected target position around its own central axis such that the distal side of the virtual object moves toward the user's virtual position in a first view of the three-dimensional environment corresponding to a first viewpoint, and if the hand maintains a second type of grip during the movement, the computer system moves the viewpoint of the current display view of the three-dimensional environment along a circular track around an anchor point to a different side of the selected target position in the three-dimensional environment (e.g., the virtual object at the target position, or a position without a virtual object, etc.).

[0140] In some implementations, the computer system requires that the distance between a user's two hands, while maintaining a first-type hand gesture or grasp and / or a second-type hand gesture or grasp, does not change by more than a threshold amount during hand movement, in order to recognize hand movement as a request to move a virtual object or a request to move the viewpoint relative to the 3D environment. In some implementations, if the computer system detects that the distance between a user's two hands changes more than a threshold amount during hand movement while maintaining a first-type hand gesture or grasp and / or a second-type hand gesture or grasp, the computer system recognizes hand movement as a request to adjust the size or shape of a virtual object.

[0141] In some implementations, based on determining that the user's hands maintain a third type of hand posture or grasp (e.g., a first type of hand posture or grasp, a second type of hand posture or grasp, or a hand posture or grasp type different from the first and second types of hand postures and grasps, etc.) while performing a third type of movement (e.g., coordinated movement of both hands, such as separating or moving both hands toward each other), the computing system adjusts the size of a selected virtual object (e.g., virtual table 7142, virtual object 7140, etc.) in the three-dimensional environment relative to the three-dimensional environment based on the movement of the two hands in the physical environment (e.g., relative movement of the two hands). In some implementations, the size change of the virtual object is proportional to or otherwise related to the change in the distance between the two hands in the physical environment, which begins when the two hands first form the third type of hand posture or grasp.

[0142] In some implementations, based on determining that the user's hands maintain a fourth type of hand posture or grasp that differs from the third type of hand posture or grasp when performing a fourth type of movement (e.g., a first type of hand posture or grasp, a second type of hand posture or grasp, or a hand posture or grasp type different from the first and second types of hand posture or grasp, etc.), the computing system adjusts the size of the three-dimensional environment and, optionally, all virtual objects within the three-dimensional environment according to the movement of the hands in the physical environment (e.g., the relative movement of the two hands). In some implementations, the change in the size of the three-dimensional environment as a whole is proportional to or otherwise related to the change in the distance between the two hands in the physical environment, starting from the moment the two hands first form the fourth type of hand posture or grasp.

[0143] In some implementations, the computer system determines whether to move a virtual object or viewpoint based on whether gaze input is detected on a movable virtual object or on a portion of the 3D environment where no movable virtual object exists (e.g., a virtual object or surface exists but is immovable or empty). For example, if a user gazes at a virtual vase on a virtual table, a movement of the user's hand (e.g., while maintaining a preset hand grip) is recognized as a request to move the virtual object relative to the 3D environment and viewpoint; and if the user gazes at a virtual wall, a movement of the user's hand (e.g., while maintaining a preset hand grip) is recognized as a request to move the viewpoint relative to the 3D environment. In some implementations, a virtual object is movable if the physical object simulated by the virtual object can be physically moved by the user (e.g., rotatable, repositionable, etc.) (e.g., movable based on its medium size and weight, based on whether it is a fixed object or an object, etc.).

[0144] In some implementations, in response to a flexing gesture of both hands (e.g., both hands are in their respective grasps when they move parallel to each other toward or away from the user's torso), the computer system changes the height of the selected virtual object based on the user's hand movements. For example, as the user raises or lowers both hands together in front of the user (e.g., by bending the hands upward around their respective wrists as the arms extend forward in front of the user, or by rotating the forearm upward or downward around the elbow in front of the user when the elbow is stationary in front of or near the user's side, or by rotating the entire arm upward or downward around the shoulder joint in front of the user when the forearm is curled up or extended upward, etc.), the computing system adjusts the height of the virtual object in the three-dimensional environment based on the flexing movement of the hands. In some implementations, the object's height increases with each flexing movement. In some implementations, the object's height increases as the hands move upward and decreases as the hands move downward.

[0145] Figures 7J to 7O This illustrates, according to some embodiments, the selection of a suitable viewpoint for observing a target object or location in response to a user request for navigation to a target object or location in a three-dimensional environment. For example, in response to detecting hand movement corresponding to a request to move towards a selected virtual object in the three-dimensional environment, the computer does not simply move the viewpoint to the location of the selected virtual object or to a fixed distance from it, but rather moves it to an appropriate viewing position for the selected virtual object based on one or more preset characteristics of the selected virtual object. In other words, depending on the virtual object selected as the navigation target, the viewpoint may move from the selected virtual object to a specific distance and / or angle from the object. According to some embodiments, if the preset characteristics of the first and second virtual objects are different, the resulting spatial relationship between the navigation target and the viewpoint position at the end of movement toward the navigation target differs between selecting the first virtual object as the navigation target and selecting the second virtual object as the navigation target.

[0146] Figures 7J to 7K , Figures 7L to 7M and Figures 7N to 7O Three different scenarios are illustrated according to some implementation schemes, in which different virtual objects are selected as navigation targets for movement performed in response to user requests. Figures 7J to 7K , Figures 7L to 7M and Figures 7N to 7OIn the examples shown, the same type of user input (e.g., gaze input 7208 along with hand movements with a predefined hand gesture or grip, etc.) is used to select a navigation target and initiate movement toward that target. In some embodiments, the user input includes a sequence of multiple user inputs (e.g., hand movements repeated multiple times while maintaining a desired hand gesture or grip) corresponding to a request that enables maximum movement toward the selected navigation target. In each example, the viewpoint at the end of the requested movement is shown to be closer to the corresponding position of the navigation target than before the movement began, but still spaced apart from the corresponding position of the navigation target, and the final position of the viewpoint is rotated based on one or more predefined characteristics of the virtual object selected as the navigation target (e.g., characteristics include the size, dimensions, and / or properties of the virtual object, etc.) and in conjunction with the corresponding position of the virtual object. In some embodiments, a virtual location may be selected as the navigation target, and the computer system moves the viewpoint to that virtual location (e.g., without displacement or offset from that virtual location) based on the determination that no virtual object exists at the virtual location selected as the navigation target. In some implementations, a single approach of selecting a navigation target and making a movement request is not required; instead, different approaches of selecting a navigation target and making a movement request can be used to trigger movement toward the same object or toward different objects.

[0147] Figure 7J , Figure 7L and Figure 7N A computer system including at least a first display generation component (e.g., display 7100) according to some embodiments is shown. Figure 1 Computer system 101 or Figure 4 In a computer system 140, etc., the computer system displays computer-generated content to a user via a first display generation component (e.g., a display 7100). In some embodiments, such as Figure 7J , Figure 7L and Figure 7NAs shown, the first display generating component is a display (e.g., display 7100) placed in front of the user and not supported by or attached to the user's body. In some embodiments, the display generating component has a display side facing the user's face and eyes and displays computer-generated content. In some embodiments, the display generating component has a transparent or semi-transparent portion (e.g., the entire area of ​​the display side, a portion of the display side, etc.) through which the user on the display side facing the display generating component can see the physical environment behind the display generating component. In some embodiments, the display generating component displays a camera view of a portion of the physical environment that is blurred due to the presence of the display generating component, such that the user on the display side facing the display generating component can see the camera view of that portion of the physical environment, optionally together with virtual content overlaid on at least a portion of the camera view of that portion of the physical environment, a view that blocks at least that portion, and / or a display that replaces at least that portion. In some embodiments, the display generating component projects an image of the virtual content into the user's eyes, allowing the user to visually perceive the virtual content. In some embodiments, the display generating component projects an image of virtual content onto a physical surface, and the virtual content, along with the physical environment, is observed through a semi-transparent or transparent portion of the display generating component. In some embodiments, the display generating component is a head-mounted display attached to and / or supported by the user's head, face, ears, shoulders, and / or nose, with the display side of the head-mounted display facing the user's eyes and having a substantially fixed spatial relationship with the user's eyes. In some embodiments, the display generating component has a flat display side. In some embodiments, the display generating component has a curved display side that partially or completely surrounds the user's head or body. In some embodiments, the display generating component displays the complete virtual environment (e.g., a virtual game environment, a virtual office, etc.) without simultaneously displaying a representation of the physical environment (e.g., a see-through view of the user's physical environment (e.g., through a transparent portion of the display, or through a camera view), a recording of a camera view of the user's physical environment, etc.). In some embodiments, the display generating component displays virtual content along with a representation of the physical environment, wherein the virtual content is displayed at various locations relative to the representation of the physical environment. In some implementations, various locations have corresponding positions in the physical environment. In some implementations, spatial relationships in the physical environment are related to spatial relationships in the mixed reality and / or augmented reality environment displayed via display generation components, such that virtual objects appear to be positioned and / or moved relative to the physical environment in the mixed reality and / or augmented reality environment.While the accompanying drawings illustrate one type of display generation component as an example, it is important to emphasize that the techniques disclosed herein are applicable to other types of display generation components in various embodiments and may prove more advantageous for other types of display generation components (e.g., head-mounted displays, displays that obstruct the user's view of the physical environment, etc.).

[0148] In some embodiments, the 3D environment is a virtual environment that does not represent the physical environment surrounding the user. In some embodiments, the 3D environment includes a virtual environment that spatially corresponds to the physical environment (e.g., including a virtual wall at a location corresponding to the physical location of a wall in the physical environment). In some embodiments, portions of the virtual environment are transparent to display portions of the physical environment at locations corresponding to virtual locations of portions of the virtual environment. For example, even if the virtual environment obscures a large portion of the view of the physical environment, a virtual window may be included in the virtual environment displaying a portion of the physical environment at a location corresponding to the virtual location of a virtual window in the virtual environment. In some embodiments, the view change resulting from moving the viewpoint relative to a selected anchor position in the virtual environment is the same as the view change resulting from translating and / or reorienting the entire 3D environment, including all virtual objects contained therein, relative to the virtual position of the viewpoint in the 3D environment. In some embodiments, the virtual 3D environment occupies the entire field of view provided to the user by the display generation components. In some implementations, the virtual 3D environment occupies a portion of the user's field of view provided by the display generation component (e.g., a virtual entrance or window displayed in parallel with a representation of the physical environment (e.g., a view that overlays a portion of the representation of the physical environment, replaces the display of that portion, or blocks the view of that portion)).

[0149] like Figure 7J , Figure 7L and Figure 7NAs shown, the first display generating component (e.g., display 7100) displays a virtual three-dimensional environment (e.g., virtual three-dimensional environment 7106), which includes various virtual objects (e.g., virtual picture frame 7152 on virtual wall 7134, virtual window 7154 on the same virtual wall 7134 next to virtual picture frame 7152, virtual ground 7136, virtual wall 7138 perpendicular to and intersecting virtual wall 7134 and virtual ground 7136, virtual hill 7160 outside virtual window 7154, virtual tree 7156 outside virtual window 7154 and on virtual hill 7160, virtual grid 7158 of virtual window 7154, etc.) and / or surfaces (e.g., the surface of virtual picture frame 7152 and the surface of virtual window 7154, virtual walls 7134 and 7138, virtual ground 7136, the surface of virtual grid 7158, etc.) located at different locations in the virtual three-dimensional environment. In some implementations, the spatial relationship between virtual objects is defined by their relative positions and orientations within the virtual 3D environment. For example, virtual walls 7314 and 7318 are vertical or elongated in the 3D environment, while virtual ground 7136 is horizontal. Virtual picture frames 7152 and virtual windows 7154 are flat and located in the same plane parallel to virtual wall 7134. Virtual grid 7158 is at the same virtual depth as virtual window 7154 from the viewpoint, while virtual tree 7156 and virtual hill 7160 are much farther from the viewpoint than virtual window 7158. In this example, virtual picture frames 7152 and virtual window 7154 are at a distance of... Figure 7J , Figure 7L and Figure 7NThe virtual position of the viewpoint of the first view of the illustrated three-dimensional environment is at the same distance. Corresponding virtual objects or surfaces in the virtual three-dimensional environment have corresponding positions, spatial extents, shapes, and / or orientations that can be static or change over time (e.g., in response to user input and / or according to pre-programmed instructions, etc.). In some embodiments, the viewpoint corresponding to the current display view of the virtual three-dimensional environment refers to a virtual position, advantageous position, and / or viewing angle in the virtual three-dimensional environment from which a virtual observer placed at the virtual position, advantageous position, and / or viewing angle (e.g., his / her eyes or head are at the virtual position, standing at the virtual position, sitting at the virtual position, etc.) will see the current display view. In some embodiments, the viewpoint corresponding to the current display view of the virtual three-dimensional environment moves in the virtual three-dimensional environment according to the movement of the user's head (e.g., head movement relative to the torso, head movement due to torso movement, etc.) at the position observing the content displayed via the first display generating component. In some embodiments, the position of a user or a portion thereof in the physical environment has a corresponding position in the virtual 3D environment (e.g., a virtual position corresponding to the viewpoint of the current display view of the virtual 3D environment), and movement of the user as a whole or a portion thereof in the physical environment optionally results in a corresponding movement of the viewpoint of the current display view of the virtual 3D environment. In some embodiments, the correspondence between the movement of the user's head or torso in the physical environment and the movement of the viewpoint allows the user to experience spatial relationships in the virtual 3D environment in a more physical and realistic manner. In some embodiments, as disclosed herein, user physical input (e.g., hand gestures, finger gestures, arm movements, wrist movements, gaze input, various combinations thereof, etc.) other than the physical movement of the user's head or torso in the physical environment is used to move the viewpoint relative to virtual objects in the virtual 3D environment (e.g., moving the viewpoint closer to the virtual object, away from the virtual object, around the virtual object, etc.). In some embodiments, the movement of the viewpoint is visually represented as replacing a first viewpoint corresponding to a first viewpoint at a first virtual location with a second viewpoint corresponding to a second viewpoint at a second virtual location. In some implementations, the computer system displays an intermediate view sequence corresponding to an intermediate viewpoint sequence (e.g., a virtual position along a virtual movement path from the viewpoint of the currently displayed view, a movement path from the user's virtual position, etc.) at a virtual position between a first virtual position and a second virtual position. Figures 7J to 7K , Figures 7L to 7M and Figures 7N to 7O In some implementation schemes, the viewpoint is shifted to different virtual objects in the 3D environment that the user selects as navigation targets for the movement request. Figures 7J to 7K , Figures 7L to 7M and Figures 7N to 7OIn the three example scenarios shown, the final position of the viewpoint is selected based on both the location of the virtual object chosen as the navigation target and the properties and characteristics of the virtual object chosen as the navigation target, so that when the movement performed according to the user's request ends, the virtual object chosen as the navigation target can be observed from an appropriate distance in the final view of the 3D environment.

[0150] In some implementation schemes, such as Figures 7J to 7O As shown, the first display generating component is a display placed in front of the user that is not supported by or attached to the user's body. In some embodiments, the display is a head-mounted display having a display side facing the user's face and eyes, and does not move relative to the user's face or eyes as the user moves their head or body in the physical environment. In some embodiments, when the head-mounted display is used as the display generating component, the virtual position of the viewpoint of the current display view of the physical environment corresponds (e.g., having a preset spatial relationship, having a constant spatial relationship, overlapping with, near, etc.) to the virtual position of the user's eyes or head in the virtual three-dimensional environment. Figures 7J to 7O In the examples shown, according to some implementations, the user as a whole is stationary relative to the physical environment. In some implementations, the user as a whole may move within the physical environment, but the viewpoint is not updated based on the user's movement within the physical environment. In some implementations, the user as a whole may move within the physical environment, and the viewpoint is updated based on the user's movement within the physical environment, but updates to the viewpoint are supplemented and independent of the viewpoint movement in response to user movement requests made for corresponding virtual objects.

[0151] exist Figures 7J to 7K , Figures 7L to 7M and Figures 7N to 7O In this context, the computer system detects gaze input (e.g., gaze input 7208) directed at a corresponding part of the virtual 3D environment. For example, in... Figures 7J to 7K In the image, gaze input 7208-a points to the virtual picture frame 7152 on the virtual wall 7134. Figures 7L to 7M In the middle, gaze at the point on the virtual grid 7158 of the virtual window 7154 pointed to by input 7208-b. Figures 7N to 7OIn the embodiment, gaze input 7208-c points to a virtual tree 7156 outside the virtual window 7154. In some embodiments, as the user's eyes move around on the display side facing the first display generating component, the computer system tracks the movement of the user's eyes and determines the position of the user's gaze and focus in the three-dimensional environment. For example, in some embodiments, gaze input is detected based on at least a threshold time amount within a threshold region that determines the user's gaze and focus have been positioned at a first location in the three-dimensional environment, and optionally, a virtual object present at the first location is identified as the target of the gaze input. In some embodiments, the computer system displays a visual marker to indicate the current position of the gaze input. In some embodiments, a visual marker is displayed based on determining that the target of the gaze input has been identified at the position of the gaze input. In some embodiments, a visual marker is displayed based on determining that the gaze input has met preset criteria (e.g., remaining within a threshold region for at least a threshold time amount (e.g., a threshold time amount for detecting the gaze input (e.g., a gaze detection threshold), another threshold time amount after the gaze input is detected (e.g., a dwell time threshold)), etc.). In some embodiments, the visual marker is displayed only to indicate the movement of the user's gaze and / or focus, and does not necessarily mean that the user's attention is focused at the position of the visual marker. In some implementations, if the visual markers are used to indicate only the movement of the gaze or focus, additional visual feedback is provided when gaze input is detected and / or when a target object of the gaze input is detected (e.g., based on a single gaze input, or based on gaze input detected in combination with another user input, etc.). According to some implementations, see reference to... Figures 7A to 7D The example shown describes the selection of the target object in more detail.

[0152] exist Figures 7J to 7K , Figures 7L to 7M and Figures 7N to 7OIn the three-dimensional environment view 7106, a representation (e.g., representation 7202') of the user's hand (e.g., hand 7202) is shown. In some embodiments, the representation of the user's hand is a camera view of the hand held in front of the user's body. For example, the user's hand is identified and segmented from the camera view and then overlaid or displayed in the three-dimensional environment at a location corresponding to the user's virtual position in the three-dimensional environment (e.g., the appropriate virtual position of the hand is optionally determined based on the virtual position of the viewpoint of the current display view). In some embodiments, the representation of the user's hand is a stylized graphical representation of the user's hand, which optionally moves and / or changes to show the movement and / or changing posture of the user's hand in the physical environment. In some embodiments, the representation of the user's hand is not displayed in the current display view of the three-dimensional environment. For example, in some embodiments, the user's hand is outside the user's field of view provided by the first display generating component. In some embodiments, the user's hand is located in front of the user, but this location would otherwise be in the user's field of view if the first display generating component were not present in front of the user (e.g., the view of the user's hand is obstructed by the first display generating component, and the representation of the user's hand is not displayed via the first display generating component). In other words, in some implementations, the user's hands perform gestures or movements in the physical environment, which are captured as input leading to changes in the 3D environment, even if the gestures and movements of the user's hands are not displayed via a representation of the user's hands in a view of the 3D environment. In some implementations, instead of a single hand, both of the user's hands are used to provide the necessary user input for navigation within the virtual 3D environment, and optionally, both hands are visually represented in a view of the virtual 3D environment.

[0153] exist Figure 7J After Figure 7KIn the first view 7206-1 of the three-dimensional environment, when gaze input 7208-a is detected at a location on a virtual frame 7152, the computer system detects movement of the user's hand 7202 in the physical environment (e.g., user hand movement corresponding to a change in hand grip). The movement of the user's hand is determined to correspond to movement of the fingers and / or wrist to form a first preset hand posture or grip (e.g., forming a fist, fully or partially closing the hand, with multiple fingers curled toward the palm and eventually resting on the palm, pinching multiple fingers together, etc.). In some embodiments, the first preset hand posture or grip corresponds to grasping, holding, pinching multiple fingers, etc. In some embodiments, the first preset hand posture or grip requires the hand to have a first preset orientation relative to the user (e.g., palm side toward the user's face, upward and / or sideways, etc.). In some embodiments, the user's hand movements include the movement of both hands toward each other and forming a first preset hand gesture or grasp (e.g., the fingers of both hands locking together, one hand wrapping the other in a closed fist, the two hands clasped together, etc.). In some embodiments, the computer system detects gaze input 7208-a on a first part of the virtual 3D environment (e.g., virtual picture frame 7152) in conjunction with (e.g., concurrently, within its threshold time window, etc.) the user's hand movements to form the first preset hand gesture or grasp; and in response, the computer system selects the virtual picture frame 7152 as the navigation target for subsequent movement requests. In some embodiments, the computer system displays visual feedback (e.g., highlighting, outlining, visual differentiation, etc.) to indicate which location or virtual object has been selected as the navigation target within the 3D environment (e.g., highlighting the virtual picture frame with an outline or spotlight, darkening the area around a virtual tree, etc.). In some implementations, in response to detecting gaze input moving to and / or pausing on a location or virtual object, while the user's hand has formed and maintained a first preset hand posture or grip, the computer system selects that virtual location or virtual object as the navigation target for movement. In some implementations, other methods of selecting a navigation target are possible. For example, the user may use a pointer or control device to select a navigation target, or use tap or click input combined with gaze input on a location or virtual object to select that location or virtual object as the navigation target for a subsequent movement request.

[0154] like Figure 7J After Figure 7KAs shown, when the user's hand maintains a first predefined hand posture or grip (e.g., grasping, pinching, two-handed grip, etc.), the computer system detects movement of the user's hand toward the user (e.g., movement 7162) (e.g., translation of the user's hand toward the user's body (e.g., due to movement of the user's arm attached to the hand around the elbow or shoulder, rotational movement around the wrist attached to the hand, or both)). Optionally, the start of movement of the user's hand in the first predefined hand posture or grip is detected while gaze input 7208-a is maintained at the selected navigation target for the movement. Figure 7K In response to detecting movement of the user's hand while the user's hand maintains a first predefined hand posture or grip, and based on determining that the selected navigation target is a virtual picture frame 7152 and that the corresponding movement includes (or is) a first movement amount (e.g., a movement 7162 representing the full range of movement the hand can make in this manner (e.g., rotation around the wrist, rotation around the elbow, rotation around the shoulder, etc.)), the computer system replaces a first view 7106-1 of the virtual 3D environment corresponding to a first viewpoint (e.g., a viewpoint corresponding to a first virtual position in the virtual 3D environment) with a second view 7106-2 of the virtual 3D environment corresponding to a second viewpoint (e.g., a viewpoint corresponding to a second virtual position in the 3D environment different from the first virtual position). Figure 7KAs shown, a second viewpoint is selected based on the position of the virtual object (i.e., virtual frame 7152) chosen as the navigation target and preset characteristics of the virtual frame 7152 (e.g., its size, the amount of available visual detail, the nature of its content, etc.). For example, the viewpoint may optionally move along a straight path between initial virtual positions toward the selected virtual object at the start of the movement request, but stops at a first distance from the selected virtual object, wherein the first distance is selected based on the characteristics of the virtual object as an appropriate viewing distance for observing the virtual object (e.g., viewing it as a whole and / or having the maximum amount of detail and clarity, etc.). As a result of the movement of the viewpoint of the current display view of the three-dimensional environment (e.g., from the first virtual position to the second virtual position), the virtual frame 7152 appears closer to the user, as if the user had moved toward the virtual frame 7152 to be closer to the virtual wall 7134. In some implementations, similar hand movements can be repeated while the gaze input remains on the virtual frame 7152, but the viewpoint will not move closer to the virtual frame 7152 because the nearest viewing position suitable for observing the virtual frame 7152 has been reached. In some implementations, the movement of the viewpoint in the virtual 3D environment simulates or mimics the movement of the user's virtual position caused by the user's hand gripping and pulling an invisible rubber band or taut rope around a virtual pulley anchored at a selected navigation target in the virtual 3D environment. In some implementations, other types of input are used to select a navigation target and trigger movement toward the virtual object or location selected as the navigation target, such as combining... Figures 7A to 7I and Figures 8 to 9 Those described.

[0155] and Figures 7J to 7K To create a contrast, Figure 7L After Figure 7M The diagram illustrates navigation toward different navigation targets according to several implementation schemes. For example... Figures 7L to 7MAs shown, when gaze input 7208-b is detected at a location on a virtual grid 7158 in the first view 7206-1 of the three-dimensional environment, the computer system detects the movement of the user's hand 7202 in the physical environment (e.g., user hand movement corresponding to a change in hand grip). The movement of the user's hand is determined to correspond to movement of the fingers and / or wrist to form a first preset hand posture or grip (e.g., forming a fist, fully or partially closing the hand, with multiple fingers curled toward the palm and eventually resting on the palm, pinching multiple fingers together, etc.). In some embodiments, the first preset hand posture or grip corresponds to grasping, holding, multi-finger pinching, etc. In some embodiments, the first preset hand posture or grip requires the hand to have a first preset orientation relative to the user (e.g., palm side toward the user's face, upward and / or sideways, etc.). In some implementations, the user's hand movements include the movement of both hands toward each other and forming a first preset hand gesture or grasp (e.g., the fingers of both hands locking together, one hand wrapping around the other in a closed fist, the two hands clasped together, etc.). In some implementations, the computer system detects gaze input 7208-b on a second part of the virtual 3D environment (e.g., the virtual grid 7158 of the virtual window 7154) to detect the user's hand movements to form the first preset hand gesture or grasp; and in response, the computer system selects the virtual grid 7158 as the navigation target for subsequent movement requests. In some implementations, the computer system displays visual feedback (e.g., highlighting, outlining, visual differentiation, etc.) to indicate which location or object has been selected as the navigation target within the 3D environment (e.g., highlighting the virtual grid with an outline or spotlight, darkening the area around a virtual tree, etc.). In some implementations, in response to detecting gaze input moving to and / or pausing on a virtual location or object, while the user's hand has formed and maintained a first preset hand posture or grip, the computer system selects that virtual location or object as the navigation target for movement. In some implementations, other methods of selecting a navigation target are possible. For example, the user may use a pointer or control device to select a navigation target, or use tap or click input combined with gaze input on a virtual location or object to select that virtual location or object as the navigation target for a subsequent movement request.

[0156] like Figure 7L After Figure 7MAs shown, when the user's hand maintains a first predefined hand posture or grip (e.g., grasping, pinching, two-handed grip, etc.), the computer system detects movement of the user's hand toward the user (e.g., movement 7162) (e.g., translation of the user's hand toward the user's body (e.g., due to movement of the user's arm attached to the hand around the elbow or shoulder, rotational movement around the wrist attached to the hand, or both)). Optionally, the start of movement of the user's hand in the first predefined hand posture or grip is detected while gaze input 7208-b is maintained at the selected navigation target for the movement. Figure 7M In response to detecting movement of the user's hand while the user's hand maintains a first predefined hand posture or grip, and based on determining that the selected navigation target is a virtual grid 7158 and that the corresponding movement includes (or is) a first movement amount (e.g., a movement 7162 representing the full range of movement the hand can make in this manner (e.g., rotation around the wrist, rotation around the elbow, rotation around the shoulder, etc.)), the computer system replaces a first view 7106-1 of the virtual 3D environment corresponding to a first viewpoint (e.g., a viewpoint corresponding to a first virtual position in the virtual 3D environment) with a third view 7106-3 of the virtual 3D environment corresponding to a third viewpoint (e.g., a viewpoint corresponding to a second virtual position in the 3D environment different from the first virtual position). Figure 7MAs shown, a third viewpoint is selected based on the position of the virtual object (i.e., virtual grid 7158) chosen as the navigation target and the characteristics of the virtual grid 7158 (e.g., its size, the amount of available visual detail, the nature of its content, etc.). For example, the viewpoint may optionally move along a straight path between initial virtual positions toward the selected virtual object at the start of the movement request, but stops at a second distance from the selected virtual object, wherein the second distance is selected based on the characteristics of the virtual object as an appropriate viewing distance for observing the virtual object (e.g., viewing it as a whole and / or having the maximum amount of detail and sharpness, etc.). As a result of the movement of the viewpoint of the current display view of the 3D environment (e.g., from the first virtual position to the second virtual position), the virtual grid 7158 appears closer to the user; as if the user had moved toward the virtual grid 7158 to be closer to the virtual wall 7134. In some implementations, a similar hand movement can be repeated while the gaze input remains on the virtual grid 7158, but the viewpoint will not move closer to the virtual grid 7158 because the nearest viewing position suitable for observing the virtual grid 7158 has been reached. In some implementations, the movement of the viewpoint in the virtual 3D environment simulates or mimics the movement of the user's virtual position caused by the user's hand gripping and pulling an invisible rubber band or taut rope around a virtual pulley anchored at a selected navigation target in the virtual 3D environment. In some implementations, other types of input are used to select a navigation target and trigger movement toward the virtual object or location selected as the navigation target, such as combining... Figures 7A to 7I and Figures 8 to 9 Those described.

[0157] exist Figure 7K and Figure 7M In this context, the appropriate distance chosen for observing the selected navigation target varies. Figure 7K In this context, the navigation target is a virtual picture frame 7154, and given the size of the virtual picture frame 7154, a suitable viewing distance is approximately arm's length. With the second viewpoint located at a first distance (e.g., arm's length) from the virtual picture frame 7154, the entire image of the virtual picture frame 7154 is centered within the field of view provided by the first display generating component, and the image within the virtual picture frame 7154 is rendered at full resolution and with complete detail. Other objects that might distract the user are outside the field of view provided by the first display generating component. Figure 7M In the navigation target, the virtual grid 7158 with details of interest 7161 is in the first view 7106-1 corresponding to the first viewpoint. Figure 7L(As shown) is not fully visible or does not present sufficient detail. In response to a request to move to the virtual grid 7158, given the size of the detail 7161 on the virtual grid 7158, the viewpoint is moved to an appropriate viewing distance of approximately 15 inches. With the third viewpoint located at a second distance (e.g., fifteen inches) from the virtual grid 7158, the specific visual feature 7161 on the virtual grid 7158 is centered in the field of view provided by the first display generating component, and the visual feature 7161, along with the virtual grid 7158, is presented at full resolution and full detail at the user's focus. The virtual grid 7158 is not fully included in the field of view provided by the first display generating component at the end of the movement because the outer portion of the virtual grid does not have additional visual features that have not yet been represented by the portion of the virtual grid within the field of view. In some embodiments, other portions of the field of view include the bases of virtual hills 7160 and virtual trees 7156, which are optionally out of focus due to their distance from the virtual position of the third viewpoint.

[0158] exist Figure 7N After Figure 7O The diagram illustrates a combination of several implementation schemes. Figures 7J to 7K and Figures 7L to 7MA similar interaction is shown in the example. In this example, when the user's hand forms a first predefined hand gesture or grip, gaze input 7208-c is detected on the virtual tree 7156 outside the virtual window 7154. The virtual tree 7156 is selected as the navigation target based on the user input. In response to a movement request to navigate to the virtual tree 7156 (e.g., hand movement 7162 when in the first predefined hand gesture or grip, or other types of user input corresponding to the movement request), given the size and height of the virtual tree 7156, the viewpoint is moved to an appropriate viewing distance (e.g., a third distance) approximately ten feet above the virtual hill 7160 (e.g., where one stands at the base of the virtual tree) and ten feet from the surface of the virtual treetop. For a fourth viewpoint (e.g., the viewpoint at the end of movement) at a third distance (e.g., ten feet, some other suitable distance, etc.) from the root and top of the virtual tree 7156, the top of the virtual tree 7156 is centered in the field of view provided by the first display generating component, and the treetop, along with a portion of the trunk, is rendered in full resolution and complete detail. The base of the virtual tree trunk is not fully included in the field of view provided by the first display generating component because the focus is on the treetop, and to render the full detail of the treetop, the distance, for example, cannot exceed ten feet from the treetop. In some embodiments, other portions of the field of view include virtual hills 7160 that are optionally out of focus due to their distance from the virtual position of the fourth viewpoint. In some embodiments, other virtual objects located between the viewpoint and the virtual position of the navigation target are optionally removed from the field of view. In some embodiments, other constraints present in the virtual 3D environment are optionally used to determine the appropriate viewing distance for the navigation target. For example, in some embodiments, in Figures 7N to 7O In the example shown, the fourth viewpoint can be located to the right of the transparent portion of the virtual window 7154 to simulate an observer viewing the virtual tree from the right side of the virtual window 7154, since the observer cannot move beyond the virtual window in the 3D environment.

[0159] In some implementations, the input and output specify a navigation target located at a corresponding position in the three-dimensional environment and trigger a navigation operation toward the navigation target. Figures 7J to 7O The inputs shown differ. For example, in some embodiments, a navigation target is selected by tapping the controller in the physical environment when the controller is pointed to a corresponding location in the physical environment corresponding to the expected navigation target. In some embodiments, a pointer that can be moved by touch input on a touch-sensitive surface is used to select a navigation target. In some embodiments, a general area can be selected based on gaze input (e.g., difficulty in holding a gaze in a precise position for a period of time), and subsequent touch or gesture input can be used to fine-tune the position of the selector to be close to or within the general area to select a navigation target within the general area.

[0160] In some implementations, the appropriate position of the viewpoint at the end of the movement is selected based on the size of the navigation target. Specifically, an appropriate viewing distance is selected that separates the position of the navigation target from the virtual position of the viewpoint facing the navigation target at the end of the movement. For example, in some cases, larger navigation targets require correspondingly larger viewing distances. In some implementations, the selected viewing distance places the viewpoint outside and separates it from the minimum bounding box of the navigation target (e.g., not on the surface of the navigation target, separated by a finite distance chosen based on the characteristics of the virtual object selected as the navigation target, etc.). In some implementations, if the navigation target is not a virtual object but simply a location in empty space, the viewing distance will be zero and the viewpoint will move directly to the selected target location.

[0161] In some implementations, the location of the viewpoint at the end of the move is selected based on the nature of the navigation target (e.g., whether it is intended to be explored within or near the navigation target, or observed and appreciated as a whole). Specifically, an appropriate viewing distance is chosen that separates the location of the navigation target from the virtual location of the viewpoint facing the navigation target at the end of the move. For example, when the selected navigation target is a virtual museum viewed from a distance, at the end of the move, the viewpoint is in front of the virtual museum's entrance and only a partial view of the museum building is within the user's field of view. When the selected navigation target is a virtual lightbox viewed from a distance, at the end of the move, the viewpoint is at a distance that provides a complete view of the virtual lightbox. In some implementations, the virtual location of the viewpoint at the end of the move is selected based on the type of object the navigation target is designed for. For example, the viewpoint chosen for a building or landmark is above the virtual ground (e.g., ten feet or more above the ground) and relatively far from the building or landmark, such that it can be considered as a whole given the limited angular span of the user's field of view. Conversely, the viewpoint chosen for street signs, flagpoles, park entrances, etc., is below the virtual ground (e.g., approximately eye level for the average user) and close to the selected navigation target (e.g., a few feet away from the street sign, flagpole, or park entrance). This is a separate consideration from the size (e.g., width and height) of the navigation target.

[0162] In some implementations, the computer system displays an animated transition of an intermediate view sequence corresponding to a sequence of viewpoints in a three-dimensional environment, between an initial viewpoint (e.g., the viewpoint where the current display viewpoint is at the initial position) and a final viewpoint after movement (e.g., the viewpoint where the current display viewpoint is at the final position), wherein the peripheral portions of the intermediate views are visually faded (e.g., blurred and / or darkened, etc.) relative to the central portions of the intermediate views to simulate a vignetting effect during viewpoint movement.

[0163] In some implementations, the simulated vignetting effect varies based on the viewpoint's movement speed. In some implementations, the viewpoint's movement speed depends on various factors, such as the distance between the viewpoint's initial position before movement and its final position at the end of movement, the speed and / or magnitude of the user input that triggers and / or sustains the movement, etc. In some implementations, the amount of change in a first set of display attributes applied to the outer portion of the corresponding intermediate view (e.g., blurriness, size of the blurred area, blur radius, amount of desaturation, degree of darkening, size of the darkened area, etc.) is adjusted according to the transition rate represented by the sequence of intermediate views (e.g., the viewpoint's movement speed of the currently displayed view, as shown in the animation, etc.). For example, for faster viewpoint movement during movement, the larger outer portion of the intermediate viewpoint becomes blurred and / or darkened relative to the central portion of the intermediate viewpoint; and for slower viewpoint movement during movement, the smaller outer portion of the intermediate viewpoint becomes blurred and / or darkened relative to the central portion of the intermediate viewpoint. In some implementations, the speed of viewpoint movement is measured by the number of frames per second displayed between a frame showing a viewpoint at a first location and a frame showing a viewpoint at a second location, which is closer to the navigation target and is a unit distance from the first location.

[0164] In some implementations, the vignetting amount is gradually increased during movement. For example, in some implementations, the size of the peripheral portion of the intermediate view, which is visually faded relative to the central portion of the intermediate view, gradually increases during at least one subsequence of the intermediate view sequence (e.g., in the initial subsequence of the intermediate view sequence).

[0165] In some implementations, the input gestures used in the various examples and implementations described herein optionally include discrete micromotor gestures (which are performed by moving the user’s fingers relative to other fingers or portions of the user’s hand), and optionally, it is not necessary to primarily move the user’s entire hand or arm away from its natural position and posture to perform the operation immediately before or during the gesture for interacting with the virtual or mixed reality environment.

[0166] In some implementations, input gestures are analyzed by a sensor system (e.g., sensor 190, ...). Figure 1 Image sensor 314, Figure 3 The sensor system uses captured data or signals for detection. In some implementations, the sensor system includes one or more imaging sensors (e.g., one or more cameras, such as motion RGB cameras, infrared cameras, depth cameras, etc.). For example, the one or more imaging sensors are computer systems (e.g., Figure 1The computer system 101 (e.g., portable electronic device 7100 or HMD) includes components that provide data to the computer system, and the computer system includes display generation components (e.g., Figure 1 , Figure 3 and Figure 4 The display generating component 120 (e.g., a touchscreen display, stereoscopic display, display with a transparent portion, etc., used as a display and touch-sensitive surface) is included. In some embodiments, the one or more imaging sensors include one or more rear-facing cameras on the side of the device opposite to the display of the device. In some embodiments, input gestures are detected by a sensor system of a head-mounted system (e.g., a VR headset including a stereoscopic display that provides a left image for the user's left eye and a right image for the user's right eye). For example, one or more cameras, as components of the head-mounted system, are mounted on the front and / or lower side of the head-mounted system. In some embodiments, one or more imaging sensors are located in the space in which the head-mounted system is used (e.g., arranged around the head-mounted system at various locations in a room), such that the imaging sensors capture images of the head-mounted system and / or the user of the head-mounted system. In some embodiments, input gestures are detected by a sensor system of a heads-up device (e.g., a heads-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics). For example, one or more imaging sensors are attached to the interior surface of a car. In some embodiments, the sensor system includes one or more depth sensors (e.g., a sensor array). For example, the one or more depth sensors may include one or more light-based (e.g., infrared) sensors and / or one or more sound-based (e.g., ultrasonic) sensors. In some embodiments, the sensor system includes one or more signal transmitters, such as light transmitters (e.g., infrared transmitters) and / or sound transmitters (e.g., ultrasonic transmitters). For example, when light (e.g., light from an array of infrared light transmitters with a predetermined pattern) is projected onto a hand (e.g., hand 7200), the one or more cameras capture an image of the hand under illumination, and the captured image is analyzed to determine the position and / or structure of the hand. Using signals from image sensors pointing towards the hand to determine input gestures, rather than using signals from touch-sensitive surfaces or other direct contact mechanisms or proximity-based mechanisms, allows the user the freedom to choose whether to perform large movements or remain relatively still when providing input gestures using his / her hand, without being subject to limitations imposed by a particular input device or input area.

[0167] In some implementations, micro-tap input is a tapping input where the user's thumb is over the index finger (e.g., on the side of the index finger adjacent to the thumb). In some implementations, tapping input is detected without requiring the thumb to be lifted from that side of the index finger. In some implementations, tapping input is detected based on determining that a downward movement of the thumb is followed by an upward movement of the thumb, wherein the thumb is in contact with that side of the index finger for less than a threshold amount of time. In some implementations, tap-hold input is detected based on determining that the thumb moves from the lifted position to the pressed position and remains in the pressed position for at least a first threshold amount of time (e.g., a tap time threshold or another time threshold longer than the tap time threshold). In some implementations, the computer system requires the hand as a whole to remain substantially still in the position for at least a first threshold amount of time in order to detect tap-hold input performed by the thumb on the index finger. In some implementations, touch-hold input is detected without requiring the hand as a whole to remain substantially still (e.g., the hand as a whole can move while the thumb rests on that side of the index finger). In some implementations, tap-hold-drag input is detected when the thumb presses against the side of the index finger and the hand as a whole moves while the thumb rests on the side of the index finger.

[0168] In some embodiments, the micro-flick gesture is a push or flick input performed by moving the thumb across the index finger (e.g., from the palmar side to the back side of the index finger). In some embodiments, the extension movement of the thumb is accompanied by an upward movement on that side away from the index finger, for example, as in an upward flick input performed by the thumb. In some embodiments, the index finger moves in the opposite direction to the thumb during the forward and upward movement of the thumb. In some embodiments, a reverse flick input is performed by moving the thumb from an extended position to a retracted position. In some embodiments, the index finger moves in the opposite direction to the thumb during the backward and downward movement of the thumb.

[0169] In some embodiments, a micro-swipe gesture is a swipe input performed by moving the thumb along the index finger (e.g., along the side of the index finger adjacent to the thumb or on that side of the palm). In some embodiments, the index finger is optionally in an extended (e.g., substantially straight) or curled state. In some embodiments, the index finger moves between an extended and curled state during the movement of the thumb in the swipe input gesture.

[0170] In some implementations, different phalanges of various fingers correspond to different inputs. Micro-tapping inputs by the thumb above the various phalanges of various fingers (e.g., index, middle, ring, and optionally, little finger) are optionally mapped to different operations. Similarly, in some implementations, different push or click inputs may be performed by the thumb across different fingers and / or different parts of the fingers to trigger different operations in the corresponding user interface context. Similarly, in some implementations, different swipe inputs performed by the thumb along different fingers and / or in different directions (e.g., towards the distal or proximal end of the finger) trigger different operations in the corresponding user interface context.

[0171] In some implementations, the computer system treats tapping, flicking, and swiping inputs as different types of input based on the type of thumb movement. In some implementations, the computer system treats inputs with different finger positions that are tapped, touched, or swipened by the thumb as different sub-input types (e.g., proximal, middle, distal sub-types, or index, middle, ring, or little finger sub-types) of a given input type (e.g., tapping, flicking, swiping, etc.). In some implementations, the amount of movement performed by moving a finger (e.g., thumb) and / or other movement metrics associated with finger movement (e.g., velocity, initial velocity, ending velocity, duration, direction, movement pattern, etc.) are used to quantitatively influence the operation triggered by finger input.

[0172] In some implementations, the computer system recognizes combined input types that combine a series of movements performed by the thumb, such as tap-swipe input (e.g., thumb pressing on a finger, followed by a swipe along that side of the finger), tap-flick input (e.g., thumb pressing on a finger, followed by a flick across the finger from the palm side to the back side), double-tap input (e.g., two consecutive taps on that side of the finger at approximately the same location), and so on.

[0173] In some implementations, gesture input is performed by the index finger instead of the thumb (e.g., the index finger performs a tap or swipe on the thumb, or the thumb and index finger move toward each other to perform a pinch gesture, etc.). In some implementations, compared to finger movement input which does not have modified input via wrist movement, wrist movement (e.g., a flick of the wrist in the horizontal or vertical direction) is performed immediately before, immediately after (e.g., within a threshold amount of time), or simultaneously with finger movement input to trigger additional, different, or modified actions in the current user interface context. In some implementations, finger input gestures performed with a user's palm facing the user's face are considered different types of gestures from those performed with a user's palm facing away from the user's face. For example, an action performed with a user's palm facing the user has increased (or decreased) privacy protection compared to an action performed in response to a tap gesture performed with a user's palm facing away from the user's face (e.g., the same action).

[0174] Although one type of finger input may be used to trigger a certain type of operation in the examples provided in this disclosure, in other embodiments, other types of finger input may optionally be used to trigger the same type of operation.

[0175] The following text is for reference only. Figures 8 to 10 Methods 8000, 9000, and 10000 provide information about Figures 7A to 7O Additional description.

[0176] Figure 8 This is a flowchart of method 8000 according to some implementation schemes, which is used to select a navigation target and navigate to the navigation target based on physical gestures.

[0177] In some implementations, in computer systems (e.g., Figure 1 Method 8000 is executed at a computer system 101, the computer system including a first display generation component (e.g., Figures 7A to 7OThe system includes a display 7100 (such as the internal display of an HMD, etc.) and one or more input devices (e.g., cameras, controllers, touch-sensitive surfaces, joysticks, buttons, gloves, watches, motion sensors, orientation sensors, etc.). In some embodiments, the first display generating component is a head-up display, a head-mounted display (HMD), a display, a touchscreen, a projector, etc. In some embodiments, the first display generating component is a user-facing display component and provides a CGR experience to the user. In some embodiments, the first display generating component is the internal display of a dual-sided display device (e.g., a dual-sided HMD) that displays a first user interface on a first side corresponding to the first display generating component and a second user interface on a second side corresponding to the second display generating component. In some embodiments, the computer system is an integrated device having one or more processors and memories enclosed in the same housing as the first display generating component, and at least some of the input devices among one or more input devices. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smartphone or tablet, a wearable device such as a watch, wristband or handset, a desktop computer, a laptop computer, etc.) that includes one or more processors and memory separate from the display generation component and / or one or more input devices. In some embodiments, the display generation component and one or more input devices are integrated and enclosed in the same housing. In some embodiments, method 8000 is performed by storing data in a non-transitory computer-readable storage medium and by one or more processors of the computer system (such as one or more processors 202 of computer system 101) (e.g., Figure 1 The control unit 110 in A executes instructions to manage the process. Some operations in method 8000 are optionally combined, and / or the order of some operations is optionally changed.

[0178] In some implementations, method 8000 is performed at a computing system including a first display generation component (e.g., head-up display, HMD, monitor, touchscreen, projector, etc.) and one or more input devices (e.g., camera, controller, touch-sensitive surface, joystick, button, etc.). The computing system displays (8002) a first view of a three-dimensional environment (e.g., virtual environment) corresponding to at least a portion of a first viewpoint via the display generation component. Figure 7A The first view shown is 7205. When displaying at least that portion of the three-dimensional environment in the first view, the computing system detects (8004) the user's hand (e.g., ...). Figure 7AThe gripping changes of the hand (7202) in the image (e.g., detecting movement of the user's thumb, fingers, wrist, hand, etc. via one or more input devices) and combined with (e.g., detecting gaze input simultaneously or within a threshold time amount (e.g., before or after)) the detection of the corresponding position pointing towards that part of the three-dimensional environment (e.g., ... Figure 7A The gaze input (e.g., the virtual treetop 7120 in the virtual treetop) Figure 7A The gaze input 7206 (e.g., detecting gaze input by tracking the user's eye movements using one or more input devices and calculating the target position of the eye gaze when the eye focuses on a location in a virtual 3D environment with a movement less than a threshold amount during a threshold time period (e.g., the eye gaze meets a preset stability threshold)) is wherein a change in the user's hand grip results in a first predefined hand grip (e.g., a full grip gesture, a grasping gesture, a pinching gesture, etc.) (e.g., when the gaze input points to the corresponding location, or within a threshold time period (e.g., before or after)). When the user's hand maintains the first predefined hand grip (e.g., ... Figure 7B When the user's hand is in a grasping position (e.g., a first predefined grasp is formed when all the fingers of the user's hand curl upwards to form a full grasp with the thumb closed over the other fingers, and the user's hand remains in a full grasping position after the fingers curl upwards), the computing system detects (8006) the corresponding movement of the user's hand (e.g., Figure 7C Movement 7130 of the hand 7202 (e.g., lateral movement toward the user's body (e.g., due to movement of the user's arm attached to the hand), or rotational movement about the wrist attached to the hand (e.g., movement of the entire hand in a full grasping gesture, optionally while the user's gaze remains in the corresponding position). Responding to when the user's hand maintains a first predefined hand grasp (e.g., as... Figure 7C As shown, (8008) a corresponding movement of the user's hand was detected: based on the determination that the corresponding position is the first position (e.g., Figure 7B The virtual table 7120 in the middle) and the corresponding movement includes (or is) the first movement amount (e.g., Figure 7C The mobile 7130 is Figure 7D (a portion of the entire range of movement 7132 in the calculation system), the calculation system will take at least that portion of the three-dimensional environment as a first view corresponding to the first viewpoint (e.g., Figure 7B View 7205 in the image is replaced with (8010) a second view of at least that portion of the three-dimensional environment corresponding to the second viewpoint (e.g., Figure 7CIn view 7207), the second viewpoint is different from the first viewpoint, wherein the selection of the second viewpoint is based on the corresponding movement of the user's hand (e.g., movement 7130) and a first position in the three-dimensional environment (e.g., in response to hand movement while the hand is holding a first predefined hand grip, the user moves closer to the first position in the three-dimensional environment that the eye is looking at); and based on determining that the corresponding position is a second position different from the first position (e.g., virtual mountaintop 7108) and the corresponding movement includes (or is) a first movement amount (e.g., movement 7130), the computing system replaces the first viewpoint corresponding to the first viewpoint of at least that part of the three-dimensional environment with (8012) a third viewpoint corresponding to the third viewpoint of at least that part of the three-dimensional environment, wherein the third viewpoint is selected based on the corresponding movement of the user's hand and the second position in the three-dimensional environment, and the third viewpoint is different from the first viewpoint and the second viewpoint (e.g., in response to hand movement while the hand is holding a first predefined hand grip, the user moves closer to the second position in the three-dimensional environment that the eye is looking at).

[0179] In some implementations, in response to detecting a corresponding movement of the user's hand while the user's hand maintains a first predefined hand grip: based on determining that the corresponding position is a first position and the corresponding movement includes (or is) a second movement amount different from the first movement amount (e.g., Figure 7D In the movement 7132), the computing system will calculate at least that part of the three-dimensional environment corresponding to a first viewpoint (e.g., Figure 7A View 7205 in the image is replaced with a fourth view of at least that part of the three-dimensional environment corresponding to the fourth viewpoint (e.g., Figure 7D(7210) wherein a fourth viewpoint is selected based on a corresponding movement of the user's hand and a first position in the three-dimensional environment, and the fourth viewpoint is different from the first and second viewpoints (e.g., if the second movement is less than the first movement, the fourth viewpoint is between the first and second viewpoints; if the second movement is greater than the first movement, the second viewpoint is between the first and fourth viewpoints); and based on determining that the corresponding position is the second position and the corresponding movement includes (or is) the second movement, the computing system replaces the first viewpoint corresponding to at least that part of the three-dimensional environment with a fifth viewpoint corresponding to at least that part of the three-dimensional environment, wherein the fifth viewpoint is selected based on a corresponding movement of the user's hand and a second position in the three-dimensional environment, and the fifth viewpoint is different from the first, third, and fourth viewpoints (e.g., if the second movement is less than the first movement, the fifth viewpoint is between the first and third viewpoints; if the second movement is greater than the first movement, the third viewpoint is between the first and fifth viewpoints). Based on determining that the corresponding position is a first position and the corresponding movement includes a second movement amount different from the first movement amount, a first view of at least that portion of the 3D environment corresponding to a first viewpoint is replaced with a fourth view of at least that portion of the 3D environment corresponding to a fourth viewpoint, selected based on the corresponding movement of the user's hand. Furthermore, based on determining that the corresponding position is a second position and the corresponding movement includes a second movement amount, the first view of at least that portion of the 3D environment corresponding to the first viewpoint is replaced with a fifth view of at least that portion of the 3D environment corresponding to a fifth viewpoint, selected based on the corresponding movement of the user's hand and the second position in the 3D environment. Additional control options are provided without cluttering the UI with additional displayed controls (e.g., additional displayed controls for replacing the first view, additional displayed controls for selecting between a third and a fourth viewpoint, etc.). Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0180] In some implementations, in response to detecting gaze input pointing to a corresponding location within a portion of the 3D environment (e.g., the corresponding location is position 1, 2, 3, 4, or 5, etc.), the computing system applies visual effects (e.g., ...) at the corresponding location. Figure 7AVisual markers 7122 on the virtual tree 7120 in the 3D environment visually distinguish the corresponding location from its surrounding area (e.g., displaying a spotlight at the corresponding location or on an object at the corresponding location, where the spotlight highlights the corresponding location pointed to by the gaze input to indicate the current location of the gaze input in the 3D environment). In response to detecting gaze input pointing to the corresponding location in a portion of the 3D environment, visual effects are applied at the corresponding location to visually distinguish the corresponding location from its surrounding area in the 3D environment, thereby providing the user with improved visual feedback (e.g., improved visual feedback that the computer system has detected user gaze input pointing to the corresponding location in a portion of the 3D environment, improved visual feedback identifying the corresponding location, etc.). Providing improved feedback enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and effectively.

[0181] In some implementations, before detecting a change in the grip of a user's hand that is receiving a first predefined grip, the computing system detects gaze input at a third location in the 3D environment, different from the corresponding location. In response to detecting gaze input at the third location, the computing system applies a visual effect at the third location in the 3D environment to visually distinguish the third location from its surrounding area. While displaying the visual effect at the third location in the 3D environment, the computing system detects movement of gaze input from the third location to the corresponding location within the 3D environment. In response to detecting movement of gaze input from the third location to the corresponding location, the computing system stops applying the visual effect at the third location and applies the visual effect to the corresponding location (e.g., moving a spotlight from the third location to the corresponding location in the 3D environment based on movement of gaze input within at least that portion of the 3D environment (e.g., following the same path as the gaze input). In response to the detection of gaze input at a third location, a visual effect is applied at the third location in the 3D environment to visually distinguish the third location from its surrounding area. In response to the detection of gaze input moving from the third location to a corresponding location, the application of the visual effect at the third location is stopped, and the visual effect is applied to the corresponding location, thereby providing the user with improved visual feedback (e.g., improved visual feedback regarding the gaze input moving from the third location to the corresponding location). Providing improved feedback enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and effectively.

[0182] In some implementations (e.g., before detecting a change in the user's hand grip, while the user's hand is maintaining a first predefined hand grip, etc.), the computing system detects a corresponding gesture, which includes a predefined movement of a first portion of the user's hand relative to a second portion of the user's hand (e.g., rubbing the user's thumb against the user's index finger, moving one or more fingers while maintaining the first predefined hand grip, etc.), and in response to detecting the corresponding gesture, the computing system offsets the corresponding application position of a visual effect in the view portion of the corresponding hand in the three-dimensional environment (e.g., selecting the direction and amount of movement of the application position of the visual effect based on the direction, speed, and / or amplitude of the corresponding gesture). For example, before forming a predefined hand grip or while maintaining a predefined hand grip, one or more gestures can be performed using the same hand to fine-tune a target position selected by gaze input, and then the hand movement can be used to navigate a viewpoint (e.g., the target position itself, an appropriate viewpoint for observing objects at the target position, etc.) towards a preset viewpoint associated with the target position. The detection includes corresponding gestures involving a predefined movement of the first part of the user's hand relative to a second part of the user's hand, and the corresponding applied position of the visual effect offset within the view portion of the 3D environment based on the corresponding gesture. Additional control options are provided without cluttering the UI with additional controls (e.g., additional controls for offsetting the applied position of the visual effect). Providing additional control options without cluttering the UI with additional controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0183] In some implementations, detecting corresponding movement of the user's hand while maintaining a first predefined hand grip includes detecting movement of the user's hand while multiple fingers (e.g., all fingers, four fingers) remain in contact with the palm (e.g., the user maintains a closed fist or a full grip). This includes detecting rotational movement of the wrist while the arm attached to the wrist remains stationary or substantially stationary (e.g., less than a threshold amount of movement), or moving the arm and wrist toward the user together. In response to detecting corresponding movement of the user's hand while maintaining a first predefined hand grip, replacing a first view of at least that portion of the three-dimensional environment corresponding to a first viewpoint with a second or third view selected based on the corresponding movement of the user's hand and a first or second position, also includes detecting movement of the user's hand while multiple fingers remain in contact with the palm, providing additional control options without cluttering the UI with additional displays (e.g., additional displays for replacing the first view with the second or third view, additional displays for selecting the second or third view, etc.). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0184] In some implementations, selecting a second or third viewpoint based on the corresponding hand movement includes selecting the second or third viewpoint based on the amount of rotation of the hand about the wrist attached to the hand during the corresponding hand movement. In some implementations, lateral movement of the hand due to movement of the user's arm or person is not considered when determining the amount of movement caused by the corresponding hand movement. In some implementations, rotational movement of the hand about the elbow due to movement of the user's arm is not considered when determining the amount of movement caused by the corresponding hand movement. Selecting a second or third viewpoint based on the amount of rotation of the hand about the wrist attached to the hand during the corresponding hand movement provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or choosing between the second and third viewpoints). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0185] In some implementations, selecting a second or third viewpoint based on the corresponding hand movement includes selecting the second or third viewpoint based on the amount of rotation of the hand around (e.g., via the forearm and wrist) the elbow attached to the hand during the corresponding hand movement. In some implementations, lateral movement of the hand due to movement of the user's arm or person is not considered when determining the amount of movement caused by the corresponding hand movement. In some implementations, rotational movement of the hand around the wrist is not considered when determining the amount of movement caused by the corresponding hand movement. Selecting a second or third viewpoint based on the amount of rotation of the hand around the elbow attached to the hand during the corresponding hand movement provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or choosing between the second and third viewpoints). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0186] In some implementations, when selecting a second or third viewpoint based on corresponding hand movements, a full-range rotational movement of the user's hand about the wrist attached to the hand results in navigation to a corresponding (e.g., pre-selected) viewpoint corresponding to the location pointed to by the gaze input (e.g., the location itself, the nearest suitable viewpoint for observing objects at the location), and different amounts of rotational movement of the user's hand about the wrist than the full-range rotational movement of the user's hand about the wrist result in different amounts of navigation toward the corresponding viewpoint corresponding to the location pointed to by the gaze input (e.g., positively correlated with the amount of rotational movement). In some implementations, different amounts of navigation are proportional to different amounts of rotational movement. In some implementations, different mappings (e.g., logarithmic mapping, exponential mapping, etc.) are used between the amount of rotational movement corresponding to the destination selected by the gaze and the amount of navigation toward the pre-selected viewpoint. Selecting a second or third viewpoint based on corresponding hand movements, where a full-range rotation of the user's hand around the wrist attached to the hand results in navigation to the corresponding viewpoint corresponding to the position pointed to by the gaze input, and different amounts of rotation of the user's hand around the wrist (less than the full-range rotation of the user's hand around the wrist) result in different amounts of navigation toward the corresponding viewpoint corresponding to the position pointed to by the gaze input, provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for selecting or choosing between the second and third viewpoints, additional displayed controls for selecting the amount of navigation toward the corresponding viewpoint). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0187] In some embodiments, detecting corresponding movement of the user's hand while the user maintains a first predefined hand grip also includes detecting one or more tapping gestures performed by the user's hand (e.g., as the user taps their hand while maintaining the predefined hand grip, the first view changes to a view corresponding to a different viewpoint). In some embodiments, one or more features of the changes in the view (e.g., the distance between the first viewpoint and subsequent viewpoints, the rate at which the first viewpoint changes to a different viewpoint, etc.) depend on one or more features of the tapping gesture (e.g., a rapid tap results in faster navigation between views, while a slower tap results in slower navigation between views). In some embodiments, the tapping gesture is, for example, a micro-gesture performed by a finger (e.g., the thumb) on another part of the same hand (e.g., on the side of the index finger, on the side of the middle finger, etc.) when other fingers are curled upward relative to the palm. In some embodiments, each tap corresponds to a fixed amount of navigation toward a selected target location (e.g., selected by gaze input and optionally adjusted by a predetermined micro-gesture detected prior to the tapping gesture), and repeated taps cause progressive navigation toward the selected target location. In some implementations, a tapping gesture on the hand begins when the finger lands on another part of the same hand and continues while the contact on the hand is maintained; and the navigation amount is based on the duration of the contact. In some implementations, the navigation speed increases from a first speed to a second speed as the contact duration exceeds a threshold duration. In some implementations, an animated transition between a first view and a subsequent view is generated based on the acceleration and / or speed of the tapping gesture. In response to detecting a corresponding movement of the user's hand while maintaining a first predefined hand grip, a first view of at least that portion of the 3D environment corresponding to a first viewpoint is replaced with a second or third view selected based on the corresponding movement of the user's hand and a first or second position, respectively. This also includes detecting one or more tapping gestures performed by the user's hand, providing additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for replacing the first view with the second or third view, additional displayed controls for selecting the second or third view, etc.). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0188] In some embodiments, detecting corresponding movements of the user's hand while the user maintains a first predefined hand grip also includes detecting one or more swipe gestures performed by the user's hand while maintaining the predefined hand grip (e.g., the user swipes forward (or backward) while maintaining the predefined hand grip (e.g., swiping the thumb on or along the side of the index finger, etc.)), and in response, changing the first view to a view corresponding to a different viewpoint. In some embodiments, one or more features of the change in the view (e.g., the distance between the first viewpoint and subsequent viewpoints, the rate at which the first viewpoint changes to a different viewpoint, etc.) depend on one or more features of the swipe gesture (e.g., a fast swipe results in faster navigation between views, while a slower swipe results in slower navigation between views). In some embodiments, the swipe gesture is, for example, a micro-gesture performed by a finger (e.g., the thumb) on another part of the same hand (e.g., on the side of the index finger, on the side of the middle finger, etc.) while other fingers are curled up relative to the palm. In some implementations, each swipe corresponds to a fixed amount of navigation toward a selected target location (e.g., selected by gaze input and optionally adjusted by a predetermined micro-gesture detected prior to the tap gesture), and repeated swipes cause progressive navigation toward the selected target location. In some implementations, a swipe gesture on the hand begins when a finger lands on another part of the same hand and continues while contact is maintained on the hand; and the navigation amount is based on the amount of movement of the finger on the hand while contact is maintained. In some implementations, an animated transition between a first view and a subsequent view is generated based on the acceleration and / or velocity of the swipe gesture. In response to detecting a corresponding movement of the user's hand while maintaining a first predefined hand grip, the system replaces a first view of at least that portion of the 3D environment corresponding to a first viewpoint with a second or third view selected based on the corresponding movement of the user's hand and a first or second position, respectively. It also includes detecting one or more swipe gestures performed by the user's hand and providing additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for replacing the first view with the second or third view, additional displayed controls for selecting the second or third view, etc.). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0189] In some implementations, detecting corresponding movement of the user's hand while the user maintains a first predefined hand grip also includes detecting movement of the user's hand toward the user while maintaining the first predefined hand grip (e.g., corresponding to a "pull-back" gesture as a first part of a "slingshot motion"); and stopping the maintenance of the first predefined hand grip after the user's hand has moved toward the user (e.g., corresponding to a release gesture as a second part of a "slingshot motion"). In some implementations, upon detecting a release motion (e.g., the cessation of the first predefined hand grip), the computer system begins navigation from a first view of the 3D environment to a subsequent view. In some implementations, the distance between the first viewpoint and the subsequent viewpoint is based on the distance the hand moved before the first predefined hand grip stopped (e.g., proportionally, otherwise positively correlated, etc.). In some implementations, how quickly the view changes from the first viewpoint to the subsequent viewpoint is based on how far the user pulls their hand back during the first part of the slingshot motion (e.g., proportionally, otherwise positively correlated, etc.). In response to detecting a corresponding movement of the user's hand while maintaining a first predefined hand grip, the system replaces a first view of at least that portion of the 3D environment corresponding to a first viewpoint with a second or third view selected based on the corresponding movement of the user's hand and a first or second position, respectively. The system also includes detecting movement of the user's hand toward the user while maintaining the first predefined hand grip and stopping the first predefined hand grip after the hand has moved toward the user. Additional control options are provided without cluttering the UI with additional displayed controls (e.g., additional displayed controls for replacing the first view with the second or third view, additional displayed controls for selecting the second or third view, etc.). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0190] In some implementations, the computing system displays an intermediate view sequence corresponding to an intermediate view sequence between a first viewpoint and a second or third viewpoint, wherein the intermediate view sequence corresponds to the movement of a virtual observer in a three-dimensional environment based on one or more movement features (e.g., rotational speed, rotational amount, or combinations thereof around the wrist, elbow, etc.; tap speed, tap duration, swipe speed, swipe distance, etc.) of a corresponding movement of the user's hand (e.g., a first or second movement amount). Displaying the intermediate view sequence corresponding to the intermediate view sequence between the first and second or third viewpoints, wherein the intermediate view sequence corresponds to the movement of a virtual observer in a three-dimensional environment based on one or more movement features of the corresponding movement of the user's hand, provides the user with improved visual feedback (e.g., improved visual feedback regarding one or more movement features of the corresponding movement of the user's hand). Providing improved feedback enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and effectively.

[0191] In some implementations, when replacing a first view of at least that portion of a 3D environment corresponding to a first viewpoint with a subsequent view of at least that portion of the 3D environment corresponding to a subsequent viewpoint (e.g., a second view, third view, fourth view, or fifth viewpoint, etc.) selected based on the corresponding position of the gaze input, the computing system displays an animated transition between the first and subsequent views, while visually fading (e.g., blurring, darkening, desaturating, etc.) the peripheral portion of the user's field of view entering the 3D space during the animated transition (e.g., occluding the peripheral field of view as the first view transitions to the subsequent view to alleviate motion sickness for the user). Displaying an animated transition between the first and subsequent views while visually fading the peripheral portion of the user's field of view entering the 3D environment during the animated transition provides the user with improved visual feedback (e.g., improved visual feedback that the computer system is replacing the first view with the subsequent view). Providing improved feedback enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and effectively.

[0192] In some implementations, replacing the first view with another view of the 3D environment corresponding to a viewpoint different from the first viewpoint (e.g., a second viewpoint, a third viewpoint, etc.) is performed based on determining that the corresponding position pointed to by the gaze input corresponds to the position of an immovable object (e.g., a representation of a physical object, a virtual object fixed to the 3D environment, etc.). In response to detecting a corresponding movement of the user's hand while the user's hand maintains a first predefined hand grip, and based on determining that the corresponding position in at least that part of the 3D environment corresponds to the position of a movable object (e.g., a virtual object not fixed to the virtual environment): the computing system maintains the display of the first view of at least that part of the 3D environment; and the computing system moves the movable object from the corresponding position toward the first viewpoint (e.g., the user appears stationary within the virtual environment, and only the object moves relative to the user). Based on the determination that a corresponding position in at least that part of the 3D environment corresponds to the position of a movable object, the display of a first view of at least that part of the 3D environment is maintained, and the movable object is moved from the corresponding position toward a first viewpoint. The movable object (instead of the first view) is moved when a set of conditions are met without further user input (e.g., further user input to select and move the movable object, further user input to select between moving the first view or moving the first movable object, etc.). Performing the operation when a set of conditions are met without further user input enhances the operability of the device, which in turn reduces power consumption and improves the device's battery life by enabling users to use the device more quickly and efficiently.

[0193] It should be understood that, Figure 8 The specific order in which the operations described herein are presented is merely an example and is not intended to indicate that the order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein relative to other methods (e.g., methods 9000 and 10000) also apply in a similar manner to the above descriptions relative to other methods described herein. Figure 8The method described above with reference to method 8000 may optionally have one or more of the features of gestures, navigation targets, anchor positions, target positions, virtual objects, views, viewpoints, virtual positions of viewpoints, movement requests, hand gestures, hand grips, gaze input, physical objects, user interface objects, intermediate views, object types, and / or animations described with reference to other methods described herein (e.g., methods 9000 and 10000). For the sake of brevity, these details will not be repeated here.

[0194] Figure 9 This is a flowchart of method 9000 according to some implementation schemes, which is used to selectively move virtual objects relative to a three-dimensional environment (and viewpoint) or move the viewpoint relative to a three-dimensional environment (and all virtual objects within the three-dimensional environment) based on a hand grip held during a user input sequence.

[0195] In some implementations, in computer systems (e.g., Figure 1 Method 9000 is executed at a computer system 101, the computer system including a first display generation component (e.g., Figures 7A to 7OThe system includes a display 7100 (such as the internal display of an HMD, etc.) and one or more input devices (e.g., cameras, controllers, touch-sensitive surfaces, joysticks, buttons, gloves, watches, motion sensors, orientation sensors, etc.). In some embodiments, the first display generating component is a head-up display, a head-mounted display (HMD), a display, a touchscreen, a projector, etc. In some embodiments, the first display generating component is a user-facing display component and provides a CGR experience to the user. In some embodiments, the first display generating component is the internal display of a dual-sided display device (e.g., a dual-sided HMD) that displays a first user interface on a first side corresponding to the first display generating component and a second user interface on a second side corresponding to the second display generating component. In some embodiments, the computer system is an integrated device having one or more processors and memories enclosed in the same housing as the first display generating component, and at least some of the input devices among one or more input devices. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smartphone or tablet, a wearable device such as a watch, wristband or handset, a desktop computer, a laptop computer, etc.) that includes one or more processors and memory separate from the display generation component and / or one or more input devices. In some embodiments, the display generation component and one or more input devices are integrated and enclosed in the same housing. In some embodiments, method 8000 is performed by storing data in a non-transitory computer-readable storage medium and by one or more processors of the computer system (such as one or more processors 202 of computer system 101) (e.g., Figure 1 The control unit 110 in A executes instructions to manage the process. Some operations in method 9000 may be optionally combined, and / or the order of some operations may be optionally changed.

[0196] In some embodiments, method 9000 is performed at a computing system including a first display generation component (e.g., head-up display, HMD, monitor, touchscreen, projector, etc.) and one or more input devices (e.g., camera, controller, touch-sensitive surface, joystick, button, etc.). The computing system displays (9002) a view (e.g., of at least a portion of a three-dimensional environment (e.g., virtual environment) corresponding to a first viewpoint via the display generation component. Figure 7E , Figure 7F and Figure 7H The view shown). A view corresponding to at least that portion of the three-dimensional environment (e.g., the view shown). Figure 7E , Figure 7F and Figure 7HWhen the view shown is obtained, the computing system detects (9004) the corresponding user input, including detecting the movement of one or both of the user's hands in the physical environment (e.g., movement of the entire hand around the user's wrist, elbow, and / or shoulder) (e.g., a single-hand two-finger pinch gesture, a single-hand full-hand grasp, both hands in a two-finger pinch gesture, both hands in a full-hand grasp, etc.). In some embodiments, detecting the corresponding user input also includes detecting the selection of a target location in the three-dimensional environment in conjunction with the movement of one or both of the user's hands (e.g., detecting gaze input pointing to the target location within a time window before, at the same time, or after this). In response to detecting (9006) the corresponding user input: based on determining that the corresponding user input meets a first criterion (e.g., one or both hands are in a full-hand grasp) (e.g. Figure 7F and Figure 7G As shown), wherein the first criterion includes a first criterion that is satisfied when one or both hands maintain a first grip (e.g., a two-finger pinch or a full grip when moving clockwise or counterclockwise around a position in a virtual object or virtual environment, or a single hand maintains a two-finger pinch or a full grip when moving laterally or toward the user, etc.) while one or both hands maintain a first grip (e.g., a two-finger pinch or a full grip, etc.), the computing system moves relative to the three-dimensional environment (9008) in the three-dimensional environment based on the movement of one or both hands in the physical environment (while maintaining a first viewpoint) (e.g., in a 3D environment and when the user remains stationary (e.g., as viewed from a first observation position) of the virtual object (e.g., the movement of the virtual object) in the physical environment (while maintaining a first viewpoint) (e.g., in a 3D environment and when the user remains stationary (e.g., as viewed from a first observation position)). Figure 7F and Figure 7G The virtual table 7142 in the 3D environment (e.g., a virtual object located at a target position selected by gaze input detected in conjunction with user hand movements). In some implementations, the virtual object may move outside a portion of the 3D environment based on hand movements (e.g., throwing). In response to detecting a corresponding user input and determining that the corresponding user input meets a second criterion different from the first criterion (e.g., a full-hand grip with one or both hands) (e.g., as...), Figures 7H to 7I(as shown in the illustration) (e.g., one or both hands are in a user grip), wherein the second criterion includes a second criterion that is satisfied when, while detecting a second type of movement of one of the user's two hands (e.g., both hands each maintaining a two-finger pinch or full grip while moving clockwise or counterclockwise around a position in a virtual object or virtual environment, or a single hand maintaining a two-finger pinch or full grip while moving laterally or toward the user, etc.), one or both hands maintain a second grip different from the first grip (e.g., a full grip, or a two-finger pinch, etc.), the computing system replaces the view of at least that part of the three-dimensional environment corresponding to the first viewpoint with (9010) a view of at least that part of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint (e.g., a viewpoint corresponding to the selected target location) (e.g., ...). Figure 7I (The view shown) (e.g., in response to hand movement, the user moves to a position in the 3D environment closer to the eye gaze; or the user moves around an anchor position or anchor object selected by gaze input or the starting position of the hand, etc.) (e.g., virtual objects remain stationary within the virtual environment, and the user's position within the virtual environment (e.g., as reflected by the viewpoint corresponding to the current view of the virtual environment) moves according to hand movement).

[0197] In some implementations, detecting that one or both of the user's hands are holding a first grasp includes detecting that the first set of fingers (e.g., the last three or four fingers, all fingers, etc.) of the corresponding hand (e.g., each hand, first hand, second hand, first hand and second hand, etc.) of the user's one or both hands are in contact with the palm of the corresponding hand (e.g., a full-hand grasp may involve grasping by closing the entire hand (e.g., similar to a closed fist of one or both hands), and a second type of movement of the hand while holding a full-hand grasp causes the object in the three-dimensional environment to move relative to the three-dimensional environment), and detecting that one or both of the user's hands are holding a first grasp. Maintaining a second grip with both hands includes detecting that a third set of fingers (e.g., thumb) of one or both hands (e.g., each hand, first hand, second hand, first hand and second hand) remains in contact with a fourth set of fingers (e.g., index finger and / or middle finger, etc.) of one or both hands. (For example, a two-finger grip may involve gripping by closing (e.g., the thumb of one or both hands) and another finger, and a first type of movement (e.g., rotation) of the hands about a common center of rotation while maintaining a two-finger grip causes the user's position (e.g., viewpoint) in the three-dimensional environment to move relative to the three-dimensional environment.) In some embodiments, the functions of the first and second grips are reversed, and movement of one or both hands while maintaining a second grip causes objects in the three-dimensional environment to move relative to the three-dimensional environment, and movement of one or both hands while maintaining a first grip causes the user's position (e.g., viewpoint) in the three-dimensional environment to change relative to the three-dimensional environment. Based on a first criterion that the user's input satisfies the requirement that the first set of fingers of one or both hands of the corresponding hand maintain contact with the palm of the corresponding hand, the virtual object in the 3D environment is moved relative to the 3D environment. Furthermore, based on a third set of fingers of one or both hands maintaining contact with a fourth set of fingers of the corresponding hand of the corresponding hand, at least that portion of the 3D environment corresponding to a first viewpoint is replaced with at least that portion of the 3D environment corresponding to a second viewpoint different from the first viewpoint. Additional control options are provided without cluttering the UI with additional displayed controls (e.g., additional displayed controls for moving virtual objects and / or replacing at least that portion of the 3D environment corresponding to the first viewpoint). Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0198] In some implementations, a first criterion is met when a user's single hand maintains a first grip while a first type of movement of the user's single hand is detected. In some implementations, a second criterion is met when a user's single hand maintains a second grip while a second type of movement of the user's single hand is detected. Based on determining that the corresponding user input meets the first criterion of maintaining a first grip while the user's single hand is detected with a first type of movement, virtual objects in the 3D environment are moved relative to the 3D environment. Based on determining that the corresponding user input meets the second criterion of maintaining a second grip while the user's single hand is detected with a second grip, at least that portion of the 3D environment corresponding to a first viewpoint is replaced with at least that portion of the 3D environment corresponding to a second viewpoint different from the first viewpoint. Additional control options are provided without cluttering the UI with additional displayed controls (e.g., additional displayed controls for moving virtual objects and / or replacing at least that portion of the 3D environment corresponding to a first viewpoint). Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0199] In some implementations, the first type of movement is the same as the second type of movement (e.g., the same movement of the user's hand while maintaining different grips results in the movement of the virtual object relative to the three-dimensional environment and the user's virtual position in the three-dimensional environment, or results in the movement of the user's virtual position in the three-dimensional environment).

[0200] In some implementations, moving virtual objects in a 3D environment based on the movement of one or both hands in the physical environment includes moving virtual objects in the 3D environment based on the movement of one hand in the physical environment while the user's single hand holds a first grip (e.g., a full-hand grip) (e.g., lateral movement, rotational movement about the wrist or elbow toward or away from the user, etc.).

[0201] In some implementations, replacing a view corresponding to a first viewpoint with a view corresponding to a second viewpoint includes a display animation transition that corresponds to moving (e.g., translating, rotating, etc.) the viewpoint of the currently displayed view of the 3D environment (e.g., the viewpoint corresponding to the user's virtual position) from the first viewpoint to the second viewpoint based on the movement of the single hand in the physical environment while the user holds a second grip (e.g., a user grip). This movement of virtual objects in the 3D environment relative to the 3D environment based on the movement of the single hand while the user holds a first grip, and the replacement of at least that portion of the view of the 3D environment corresponding to the first viewpoint with at least that portion of the view of the 3D environment corresponding to a second viewpoint different from the first viewpoint based on the movement of the single hand while the user holds a second grip, provides additional control options without cluttering the UI with additional displayed controls (e.g., controls for initiating and / or stopping the movement of virtual objects and / or replacing at least that portion of the view of the 3D environment corresponding to the first viewpoint). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0202] In some implementations, moving a virtual object in a three-dimensional environment includes translating the virtual object in the three-dimensional environment relative to a position corresponding to a first viewpoint based on the movement of a single hand while the single hand maintains a first grip. In some implementations, moving the viewpoint of the current display view of the three-dimensional environment from a first viewpoint to a second viewpoint includes translating the viewpoint based on the movement of a single hand while the single hand maintains a second grip. For example, in some implementations, the user moves their hand toward themselves, and thus, if the hand maintains a first grip during the hand movement, the computer system linearly moves the virtual object located at a selected target position toward itself in a first view of the three-dimensional environment corresponding to the first viewpoint, and if the hand maintains a second grip during the hand movement, the computer system moves the viewpoint of the current display view of the three-dimensional environment toward the selected target position in the three-dimensional environment (e.g., to the target position, to the nearest suitable viewpoint corresponding to the selected target position, etc.). The virtual object is translated in the 3D environment relative to a position corresponding to a first viewpoint based on the movement of the single hand while maintaining a first grip, and the viewpoint is translated based on the movement of the single hand while maintaining a second grip. Additional control options are provided without cluttering the UI with additional displayed controls (e.g., additional displayed controls for translating the virtual object, additional displayed controls for translating the viewpoint, additional displayed controls for selecting between translating the virtual object or the viewpoint). Providing additional control options without cluttering the UI with additional displayed controls enhances the device's operability, which in turn reduces power consumption and extends battery life by allowing users to use the device more quickly and efficiently.

[0203] In some embodiments, moving a virtual object in a 3D environment includes rotating the virtual object in the 3D environment about a position corresponding to a first viewpoint (e.g., such that the virtual object rotates in the 3D environment about the user's virtual position without moving closer to the user's virtual position, and during the rotation of the virtual object, the same side of the virtual object is shown to the user in a first view of the 3D environment). In some embodiments, moving the viewpoint of the current display view of the 3D environment from a first viewpoint to a second viewpoint includes rotating the viewpoint about an anchor position with a predetermined spatial relationship relative to the first viewpoint (e.g., in front of the position of the first viewpoint) (e.g., such that the user can see the virtual object at the target position from another angle without moving the virtual object in the 3D environment). For example, in some implementations, when a user moves their hand toward themselves clockwise or counterclockwise around a horizontal circle, and thus, if the hand maintains a first grip during the hand movement, the computer system rotates a virtual object located at the selected target position around its own central axis, such that the distal side of the virtual object moves toward the user in a first view of the 3D environment corresponding to a first viewpoint, and if the hand maintains a second grip during the hand movement, the computer system moves the viewpoint of the currently displayed view of the 3D environment along a circular track around an anchor point to a different side of the selected target position in the 3D environment. Rotating the virtual object in the 3D environment around the position corresponding to the first viewpoint based on the movement of the single hand while maintaining a first grip, and rotating the viewpoint around the anchor position in a predetermined spatial relationship relative to the first viewpoint based on the movement of the single hand while maintaining a second grip, provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for rotating the virtual object, additional displayed controls for rotating the viewpoint, additional displayed controls for selecting between rotating the virtual object or the viewpoint, etc.). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0204] In some implementations, a first criterion is met when the user's two hands maintain a first grasp while a first type of movement of the user's two hands is detected. In some implementations, a second criterion is met when the user's two hands maintain a second grasp while a second type of movement of the user's two hands is detected. Based on determining that the corresponding user input meets the first criterion requiring the user's two hands to maintain a first grasp while the first type of movement of the user's two hands is detected, virtual objects in the 3D environment are moved relative to the 3D environment. Based on determining that the corresponding user input meets the second criterion requiring the user's two hands to maintain a second grasp while the second type of movement of the user's two hands is detected, a view corresponding to a first viewpoint in at least that portion of the 3D environment is replaced with a view corresponding to a second viewpoint in at least that portion of the 3D environment, different from the first viewpoint. Additional control options are provided without cluttering the UI with additional displayed controls (e.g., additional displayed controls for moving virtual objects and / or replacing views corresponding to the first viewpoint in at least that portion of the 3D environment). Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0205] In some implementations, the first type of movement is the same as the second type of movement. In some implementations, moving a virtual object in a three-dimensional environment based on the movement of one or both hands in the physical environment includes rotating the virtual object about an axis (e.g., the x-axis, y-axis, or z-axis) based on the movement of one or both hands of the user in the physical environment while holding a first grip (e.g., relative movement of the two hands while holding a first grip, such as rotating a virtual wheel, causing the virtual object at the selected target location to rotate about its own axis, such that the other side of the virtual object faces the user). In some implementations, replacing the view corresponding to the first viewpoint with a view corresponding to the second viewpoint includes displaying an animated transition showing the rotation of the user's virtual position in the three-dimensional environment about an axis of the virtual object based on the movement of one or both hands of the user in the physical environment while holding a second grip (e.g., relative movement of the two hands while holding a second grip causes the three-dimensional environment (or the user's facing direction in the three-dimensional environment) to rotate about the axis of the object, allowing the user to see the virtual object and the three-dimensional environment from different viewing angles). Based on the movement of the user's hands in the physical environment while the user holds a first grip, virtual objects in the 3D environment are moved relative to the 3D environment; and based on the movement of the user's hands while the user holds a second grip, at least that portion of the 3D environment corresponding to a first viewpoint is replaced with at least that portion of the 3D environment corresponding to a second viewpoint different from the first viewpoint. Additional control options are provided without cluttering the UI with additional displayed controls (e.g., additional displayed controls for initiating and / or stopping the movement of virtual objects and / or replacing at least that portion of the 3D environment corresponding to the first viewpoint). Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0206] In some implementations, in response to detecting a corresponding user input and determining that the corresponding user input meets a third criterion, wherein the third criterion includes a third criterion that is met when both hands maintain a third grip (e.g., a first grip or a second grip, or a grip type different from the first and second grips, pointing towards a selected target location, etc.) while a third type of movement of the user's two hands (e.g., coordinated movement of the two hands, such as separating the two hands or moving them toward each other) is detected, the computing system adjusts the size of a virtual object in the three-dimensional environment relative to the three-dimensional environment based on the movement of the two hands in the physical environment (e.g., relative movement of the two hands). (e.g., the change in the size of the virtual object is proportional to or otherwise related to the change in the distance between the two hands in the physical environment from the time when the two hands first form a third grip in the three-dimensional environment.) Adjusting the size of the virtual object in the three-dimensional environment relative to the three-dimensional environment based on the movement of the user's two hands in the physical environment, wherein the user's two hands maintain a third grip when a third type of movement of the two hands is detected, provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for adjusting the size of the virtual object). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0207] In some implementations, detecting corresponding user input includes detecting gaze input pointing towards a part of the 3D environment by combining the detection of the movement of the user's two hands in the 3D environment. In some implementations, in response to the detection of a corresponding user input: based on determining that the corresponding user input satisfies a fourth criterion (e.g., the hand adopts a predefined grip and the gaze input points to a target location in the 3D environment), wherein the fourth criterion includes a fourth criterion that is satisfied when the user's two hands maintain a fourth grip while a fourth type of movement of the user's two hands is detected and a portion of the 3D environment corresponds to the location of a movable virtual object, the computing system rotates the movable virtual object in the 3D environment (e.g., around its own axis, relative to other objects in the 3D environment, etc.); and based on determining that the corresponding user input satisfies a fifth criterion, wherein the fifth criterion includes a first criterion that is satisfied when the user's two hands maintain a fifth grip while a fifth type of movement of the user's two hands is detected (e.g., the fifth grip may be the same as or different from the fourth grip) and a portion of the 3D environment does not correspond to a movable virtual object (e.g., the second location corresponds to an unoccupied area in the 3D environment or an immovable object in the 3D environment, etc.), the computing system rotates the 3D environment (e.g., together with all virtual objects in the 3D environment) around the user's virtual location in the 3D environment (e.g., the virtual location of the current viewpoint). For example, in some implementations, the animation transition shows an intermediate view sequence corresponding to a viewpoint sequence that has the same virtual position as the current viewpoint but gradually changes the viewing direction around the virtual position of the current viewpoint. Based on determining that the user's two hands maintain a fourth grip when a fourth type of movement of the user's two hands is detected and a portion of the 3D environment corresponds to the position of the movable virtual object, the movable virtual object is rotated in the 3D environment; and based on determining that the user's two hands maintain a fifth grip when a fifth type of movement of the user's two hands is detected, the 3D environment is rotated around the user's virtual position in the 3D environment, additional control options are provided without cluttering the UI with additional displayed controls (e.g., additional displayed controls for rotating the virtual object, additional displayed controls for rotating the 3D environment around the user's virtual position, additional displayed controls for selecting between rotating the virtual object or the 3D environment, etc.). Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0208] In some implementations, in response to detecting a corresponding user input and determining that the corresponding user input meets a sixth criterion, wherein the sixth criterion includes a sixth criterion that is met when both hands maintain a sixth grip (e.g., any of the first to fifth grips, or different grips) while detecting flexion movements of the user's two hands (e.g., by bending the hand upward around the corresponding wrist as the arm extends forward in front of the user, or by rotating the forearm upward around the elbow as the elbow is stationary in front of or near the user's side, or by rotating the entire arm in front of the user's shoulder joint in front of the user when the forearm is curled upward, with both hands raised or lowered together in front of the user) while the system is in a sixth grip. The computing system adjusts the height of a virtual object in the 3D environment based on the flexion movements of the two hands (e.g., the height of the object continues to increase as the flexion movements continue, or with each flexion movement). Adjusting the height of the virtual object in the 3D environment based on the flexion movements of the two hands provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for adjusting the height of the virtual object). Providing additional control options without cluttering the UI with additional displayed controls enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0209] In some implementations, in response to detecting a corresponding user input and determining that the corresponding user input meets a seventh criterion, wherein the seventh criterion includes a seventh criterion that is met when the user's single hand maintains a seventh grip (e.g., any one of the first to sixth grips, or different grips) while a flexion movement of the user's single hand is detected, the computing system moves a virtual object in the 3D environment toward the user's virtual position in the 3D environment (e.g., the virtual object moves closer to a first viewpoint with each flexion movement). Moving the virtual object in the 3D environment toward the user's virtual position in the 3D environment based on the determination that the corresponding user input meets the seventh criterion requiring the user's single hand to maintain a seventh grip while a flexion movement of the single hand is detected provides additional control options without cluttering the UI with additional displayed controls (e.g., additional displayed controls for moving the virtual object, additional displayed controls for selecting the direction of moving the virtual object, etc.). Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device, which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0210] It should be understood that, Figure 9The specific order in which the operations described herein are presented is merely an example and is not intended to indicate that the order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein relative to other methods (e.g., methods 8000 and 10000) also apply in a similar manner to the above descriptions relative to other methods described herein. Figure 9 The method described above with reference to method 9000 may optionally have one or more of the features of gestures, navigation targets, anchor positions, target positions, virtual objects, views, viewpoints, virtual positions of viewpoints, movement requests, hand gestures, hand grips, gaze inputs, physical objects, user interface objects, intermediate views, object types, and / or animations described with reference to other methods described herein (e.g., methods 8000 and 10000). For the sake of brevity, these details will not be repeated here.

[0211] Figure 10 This is a flowchart of method 10000 according to some implementation schemes, which is used to move a viewpoint toward a selected navigation target in response to a motion request, wherein the position of the viewpoint is selected based on the location of the navigation target and one or more other features of the navigation target.

[0212] In some implementations, in computer systems (e.g., Figure 1 Method 10000 is executed at a computer system 101, the computer system including a first display generation component (e.g., Figures 7A to 7OThe system includes a display 7100 (such as the internal display of an HMD, etc.) and one or more input devices (e.g., cameras, controllers, touch-sensitive surfaces, joysticks, buttons, gloves, watches, motion sensors, orientation sensors, etc.). In some embodiments, the first display generating component is a head-up display, a head-mounted display (HMD), a display, a touchscreen, a projector, etc. In some embodiments, the first display generating component is a user-facing display component and provides a CGR experience to the user. In some embodiments, the first display generating component is the internal display of a dual-sided display device (e.g., a dual-sided HMD) that displays a first user interface on a first side corresponding to the first display generating component and a second user interface on a second side corresponding to the second display generating component. In some embodiments, the computer system is an integrated device having one or more processors and memories enclosed in the same housing as the first display generating component, and at least some of the input devices among one or more input devices. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smartphone or tablet, a wearable device such as a watch, wristband or handset, a desktop computer, a laptop computer, etc.) that includes one or more processors and memory separate from the display generation component and / or one or more input devices. In some embodiments, the display generation component and one or more input devices are integrated and enclosed in the same housing. In some embodiments, method 8000 is performed by storing data in a non-transitory computer-readable storage medium and by one or more processors of the computer system (such as one or more processors 202 of computer system 101) (e.g., Figure 1 The control unit 110 in A executes instructions to manage the process. Some operations in method 10000 may be optionally combined, and / or the order of some operations may be optionally changed.

[0213] In some embodiments, method 10000 is performed at a computing system including a first display generation component (e.g., head-up display, HMD, display, touchscreen, projector, etc.) and one or more input devices (e.g., camera, controller, touch-sensitive surface, joystick, button, etc.). The computing system displays (10002) a first view (e.g., [image of a first viewpoint]) of at least a portion of a three-dimensional environment (e.g., virtual environment) corresponding to a first viewpoint via the display generation component. Figure 7J , Figure 7L and Figure 7N (The view shown). When displaying a first view corresponding to a first viewpoint of at least that portion of the three-dimensional environment, the computing system detects (10004) a set of one or more first user inputs specifying a navigation target located at the corresponding position in the three-dimensional environment and triggers a navigation operation toward the navigation target (e.g., gaze input 7208 and in). Figure 7K, Figure 7M and Figure 7O (e.g., when the controller points to a corresponding location in the physical environment corresponding to a corresponding target location, the corresponding target location is selected by clicking the controller in the physical environment; or a gaze is used to select a virtual object or location in the virtual environment as the target object or location, and a gesture is executed to trigger navigation toward the target virtual object or target location, etc.). In response to detecting a set of one or more first user inputs, the computing system displays (10006) an updated view of at least that portion of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint by displaying generated components (e.g., gestures detected in the physical environment). Figure 7K , Figure 7M and Figure 7O The second viewpoint is located closer to the corresponding position of the navigation target than the first viewpoint, but spaced apart from the corresponding position of the navigation target. The second viewpoint is selected based on predefined characteristics of the navigation target (e.g., virtual frame 7152, virtual grid 7158, virtual tree 7156, etc.) (e.g., whether it is a virtual object or location or the size, dimension, properties, etc. of the virtual object or location) and in combination with the corresponding position of the navigation target (e.g., the second viewpoint is closer to the corresponding position of the navigation target than the first viewpoint, but offset from the corresponding position in a manner based on the predefined characteristics of the navigation target).

[0214] In some implementations, sele...

Claims

1. A method comprising: In a computer system that includes a display generation component and one or more input devices: The display generating component displays a first view of at least a portion of the three-dimensional environment, corresponding to a first viewpoint of the user of the computer system; When displaying the first view of at least the portion of the three-dimensional environment, a change in the user's hand grip is detected in conjunction with the detection of gaze input pointing to a corresponding position in the portion of the three-dimensional environment, wherein the change in the user's hand grip is obtained as a first predefined hand grip. While the user's hand maintains the first predefined hand grip, detect the corresponding movement of the user's hand; and In response to detecting the corresponding movement of the user's hand while the user's hand maintains the first predefined hand grip: Based on determining that the corresponding position is a first position and the corresponding movement includes a first movement amount, the first view of at least the portion of the three-dimensional environment corresponding to the first viewpoint is replaced with a second view of at least the portion of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint, wherein the second viewpoint is selected based on the corresponding movement of the user's hand and the first position in the three-dimensional environment; as well as Based on determining that the corresponding position is a second position different from the first position and that the corresponding movement includes the first movement amount, the first view of at least said portion of the three-dimensional environment corresponding to the first viewpoint is replaced with a third view of at least said portion of the three-dimensional environment corresponding to a third viewpoint, wherein the third viewpoint is selected based on the corresponding movement of the user's hand and the second position in the three-dimensional environment, and the third viewpoint is different from the first viewpoint and the second viewpoint.

2. The method according to claim 1, further comprising: In response to detecting the corresponding movement of the user's hand while the user's hand maintains the first predefined hand grip: Based on determining that the corresponding position is the first position and that the corresponding movement includes a second movement amount different from the first movement amount, the first view of at least the portion of the three-dimensional environment corresponding to the first viewpoint is replaced with a fourth view of at least the portion of the three-dimensional environment corresponding to the fourth viewpoint, wherein the fourth viewpoint is selected based on the corresponding movement of the user's hand and the first position in the three-dimensional environment, and the fourth viewpoint is different from the first viewpoint and the second viewpoint; as well as Based on determining that the corresponding position is the second position and that the corresponding movement includes the second movement amount, the first view of at least said portion of the three-dimensional environment corresponding to the first viewpoint is replaced with a fifth view of at least said portion of the three-dimensional environment corresponding to the fifth viewpoint, wherein the fifth viewpoint is selected based on the corresponding movement of the user's hand and the second position in the three-dimensional environment, and the fifth viewpoint is different from the first viewpoint, the third viewpoint, and the fourth viewpoint.

3. The method according to claim 1, comprising: In response to detecting gaze input pointing to a corresponding location within the portion of the three-dimensional environment, visual effects are applied at the corresponding location to visually distinguish the corresponding location from its surrounding area within the three-dimensional environment.

4. The method according to claim 3, further comprising: Before detecting the change in the user's hand grip that is receiving the first predefined hand grip, the gaze input at a third position in the three-dimensional environment that is different from the corresponding position is detected; In response to detecting the gaze input at the third location, the visual effect is applied at the third location in the three-dimensional environment to visually distinguish the third location from its surrounding area in the three-dimensional environment; When the visual effect is displayed at the third location in the three-dimensional environment, the movement of the gaze input from the third location to the corresponding location in the three-dimensional environment is detected; as well as In response to detecting the movement of the gaze input from the third position to the corresponding position, the application of the visual effect at the third position is stopped and the visual effect is applied to the corresponding position.

5. The method according to claim 3, comprising: The detection includes a corresponding gesture involving a predefined movement of a first portion of the user's hand relative to a second portion of the user's hand; as well as In response to detecting the corresponding gesture, the corresponding application position of the visual effect is offset within the portion of the first view of the three-dimensional environment according to the corresponding gesture.

6. The method according to any one of claims 1 to 5, wherein detecting the corresponding movement of the user's hand while maintaining the first predefined hand grip includes detecting the movement of the user's hand while multiple fingers of the user's hand remain in contact with the palm of the user's hand.

7. The method according to any one of claims 1 to 5, wherein selecting the second viewpoint or the third viewpoint based on the corresponding movement of the user's hand comprises selecting the second viewpoint or the third viewpoint based on the amount of rotation of the user's hand about the wrist attached to the user's hand during the corresponding movement of the user's hand.

8. The method according to any one of claims 1 to 5, wherein selecting the second viewpoint or the third viewpoint based on the corresponding movement of the user's hand comprises selecting the second viewpoint or the third viewpoint based on the amount of rotation of the user's hand about the elbow attached to the user's hand during the corresponding movement of the user's hand.

9. The method according to any one of claims 1 to 5, wherein, When the second viewpoint or the third viewpoint is selected based on the corresponding movement of the user's hand, a full-range rotational movement of the user's hand about the wrist attached to the user's hand results in navigation to the corresponding viewpoint corresponding to the corresponding position pointed to by the gaze input, and a different amount of rotational movement of the user's hand about the wrist than the full-range rotational movement of the user's hand about the wrist results in a different amount of navigation toward the corresponding viewpoint corresponding to the corresponding position pointed to by the gaze input.

10. The method according to any one of claims 1 to 5, wherein detecting the corresponding movement of the user's hand while the user's hand maintains the first predefined hand grip further includes detecting one or more tapping gestures performed by the user's hand.

11. The method according to any one of claims 1 to 5, wherein detecting the corresponding movement of the user's hand while the user's hand maintains the first predefined hand grip further includes detecting one or more swiping gestures performed by the user's hand.

12. The method according to any one of claims 1 to 5, wherein detecting the corresponding movement of the user's hand while the user's hand maintains the first predefined hand grip further comprises: Detecting the movement of the user's hand toward the user while maintaining the first predefined hand grip; as well as The first predefined hand grip is stopped after the user's hand moves toward the user.

13. The method according to any one of claims 1 to 5, comprising: Displays an intermediate view sequence corresponding to an intermediate view sequence between the first viewpoint and the second or third viewpoint, wherein the intermediate view sequence corresponds to the movement of one or more movement features of a virtual observer in the three-dimensional environment based on the corresponding movement of the user's hand.

14. The method according to any one of claims 1 to 5, comprising: When replacing the first view of at least said portion of the three-dimensional environment corresponding to the first viewpoint with a subsequent view of at least said portion of the three-dimensional environment corresponding to a subsequent viewpoint closer to the target position, selected according to the corresponding position of the gaze input, an animated transition is displayed between the first view and the subsequent view, while the peripheral portion of the user's field of view entering the three-dimensional space is visually faded during the animated transition between the first view and the subsequent view.

15. The method according to any one of claims 1 to 5, wherein replacing the first view with another view of the three-dimensional environment corresponding to a viewpoint different from the first viewpoint is performed based on determining that the corresponding position pointed to by the gaze input corresponds to the position of an immovable object, and wherein the method comprises: In response to detecting the corresponding movement of the user's hand while the user's hand maintains the first predefined hand grip: Based on the determination that the corresponding position in at least the portion of the three-dimensional environment corresponds to the position of the movable object: Maintain the display of at least the portion of the first view of the three-dimensional environment; as well as Move the movable object from the corresponding position toward the first viewpoint.

16. A computer system, comprising: Display generated components; One or more input devices; One or more processors; and A memory storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for performing the following operations: The display generating component displays a first view corresponding to at least a portion of the three-dimensional environment from a first viewpoint; When displaying the first view of at least the portion of the three-dimensional environment, a change in the user's hand grip is detected in conjunction with the detection of gaze input pointing to a corresponding position in the portion of the three-dimensional environment, wherein the change in the user's hand grip is obtained as a first predefined hand grip. While the user's hand maintains the first predefined hand grip, detect the corresponding movement of the user's hand; and In response to detecting the corresponding movement of the user's hand while the user's hand maintains the first predefined hand grip: Based on determining that the corresponding position is a first position and the corresponding movement includes a first movement amount, the first view of at least the portion of the three-dimensional environment corresponding to the first viewpoint is replaced with a second view of at least the portion of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint, wherein the second viewpoint is selected based on the corresponding movement of the user's hand and the first position in the three-dimensional environment; as well as Based on determining that the corresponding position is a second position different from the first position and that the corresponding movement includes the first movement amount, the first view of at least said portion of the three-dimensional environment corresponding to the first viewpoint is replaced with a third view of at least said portion of the three-dimensional environment corresponding to a third viewpoint, wherein the third viewpoint is selected based on the corresponding movement of the user's hand and the second position in the three-dimensional environment, and the third viewpoint is different from the first viewpoint and the second viewpoint.

17. A computer system, comprising: Display generated components; One or more input devices; One or more processors; and A memory that stores one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for performing any one of the methods according to claims 1 to 15.

18. A computer-readable storage medium storing one or more programs, said one or more programs including instructions that, when executed by a computer system including display generation components and one or more input devices, cause the computer system to perform any one of the methods according to claims 1 to 15.

19. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1-15.

Images