Method for displaying, selecting and moving objects and containers in environment

Abstract

In some embodiments, a computer system performs different object selection related operations. In some embodiments, the computer system places one or more objects at a location in the display area based on the attention of the user. In some embodiments, a computer system displays a container virtual object having a curvature in a three-dimensional environment.

Description

[0001] This application is a divisional application of the application with international application number PCT / US2023 / 060592, international application date January 12, 2023, entry into the Chinese national phase date September 11, 2024, national application number 202380027098.7, and invention name “Method for displaying, selecting and moving objects and containers in an environment”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 63 / 266,726, filed on January 12, 2022, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0004] The present disclosure generally relates to computer systems that provide computer-generated experiences, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via displays. Background Art

[0005] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays) for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Example virtual elements include virtual objects such as digital images, videos, text, icons, and control elements (such as buttons and other graphics). Summary of the invention

[0006] Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve desired results in augmented reality environments, and systems that make virtual object manipulation complex, cumbersome, and error-prone can impose a huge cognitive burden on users and detract from the experience of virtual / augmented reality environments. In addition, these methods take longer than necessary, wasting energy on the computer system. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, there is a need for computer systems with improved methods and interfaces to provide computer-generated experiences to users, so that the user's interaction with the computer system is more efficient and more intuitive to the user. Such methods and interfaces optionally supplement or replace conventional methods for providing extended reality experiences to users. Such methods and interfaces reduce the amount, degree and / or nature of inputs from the user by helping the user understand the connection between the inputs provided and the device's response to those inputs, thereby forming a more effective human-computer interface.

[0008] The above-mentioned defects and other problems associated with the user interface of the computer system are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or a head-mounted device). In some embodiments, the computer system has a touch pad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, in addition to the display generation component, the computer system also has one or more output devices, which include one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI through contact and gestures of a stylus and / or finger on a touch-sensitive surface, movement of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body (as captured by a camera and other motion sensors), and / or voice input (as captured by one or more audio input devices). In some embodiments, the functions performed by interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, testing support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a transient and / or non-transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0009] There is a need for electronic devices with improved methods and interfaces for interacting with content in a three-dimensional environment. Such methods and interfaces can supplement or replace conventional methods for interacting with content in a three-dimensional environment. Such methods and interfaces reduce the amount, degree, and / or nature of input from a user and produce a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charges.

[0010] In some embodiments, the computer system performs different object selection related operations. In some embodiments, the computer system places the object at a location in the display area based on the user's attention. In some embodiments, the computer system displays a container virtual object with curvature in a three-dimensional environment.

[0011] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not comprehensive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art based on the drawings, the specification, and the claims. In addition, it should be noted that the language used in this specification is selected in principle for readability and instructional purposes, and may not be selected to describe or define the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0013] Figure 1 is a block diagram illustrating an operating environment of a computer system for providing an XR experience according to some embodiments.

[0014] Figure 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience according to some embodiments.

[0015] Figure 3 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of an XR experience to a user according to some embodiments.

[0016] Figure 4 is a block diagram illustrating a hand tracking unit of a computer system configured to capture gesture input from a user according to some embodiments.

[0017] Figure 5 is a block diagram illustrating an eye tracking unit of a computer system configured to capture gaze input of a user according to some embodiments.

[0018] Figure 6is a flow chart illustrating a flash-assisted gaze tracking pipeline according to some embodiments.

[0019] FIG. 7A to FIG. 7G Examples of computer systems that perform various object selection-related operations are shown according to some embodiments.

[0020] FIG. 8A to FIG. 8K is a flowchart illustrating an exemplary method for performing different object selection-related operations according to some embodiments.

[0021] 9A to 9F An example of a computer system that places objects at locations in a display area based on a user's attention is shown according to some embodiments.

[0022] FIG. 10A to FIG. 10J is a flowchart illustrating a method for placing an object at a location in a display area based on a user's attention according to some embodiments.

[0023] FIG. 11A to FIG. 11G An example of a computer system that displays a container virtual object having curvature in a three-dimensional environment is shown according to some embodiments.

[0024] FIG. 12A to FIG. 12L is a flow chart illustrating a method of displaying a container virtual object having curvature in a three-dimensional environment according to some embodiments. DETAILED DESCRIPTION

[0025] According to some embodiments, the present disclosure relates to a user interface for providing an extended reality (XR) experience to a user.

[0026] The systems, methods, and GUIs described herein provide improved ways for electronic devices to facilitate interaction with and manipulation of objects in a three-dimensional environment.

[0027] In some embodiments, a computer system displays virtual objects in an environment. In some embodiments, one or more of the virtual objects are selected and controlled by a first hand of a user. In some embodiments, an input directed to a corresponding object other than the one or more objects is detected from a second hand of the user. If the input from the second hand has a first characteristic, the corresponding object is added to the one or more objects selected and controlled by the first hand. If the input from the second hand has a second characteristic, the corresponding object is controlled by the second hand separately from the control of the one or more objects by the first hand.

[0028] In some embodiments, the computer system receives input to insert a virtual object into a zone, such as a zone in an environment. In some embodiments, the computer system displays the virtual object at a location in the zone based on where the user's attention was directed when the input was received. In some embodiments, if the location of the user's attention is occupied by another object, the computer system selectively offsets the location where the inserted virtual object is displayed.

[0029] In some embodiments, a computer system displays a virtual object in an environment. In some embodiments, a computer system receives an input to move a virtual object to a specific location in the environment. In some embodiments, if the specific location corresponds to another object (e.g., physical or virtual) in the environment, the computer system displays the virtual object at the specific location with a first curvature. In some embodiments, if the specific location does not correspond to another object in the environment, the computer system displays the virtual object at the specific location with a second curvature. In some embodiments, the curvature of the virtual object is additionally or alternatively based on the distance of the virtual object from one or more viewpoints, how many viewpoints are sharing the environment at the same time, and / or the relative position of one or more viewpoints relative to the virtual object.

[0030] Figures 1 to 6 A description of an example computer system for providing an XR experience to a user (such as described below with respect to methods 800 , 1000 , and / or 1200 ) is provided. FIG. 7A to FIG. 7G Examples of computer systems that perform various object selection-related operations are shown according to some embodiments. FIG. 8A to FIG. 8K is a flowchart illustrating an exemplary method for performing different object selection-related operations according to some embodiments. FIG. 7A to FIG. 7G The user interface in FIG. 8A to FIG. 8K process. 9A to 9F An example of a computer system that places objects at locations in a display area based on a user's attention is shown according to some embodiments. FIG. 10A to FIG. 10J is a flowchart illustrating a method for placing an object at a location in a display area based on a user's attention according to some embodiments. 9A to 9F The user interface in Figures 10A to 10J process. FIG. 11A to FIG. 11G An example of a computer system that displays a container virtual object having curvature in a three-dimensional environment is shown according to some embodiments. FIG. 12A to FIG. 12L is a flow chart illustrating a method of displaying a container virtual object having curvature in a three-dimensional environment according to some embodiments. FIG. 11A to FIG. 11G The user interface in FIG. 12A to FIG. 12L process.

[0031] The processes described below enhance the operability of the device and make the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device) through various techniques, including by providing improved visual feedback to the user, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional display controls, performing an operation without further user input when a set of conditions have been met, improving privacy and / or security, and / or additional techniques. These techniques also reduce power usage and extend the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0032] In addition, in the method described herein where one or more steps depend on one or more conditions being met, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions for determining the steps in the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then the ordinary technician will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on one or more conditions being met can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether the possible situation has been met without explicitly repeating the steps of the method until all conditions for determining the steps in the method have been met. It will also be understood by ordinary technicians in the art that, similar to the method with the contingent step, the system or computer-readable storage medium can repeat the steps of the method as many times as needed to ensure that all the contingent steps have been performed.

[0033] In some embodiments, such as Figure 1As shown in , an XR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a household appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., in a head mounted device or a handheld device).

[0034] In describing an XR experience, various terms are used to distinguishably refer to several related but distinct environments that a user can sense and / or with which the user can interact (e.g., using inputs detected by the computer system 101 generating the XR experience, which inputs cause the computer system generating the XR experience to generate audio, visual, and / or tactile feedback corresponding to the various inputs provided to the computer system 101). The following is a subset of these terms:

[0035] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0036] Extended Reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In XR, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more features of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one law of physics. For example, an XR system may detect a person's head turn, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to the characteristics of virtual objects in the XR environment may be made in response to a representation of physical movement (e.g., a voice command). People can sense and / or interact with XR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects, which create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some XR environments, a person can sense and / or interact only with an audio object.

[0037] Examples of XR include virtual reality and mixed reality.

[0038] Virtual Reality: A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0039] Mixed Reality: In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment that is designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between, but not including, a fully physical environment at one end and a virtual reality environment at the other end. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render MR environments can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment, or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0040] Examples of mixed reality include augmented reality and augmented virtuality.

[0041] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. A person uses the system to indirectly view the physical environment via an image or video of the physical environment and perceives virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, such as as a hologram or on a physical surface, so that a person using the system perceives virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not real version of the original captured image. For another example, the representation of the physical environment may be transformed by graphically eliminating a portion thereof or blurring a portion thereof.

[0042] Augmented Virtual: An augmented virtual (AV) environment refers to a simulated environment in which a virtual environment or a computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0043] Viewpoint-locked virtual objects: When a computer system displays a virtual object at the same position and / or location in a user's viewpoint, the virtual object is viewpoint-locked even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the forward direction of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); thus, without moving the user's head, the user's viewpoint remains fixed even when the user's gaze shifts. In embodiments where the computer system has a display generation component (e.g., a display screen) that is repositionable relative to the user's head, the user's viewpoint is an augmented reality view presented to the user on the display generation component of the computer system. For example, a viewpoint-locked virtual object that is displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or location of the viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, such that the virtual object is also referred to as a "head-locked virtual object."

[0044] Environment-locked visual objects: A virtual object is environment-locked (alternatively, "world-locked") when a computer system displays a virtual object at a position and / or location in a user's viewpoint that is based on (e.g., selected with reference to and / or anchored to) a location and / or object in a three-dimensional environment (e.g., a physical environment or a virtual environment). As the user's viewpoint moves, the location and / or objects in the environment change relative to the user's viewpoint, which causes the environment-locked virtual object to be displayed at a different location and / or location in the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user is displayed at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) such that the tree is now to the left of center in the user's viewpoint (e.g., the tree location in the user's viewpoint shifts), the environment-locked virtual object locked to the tree is displayed to the left of center in the user's viewpoint. In other words, the position and / or orientation of the virtual object that is locked to the environment in the user's viewpoint depends on the position and / or orientation of the object in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference system (e.g., a coordinate system anchored to a fixed position and / or object in the physical environment) to determine the position of the virtual object that is locked to the user's viewpoint. The virtual object that is locked to the environment can be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object), or can be locked to a movable part of the environment (e.g., a vehicle, animal, person, or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm, or foot of the user) so that the virtual object moves as the viewpoint or that part of the environment moves to maintain a fixed relationship between the virtual object and that part of the environment.

[0045] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits an inertial following behavior that reduces or delays the movement of the environment-locked or viewpoint-locked virtual object relative to the movement of the reference point followed by the virtual object. In some embodiments, when exhibiting inertial following behavior, when detecting the movement of a reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 cm and 300 cm from the viewpoint) that the virtual object is following, the computer system intentionally delays the movement of the virtual object. For example, when the reference point (e.g., the portion of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits inertial following behavior, the device ignores small amounts of movement of the reference point (e.g., ignoring movements of the reference point below a threshold movement amount, such as moving 0 to 5 degrees or moving 0 to 50 cm). For example, when the reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and when the reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a second amount greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., a “lazy follow” threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some embodiments, maintaining a substantially fixed position of the virtual object relative to the reference point includes displaying the virtual object within a threshold distance (e.g., 1 cm, 2 cm, 3 cm, 5 cm, 15 cm, 20 cm, 50 cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the position of the reference point).

[0046] Hardware: There are many different types of electronic systems that enable a person to sense various XR environments and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet devices, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to become selectively opaque. The projection-based system can employ retinal projection technology that projects graphic images onto a person's retina. The projection system can also be configured to project virtual objects into a physical environment, such as as a hologram or on a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. The following description with respect to Figure 2The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device that is located locally or remotely relative to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. As another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to the display generation component 120 (e.g., HMD, display, projector, touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included within a housing (e.g., a physical housing) of the display generating component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors, etc.), one or more input devices of the input devices 125, one or more output devices of the output devices 155, one or more sensors of the sensors 190, and / or one or more peripheral devices of the peripheral devices 195, or shares the same physical housing or support structure with one or more of the above devices.

[0047] In some embodiments, the display generation component 120 is configured to provide an XR experience (e.g., at least the visual component of the XR experience) to the user. In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Display generation component 120 is described in further detail. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0048] According to some embodiments, display generation component 120 provides an XR experience to the user when the user is virtually and / or physically present within scene 105.

[0049] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on his / her head, on his / her hand, etc.). In this way, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smart phone or tablet device) configured to present XR content, and the user holds a device with a display facing the user's field of view and a camera facing the scene 105. In some embodiments, the handheld device is optionally placed in a shell worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is an XR room, shell, or room configured to present XR content, wherein the user does not wear or hold the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions occurring in the space in front of a handheld device or a tripod-mounted device may similarly be implemented with an HMD, where the interactions occur in the space in front of the HMD and responses to the XR content are displayed via the HMD. Similarly, a user interface showing interactions with XR content triggered based on movement of a handheld device or a tripod-mounted device relative to a physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)) may similarly be implemented with an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)).

[0050] Despite Figure 1 Relevant features of operating environment 100 are shown, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the example embodiments disclosed herein.

[0051] Figure 2is a block diagram of an example of a controller 110 according to some embodiments. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), IEEE802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0052] In some embodiments, the one or more communication buses 204 include circuits that interconnect and control communications between system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0053] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 230 and an XR experience module 240.

[0054] The operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0055] In some embodiments, the data acquisition unit 241 is configured to obtain Figure 1 120, and optionally acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.) from one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data acquisition unit 241 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0056] In some embodiments, tracking unit 242 is configured to map scene 105 and track at least display generation component 120 relative to Figure 1 105, and optionally relative to the position / location of one or more of the tracked input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more parts of the user's hand, and / or the position / location of one or more parts of the user's hand relative to the user's hand. Figure 1 The movement of the scene 105 relative to the display generation component 120 and / or relative to a coordinate system (the coordinate system is defined relative to the user's hand). Figure 4 The hand tracking unit 244 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the position or movement of the user's gaze (or more generally, the user's eyes, face, or head) relative to the scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hands)) or relative to the XR content displayed via the display generation component 120. Figure 5 The eye tracking unit 243 is described in more detail.

[0057] In some embodiments, coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by display generation component 120, and optionally by one or more of output device 155 and / or peripherals 195. To this end, in various embodiments, coordination unit 246 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0058] In some embodiments, data transfer unit 248 is configured to transfer data (e.g., presentation data, position data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data transfer unit 248 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0059] Although the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 are illustrated as residing on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 may be located in separate computing devices.

[0060] also, Figure 2 It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0061] Figure 31 is a block diagram of an example of a display generation component 120 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the display generation component 120 (e.g., an HMD) includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0062] In some embodiments, one or more communication buses 304 include circuits for interconnecting and controlling communications between various system components. In some embodiments, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0063] In some embodiments, one or more XR displays 312 are configured to provide an XR experience to the user. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more XR displays 312 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the display generation component 120 (e.g., HMD) includes a single XR display. In another example, the display generation component 120 includes an XR display for each eye of the user. In some embodiments, one or more XR displays 312 can present MR and VR content. In some embodiments, one or more XR displays 312 can present MR or VR content.

[0064] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to the user's hands and, optionally, at least a portion of the user's arms (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to a scene that the user would see in the absence of the display generating component 120 (e.g., an HMD) (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0065] The memory 320 includes a high-speed random access memory, such as a DRAM, SRAM, DDR RAM, or other random access solid-state memory device. In some embodiments, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 330 and an XR rendering module 340.

[0066] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR rendering module 340 is configured to present XR content to a user via one or more XR displays 312. To this end, in various embodiments, the XR rendering module 340 includes a data acquisition unit 342, an XR rendering unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0067] In some embodiments, the data acquisition unit 342 is configured to at least Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For the purposes described, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0068] In some embodiments, the XR rendering unit 344 is configured to render XR content via one or more XR displays 312. For such purposes, in various embodiments, the XR rendering unit 344 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0069] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which a computer-generated object can be placed to generate an extended reality) based on the media content data. For such purposes, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0070] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, position data, etc.) to at least the controller 110, and optionally one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For such purposes, in various embodiments, the data transmission unit 348 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0071] Although the data acquisition unit 342, the XR rendering unit 344, the XR map generation unit 346, and the data transfer unit 348 are shown as residing on a single device (e.g., Figure 1 , but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0072] also, Figure 3 More as a functional description of various features that may be present in a particular implementation, as opposed to a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0073] Figure 4 is a schematic illustration of an example implementation of the hand tracking device 140. In some implementations, the hand tracking device 140 ( Figure 1 ) is controlled by the hand tracking unit 244 ( Figure 2 ) to track the position / location of one or more parts of a user's hand and / or one or more parts of a user's hand relative to Figure 1The hand tracking device 140 can be used to monitor movement of the scene 105 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system (which is defined relative to the user's hands)). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to the head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0074] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a human user's hand 406. The image sensor 404 captures hand images with sufficient resolution to enable the fingers and their corresponding positioning to be distinguished. The image sensor 404 typically captures images of other parts of the user's body, and also or possibly all parts of the body, and may have zoom capabilities or a dedicated sensor with increased magnification to capture images of the hand with a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of the scene 105, or is used as an image sensor to capture the physical environment of the scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a manner that the field of view of the image sensor 404 or a portion thereof is used to define an interaction space, in which the hand movements captured by the image sensor are considered as input to the controller 110.

[0075] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D mapping data (and in addition, possible color image data) to the controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided to an application running on the controller via an application program interface (API), which drives the display generation component 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving his hand 406 and changing his hand posture.

[0076] In some embodiments, the image sensor 404 projects a speckled pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral offset of the spots in the pattern. This method is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. The method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In the present disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis, so that the depth coordinates of the points in the scene correspond to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods based on a single or multiple cameras or other types of sensors, such as stereo imaging or time-of-flight measurement.

[0077] In some embodiments, the hand tracking device 140 captures and processes a time series of depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or the processor in the controller 110 processes the 3D mapping data to extract image patch descriptors of the hand in these depth maps. The software can match these descriptors with image patch descriptors stored in the database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and finger tips.

[0078] The software can also analyze the trajectory of the hand and / or finger over multiple frames in the sequence to recognize gestures. The pose estimation function described herein can be alternated with the motion tracking function so that the image block-based pose estimation is performed only once every two (or more) frames, and the tracking is used to find the changes in pose that occur on the remaining frames. The pose, motion, and gesture information is provided to the application running on the controller 110 via the above-mentioned API. The program can, for example, move and modify the image presented on the display generation component 120 in response to the pose and / or gesture information, or perform other functions.

[0079] In some embodiments, gestures include air gestures. An air gesture is a gesture detected without the user touching an input element that is part of a device (e.g., computer system 101, one or more input devices 125, and / or hand tracking device 140) (or independent of an input element that is part of a device) and based on detected movement of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs) through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of a user's finger relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body)).

[0080] In some embodiments, according to some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures for interacting with an XR environment (e.g., a virtual or mixed reality environment) performed by movement of a user's fingers relative to other fingers or parts of the user's hand. In some embodiments, an air gesture is a gesture detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and based on detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture including a predetermined speed or rotation amount of a part of the user's body)).

[0081] In some embodiments where the input gesture is an in-air gesture (e.g., in the absence of physical contact with an input device that provides information to the computer system about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or trackpad to move a cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Thus, in specific implementations involving in-air gestures, for example, the input gesture is combined with (e.g., simultaneously with) movement of the user's fingers and / or hand to detect attention (e.g., gaze) toward a user interface element to perform a pinch and / or tap input, as described below.

[0082] In some embodiments, an input gesture pointing to a user interface object is performed directly or indirectly with reference to the user interface object. For example, the user input is performed directly on the user interface object according to performing an input gesture with the user's hand at a location corresponding to the location of the user interface object in the three-dimensional environment (e.g., as determined based on the user's current viewpoint). In some embodiments, when the user's attention (e.g., gaze) to the user interface object is detected, the input gesture is performed indirectly on the user interface object according to the location of the user's hand not being at the location corresponding to the location of the user interface object in the three-dimensional environment while the user performs the input gesture. For example, for a direct input gesture, the user can direct the user's input to the user interface object by initiating a gesture at or near a location corresponding to the display location of the user interface object (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge of the option or the center portion of the option). For an indirect input gesture, the user can direct the user's input to the user interface object by focusing on the user interface object (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location that does not correspond to the display location of the user interface object).

[0083] In some embodiments, according to some embodiments, the input gestures (e.g., air gestures) used in various examples and embodiments described herein include pinch input and tap input for interacting with a virtual or mixed reality environment. For example, the pinch input and tap input described below are performed as air gestures.

[0084] In some embodiments, the pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes movement of two or more fingers of a hand to contact each other, that is, optionally followed by an immediate (e.g., within 0 to 1 second) interruption of contact with each other. A long pinch gesture as an air gesture includes movement of two or more fingers of a hand in contact with each other for at least a threshold amount of time (e.g., at least 1 second) before an interruption of contact with each other is detected. For example, a long pinch gesture includes the user maintaining a pinch gesture (e.g., in which two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture as an air gesture includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected consecutively immediately (e.g., within a predefined time period) with each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts contact between two or more fingers), and performs a second pinch input within a predefined time period after releasing the first pinch input (e.g., within 1 second or within 2 seconds).

[0085] In some embodiments, the pinch and drag gesture as an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in conjunction with (e.g., following) a drag input that changes the positioning of the user's hand from a first positioning (e.g., a starting positioning for dragging) to a second positioning (e.g., an ending positioning for dragging). In some embodiments, the user maintains the pinch gesture while performing the drag input, and releases the pinch gesture (e.g., opens their two or more fingers) to end the drag gesture (e.g., at the second positioning). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers to contact each other and moves the same hand to the second positioning in the air using the drag gesture). In some embodiments, a pinch input is performed by a first hand of a user, and a drag input is performed by a second hand of the user (e.g., the second hand of the user moves in the air from a first position to a second position while the user continues the pinch input with the first hand of the user. In some embodiments, the input gesture as an air gesture includes an input performed using both hands of the user (e.g., a pinch and / or tap input). For example, the input gesture includes two (e.g., or more) pinch inputs performed in conjunction with each other (e.g., simultaneously or within a predefined time period). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) is performed using the first hand of the user, and in conjunction with the pinch input performed using the first hand, a second pinch input is performed using another hand (e.g., a second hand of the user's two hands). In some embodiments, movement between the two hands of the user (e.g., increasing and / or decreasing the distance or relative orientation between the two hands of the user)

[0086] In some embodiments, a tap input performed as an air gesture (e.g., pointing to a user interface element) includes movement of a user's finger toward the user interface element, movement of a user's hand toward the user interface element (optionally, extension of the user's finger toward the user interface element), downward movement of a user's finger (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movement of a user's hand. In some embodiments, a tap input performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture movement of the finger or hand, which is a movement of the finger or hand away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by an end of the movement. In some embodiments, the end of the movement is detected based on a change in the movement characteristics of the finger or hand performing the tap gesture (e.g., the end of the movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of movement of the finger or hand, and / or a reversal of the acceleration direction of the movement of the finger or hand).

[0087] In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment (optionally, no other conditions are required). In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment using one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., a dwell duration) and / or requiring the gaze to be directed to the portion of the three-dimensional environment when the user's viewpoint is within a distance threshold from the portion of the three-dimensional environment, so that the device determines that the user's attention is directed to the portion of the three-dimensional environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0088] In some embodiments, the detection of a ready state configuration of a user or a portion of a user is detected by a computer system. The detection of the ready state configuration of a hand is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more air gesture inputs (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein) performed by the hand. For example, the ready state of the hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart to prepare for a pinch or grab gesture, or a pre-tap with one or more fingers extended and the palm facing away from the user), based on whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or based on whether the hand has moved in a particular manner (e.g., toward an area in front of the user above the user's waist and below the user's head or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of a user interface responds to attention (e.g., gaze) input.

[0089] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or may alternatively be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is also stored in a memory associated with the controller 110. Alternatively or in addition, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although in Figure 4Controller 110 is shown in FIG. 1 , but some or all of the processing functions of the controller may be performed by a suitable microprocessor and software or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand tracking device) or other device associated with the image sensor 404, for example, as a separate unit from the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing functions of the image sensor 404 may also be integrated into a computer or other computerized device to be controlled by the sensor output.

[0090] Figure 4 Also included is a schematic diagram of a depth map 410 captured by the image sensor 404 according to some embodiments. As described above, the depth map includes a matrix of pixels with corresponding depth values. Pixels 412 corresponding to the hand 406 have been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z distance from the image sensor 404), where gray shades become darker as depth increases. The controller 110 processes these depth values ​​in order to identify and segment components of the image that have human hand features (i.e., a group of adjacent pixels). These features may include, for example, overall size, shape, and motion from frame to frame in the depth map sequence.

[0091] Figure 4 Also schematically shown is a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406 according to some embodiments. Figure 4 In the hand skeleton 414, a hand background 416 that has been segmented from the original depth map is superimposed. In some embodiments, key feature points of the hand and optionally on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, finger tips, palm center, end of the hand connected to the wrist, etc.) are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points over multiple image frames to determine the gesture performed by the hand or the current state of the hand according to some embodiments.

[0092] Figure 5 The eye tracking device 130 ( Figure 1 ). In some embodiments, the eye tracking device 130 is composed of an eye tracking unit 243 ( Figure 2) controls to track the position and movement of the user's gaze relative to the scene 105 or relative to the XR content displayed via the display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as headphones, helmets, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating XR content for viewing by the user and components for tracking the user's gaze relative to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or an XR room, the eye tracking device 130 is optionally a device separate from the handheld device or the XR room. In some embodiments, the eye tracking device 130 is a head-mounted device or a part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or a display generation component that is not head-mounted. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally used in conjunction with a head mounted display generation component. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally part of a non-head mounted display generation component.

[0093] In some embodiments, the display generation component 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display a frame including a left image and a right image in front of the user's eyes, thereby providing a 3D virtual view to the user. For example, the head-mounted display generation component may include a left optical lens and a right optical lens (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture a video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display, and display a virtual object on the transparent or translucent display, and the user can directly view the physical environment through the transparent or translucent display. In some embodiments, the display generation component projects the virtual object into the physical environment. The virtual object may, for example, be projected on a physical surface or projected as a hologram so that an individual using the system observes the virtual object superimposed on the physical environment. In this case, separate display panels and image frames for the left and right eyes may not be required.

[0094] like Figure 5As shown in , in some embodiments, the eye tracking device 130 (e.g., gaze tracking device) includes at least one eye tracking camera (e.g., an infrared (IR) or near infrared (NIR) camera), and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera can be pointed at the user's eyes to receive the IR or NIR light reflected directly from the eyes by the light source, or alternatively can be pointed at "hot" mirrors located between the user's eyes and the display panel, which reflect the IR or NIR light from the eyes to the eye tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and transmits the gaze tracking information to the controller 110. In some embodiments, the user's two eyes are tracked separately by corresponding eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by the corresponding eye tracking camera and illumination source.

[0095] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for a specific operating environment 100, such as the 3D geometric relationships and parameters of the LED, camera, thermal mirror (if present), eye lens, and display screen. The device-specific calibration process can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration process can be an automatic calibration process or a manual calibration process. According to some embodiments, the user-specific calibration process may include an estimate of the eye parameters of a specific user, such as pupil position, foveal position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific parameters and user-specific parameters are determined for the eye tracking device 130, a flash-assisted method can be used to process the images captured by the eye tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0096] like Figure 5As shown in , the eye tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system, which includes at least one eye tracking camera 540 (e.g., an infrared (IR) or near infrared (NIR) camera) positioned on the side of the user's face on which eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye tracking camera 540 can be directed toward a mirror 550 (which reflects IR or NIR light from the eye 592 while allowing visible light to pass) located between the user's eye 592 and a display 510 (e.g., a left display panel or a right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) Figure 5 ), or alternatively may be directed toward the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion of Figure 5 ).

[0097] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses the gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as for processing the frames 562 for display. The controller 110 optionally estimates the user's gaze point on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0098] Several possible use cases for the user's current gaze direction are described below and are not intended to be limiting. As an example use case, the controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, the controller 110 may generate virtual content at a higher resolution in a foveal area determined according to the user's current gaze direction than in a peripheral area. As another example, the controller may position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller may display specific virtual content in a view based at least in part on the user's current gaze direction. As another example use case in an AR application, the controller 110 may direct an external camera for capturing the physical environment of the XR experience to focus in the determined direction. The autofocus mechanism of the external camera may then focus on an object or surface in the environment that the user is currently looking at on the display 510. As another example use case, the eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of the eye lens 520 so that the virtual object that the user is currently looking at has an appropriate degree of convergence to match the convergence of the user's eye 592. The controller 110 can use the gaze tracking information to guide the eye lens 520 to adjust the focus so that nearby objects that the user is looking at appear at the correct distance.

[0099] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lenses 520), an eye tracking camera (e.g., eye tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)). The light source emits light (e.g., IR or NIR light) toward the user's eye 592. In some embodiments, the light sources may be arranged in a ring or circle around each of the lenses, such as Figure 5 In some embodiments, for example, eight light sources 530 (eg, LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0100] In some embodiments, the display 510 emits light in the visible range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. Note that the positions and angles of the eye tracking cameras 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0101] like Figure 5 Embodiments of the gaze tracking system shown in may, for example, be used in computer generated reality, virtual reality, and / or mixed reality applications to provide a computer generated reality, virtual reality, augmented reality, and / or augmented virtual experience to a user.

[0102] Figure 6 FIG. 2 shows a flash-assisted gaze tracking pipeline according to some embodiments. In some embodiments, the gaze tracking pipeline is implemented by a flash-assisted gaze tracking system (e.g., Figure 1 and Figure 5 The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and the flash in the current frame. When not in the tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and the flash in the current frame, and if successful, initializes the tracking state to "yes" and continues with the next frame in the tracking state.

[0103] like Figure 6 As shown in , the gaze tracking camera can capture left and right images of the user's left and right eyes. The captured images are then input to the gaze tracking pipeline for processing starting at 610. As indicated by the arrow returning to element 600, the gaze tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each group of captured images can be input to the pipeline for processing. However, in some embodiments or under some conditions, not all captured frames are processed by the pipeline.

[0104] At 610, for the currently captured image, if the tracking status is yes, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as indicated at 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0105] At 640, if advancing from element 610, the current frame is analyzed to track pupils and glints based in part on previous information from previous frames. At 640, if advancing from element 630, the tracking state is initialized based on the pupils and glints detected in the current frame. The processing results at element 640 are checked to verify that the results of the tracking or detection can be credible. For example, the results can be checked to determine whether the pupil and a sufficient number of glints for performing gaze estimation are successfully tracked or detected in the current frame. At 650, if the results are not likely to be credible, at element 660, the tracking state is set to no, and the method returns to element 610 to process the next image of the user's eye. At 650, if the results are credible, the method advances to element 670. At 670, the tracking state is set to yes (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0106] Figure 6 It is intended to be used as an example of an eye tracking technology that can be used for a particular implementation. As recognized by one of ordinary skill in the art, according to various embodiments, other eye tracking technologies currently existing or developed in the future can be used in place of or in combination with the flash-assisted eye tracking technology described herein in a computer system 101 for providing an XR experience to a user.

[0107] In some embodiments, the captured portion of the real-world environment 602 is used to provide an XR experience to the user, such as a mixed reality environment in which one or more virtual objects are superimposed on top of a representation of the real-world environment 602 .

[0108] Thus, the description herein describes some embodiments of a three-dimensional environment (e.g., an XR environment) including representations of real-world objects and representations of virtual objects. For example, the three-dimensional environment optionally includes a representation of a table present in a physical environment, which is captured and displayed in the three-dimensional environment (e.g., actively displayed via a camera and display of a computer system or passively displayed via a transparent or translucent display of a computer system). As previously described, the three-dimensional environment is optionally a mixed reality system, wherein the three-dimensional environment is based on a physical environment captured by one or more sensors of a computer system and displayed via a display generation component. As a mixed reality system, the computer system is optionally capable of selectively displaying parts and / or objects of the physical environment so that the corresponding parts and / or objects of the physical environment appear as if they exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system is optionally capable of displaying virtual objects in a three-dimensional environment by placing virtual objects at corresponding positions in the three-dimensional environment that have corresponding positions in the real world to appear as if the virtual objects exist in the real world (e.g., a physical environment). For example, the computer system optionally displays a vase so that the vase appears as if a real vase is placed on top of a table in the physical environment. In some embodiments, a corresponding position in the three-dimensional environment has a corresponding position in the physical environment. Thus, when a computer system is described as displaying a virtual object at a corresponding position relative to a physical object (e.g., such as a position at or near a user's hand or at or near a physical table), the computer system displays the virtual object at a specific position in the three-dimensional environment so that it appears as if the virtual object is at or near the physical object in the physical environment (e.g., the virtual object is displayed at a position in the three-dimensional environment that corresponds to the position in the physical environment where the virtual object would be displayed if it were a real object at that specific position).

[0109] In some embodiments, real-world objects present in the physical environment that are displayed in the three-dimensional environment (e.g., and / or visible via a display generation component) can interact with virtual objects that only exist in the three-dimensional environment. For example, the three-dimensional environment may include a table and a vase placed on top of the table, where the table is a view (or representation) of a physical table in the physical environment and the vase is a virtual object.

[0110] Similarly, the user is optionally able to use one or both hands to interact with virtual objects in the three-dimensional environment as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the computer system optionally capture one or more hands of the user and display representations of the user's hands in the three-dimensional environment (e.g., in a manner similar to displaying real-world objects in the three-dimensional environment described above), or in some embodiments, due to the transparency / translucency of a portion of the user interface being displayed by the display generation component, or due to the projection of the user interface onto a transparent / translucent surface or the projection of the user interface onto the user's eyes or into the field of view of the user's eyes, the user's hands can be seen via the display generation component, via the ability to see the physical environment through the user interface. Therefore, in some embodiments, the user's hands are displayed at corresponding locations in the three-dimensional environment and are treated as if they are objects in the three-dimensional environment, and these objects can interact with virtual objects in the three-dimensional environment as if these virtual objects are physical objects in the physical environment. In some embodiments, the computer system can update the display of the representation of the user's hands in the three-dimensional environment in conjunction with the movement of the user's hands in the physical environment.

[0111] In some embodiments described below, the computer system is optionally capable of determining the "effective" distance between a physical object in the physical world and a virtual object in a three-dimensional environment, for example, to determine whether the physical object is directly interacting with the virtual object (e.g., whether the hand is touching, grabbing, holding, etc., a virtual object or within a threshold distance of the virtual object). For example, the hand that directly interacts with the virtual object optionally includes one or more of the following: a finger of the hand pressing a virtual button, a user's hand grabbing a virtual vase, a user's hand closing together and pinching / holding the user interface of the application, and two fingers that perform any other type of interaction described herein. For example, when determining whether the user is interacting with a virtual object and / or how the user is interacting with the virtual object, the computer system optionally determines the distance between the user's hand and the virtual object. In some embodiments, the computer system determines the distance between the user's hand and the virtual object by determining the distance between the position of the hand in the three-dimensional environment and the position of the virtual object of interest in the three-dimensional environment. For example, the one or more hands of the user are located at a specific location in the physical world, and the computer system optionally captures the one or more hands and displays the one or more hands at a specific corresponding location in the three-dimensional environment (e.g., if the hand is a virtual hand instead of a physical hand, the location where the hand will be displayed in the three-dimensional environment). The location of the hand in the three-dimensional environment is optionally compared with the location of the virtual object of interest in the three-dimensional environment to determine the distance between the one or more hands of the user and the virtual object. In some embodiments, the computer system optionally determines the distance between the physical object and the virtual object by comparing the location in the physical world (e.g., instead of comparing the location in the three-dimensional environment). For example, when determining the distance between the one or more hands of the user and the virtual object, the computer system optionally determines the corresponding position of the virtual object in the physical world (e.g., if the virtual object is a physical object instead of a virtual object, the location where the virtual object will be located in the physical world), and then determines the distance between the corresponding physical location and the one or more hands of the user. In some embodiments, the same technology is optionally used to determine the distance between any physical object and any virtual object. Therefore, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, a computer system optionally executes any of the techniques described above to map the position of the physical object to a three-dimensional environment and / or to map the position of the virtual object to the physical environment.

[0112] In some embodiments, the same or similar techniques are used to determine where and what the user's gaze is directed to, and / or where and what the physical stylus held by the user is pointed to. For example, if the user's gaze is directed to a particular location in the physical environment, the computer system optionally determines the corresponding location in the three-dimensional environment (e.g., the virtual location of the gaze), and if a virtual object is located at the corresponding virtual location, the computer system optionally determines that the user's gaze is directed to the virtual object. Similarly, the computer system is optionally able to determine the direction in which the stylus is pointing in the physical environment based on the orientation of the physical stylus. In some embodiments, based on this determination, the computer system determines a corresponding virtual location in the three-dimensional environment that corresponds to the location in the physical environment that the stylus is pointing to, and optionally determines that the stylus is pointing to the corresponding virtual location in the three-dimensional environment.

[0113] Similarly, the embodiments described herein may refer to the position of a user (e.g., a user of a computer system) in a three-dimensional environment and / or the position of a computer system in a three-dimensional environment. In some embodiments, the user of the computer system is holding, wearing, or otherwise located at or near the computer system. Therefore, in some embodiments, the position of the computer system is used as a proxy for the position of the user. In some embodiments, the position of the computer system and / or the user in the physical environment corresponds to the corresponding position in the three-dimensional environment. For example, the position of the computer system will be a position in the physical environment (and its corresponding position in the three-dimensional environment), and if the user stands at this position, facing the corresponding part of the physical environment visible via the display generation component, the user will see from this position in the physical environment in the same positioning, orientation and / or size (e.g., in an absolute sense and / or relative to each other) of the objects displayed in the three-dimensional environment by the display generation component of the computer system or visible in the three-dimensional environment via the display generation component. Similarly, if the virtual objects displayed in the three-dimensional environment are physical objects in the physical environment (e.g., physical objects placed at the same locations in the physical environment as the virtual objects are located in the three-dimensional environment, and physical objects that have the same size and orientation in the physical environment as they do in the three-dimensional environment), then the position of the computer system and / or user is the position from which the user would see the positions of the virtual objects in the physical environment at the same positions, orientations, and / or sizes (e.g., in an absolute sense and / or relative to each other and real-world objects) as the virtual objects displayed in the three-dimensional environment by the display generation components of the computer system.

[0114] In the present disclosure, various input methods are described with respect to interaction with a computer system. When an input device or input method is used to provide an example, and another input device or input method is used to provide another example, it should be understood that each example is compatible with the input device or input method described with respect to another example and optionally utilizes the input device or input method. Similarly, various output methods are described with respect to interaction with a computer system. When an output device or output method is used to provide an example, and another output device or output method is used to provide another example, it should be understood that each example is compatible with the output device or output method described with respect to another example and optionally utilizes the output device or output method. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment by a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with the method described with respect to another example and optionally utilizes these methods. Therefore, the present disclosure discloses an embodiment as a combination of features of multiple examples, without exhaustively listing all features of the embodiment in the description of each example embodiment.

[0115] User interface and associated processes

[0116] Attention is now turned to embodiments of a user interface ("UI") and associated processes that may be implemented on a computer system (such as a portable multifunction device or a head mounted device) having display generating components, one or more input devices, and (optionally) one or more cameras.

[0117] FIG. 7A to FIG. 7G Examples of computer systems that perform various object selection-related operations are shown according to some embodiments.

[0118] Fig. 7A The display generation components (eg, Figure 1 101 ) is visible from a user's viewpoint 716 shown in a top view 714 (e.g., facing the back wall of the physical environment in which the computer system 101 is located and near the back left corner of the physical environment). Figures 1 to 6 As described above, the computer system 101 optionally includes a display generation component (e.g., a touch screen) and a plurality of image sensors (e.g., Figure 3The image sensor 314 may include one or more of: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the computer system 101 can use to capture one or more images of a user or a portion of a user (e.g., one or more hands of a user) as the user interacts with the computer system 101. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display that includes a display generation component that displays a user interface or a three-dimensional environment to the user, as well as sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors that face outward from the user) and / or sensors that detect the user's gaze (e.g., internal sensors that face inward toward the user's face).

[0119] like Fig. 7A , computer system 101 captures one or more images of a physical environment 702 (e.g., operating environment 100) surrounding computer system 101 (including one or more objects in the physical environment surrounding computer system 101). In some embodiments, computer system 101 displays a representation of the physical environment in a three-dimensional environment 704 and / or the physical environment 702 is visible in the three-dimensional environment 704 via display generation component 120. For example, the three-dimensional environment 704 visible via display generation component 120 includes a representation of the physical floor and back and side walls of the room in which computer system 101 is located. The three-dimensional environment 704 also includes a table 706 (shown via display generation component 120 and in top view 714).

[0120] exist Fig. 7A, the three-dimensional environment 704 also includes virtual objects 708-1 to 708-6, virtual object 710, and virtual object 712. Virtual objects 708-1 to 708-6 are optionally one or more of user interfaces of applications (e.g., messaging user interfaces, content browsing user interfaces, etc.), three-dimensional objects (e.g., virtual clocks, virtual balls, virtual cars, etc.), or any other elements displayed by computer system 101 that are not included in the physical environment of computer system 101. Virtual object 710 is optionally a container or canvas object that contains virtual objects 708-1 to 708-6, as described in more detail with reference to methods 800, 1000, and / or 1200. Virtual object 712 is optionally a tool palette object for performing operations related to virtual objects 708-1 to 708-6 and / or 710. For example, element 712 includes a selectable option 714-1 that can be selected to undo the last operation performed associated with virtual objects 708-1 to 708-6 and / or 710; a selectable option 714-2 that can be selected to redo the last operation canceled or undone associated with virtual objects 708-1 to 708-6 and / or 710; a selectable option 714-3 that can be selected to edit the currently selected object of virtual objects 708-1 to 708-6; a selectable option 714-4 that can be selected to add a new virtual object to object 710; and a selectable option 714-5 that can be selected to cause the computer system 101 and / or object 710 to transition to a selection mode, which will be described in more detail later.

[0121] In some embodiments, one or more of objects 708-1 through 708-6 can be selected and / or can be highlighted to designate those objects as targets for further operations (e.g., a move operation, a copy operation, or a cut operation). In some embodiments, multiple virtual objects are selected in response to a "marquee selection" input performed based on an air pinch gesture, as will now be described. For example, in Fig. 7A , when the user's gaze 720a is directed to a portion of the object 710 that does not include the object, the computer system 101 detects that the hand 703a performs an air pinch gesture in which the thumb and index finger of the hand 703a are brought together and touch a specific location in the physical environment 702. The location in the object 710 to which the gaze 720a is directed when the air pinch gesture is detected optionally defines a corner or boundary point of the selection area (e.g., with the corners or boundary points of the selection area). Fig. 7A When the thumb and index finger of the hand 703a are in contact (for example, the hand 703a is in a pinched hand shape), the computer system 101 detects that the hand 703a moves in the upward and rightward directions. Fig. 7AThe amount and / or direction of movement of hand 703a away from the position of gaze 720a when the air pinch gesture is initiated optionally defines a second corner or boundary point of the selection area (e.g., with Fig. 7A The distance and / or direction of the dotted rectangle in the object 710 corresponds to the upper right corner of the dotted rectangle in the object 710.

[0122] In some embodiments, any object that falls within the defined selection area itself becomes selected. Fig. 7A , objects 708-1 and 708-2 are selected, as indicated by the dashed borders around them. In some embodiments, objects 708-1 and 708-2 are selected as long as they fall within the selection area and before the computer system 101 detects that the hand 703a releases the hand pinch shape (e.g., the thumb and index finger of the hand 703a are separated). In some embodiments, objects 708-1 and 708-2 are not selected until the computer system 101 detects that the hand 703a releases the hand pinch shape, and become selected at that time. The selected objects are shown in the top view 714 as having a bold appearance to distinguish them from the unselected objects (which are shown as not having a bold appearance). In some embodiments, the selected two-dimensional objects are separated from object 710 (e.g., as indicated by the projected shadows shown for objects 708-1 and 708-2), and the unselected two-dimensional objects are not separated from object 710 or are less separated from object 710 (e.g., as indicated by the fact that no projected shadows are shown for objects 708-3 to 708-6). In some embodiments, the three-dimensional objects are separated from object 710 regardless of whether the three-dimensional objects are selected (e.g., as indicated by the projected shadows shown for object 708-6). In some embodiments, when the three-dimensional objects are selected, the three-dimensional objects are separated from object 710 more than when the three-dimensional objects are not selected; in some embodiments, the separation of the three-dimensional objects from object 710 does not change based on whether the three-dimensional objects are selected.

[0123] exist Figure 7B , both selected objects 708-1 and 708-2 are controlled by hand 703a. For example, in response to the movement of hand 703a, computer system 101 will move objects 708-1 and 708-2 (e.g., within object 710) in accordance with such movement (e.g., based on the magnitude and / or direction of the movement of hand 703a, as described in more detail with reference to method 800). Objects 708-1 and 708-2 are optionally moved in response to computer system 101 selecting objects 708-1 and 708-2 (e.g., in response to Fig. 7AThe object 708-1 or 708-2 is controlled by hand 703a when the user's gaze 720a is detected to be directed to at least one of objects 708-1 or 708-2 when the gaze 720a is directed to the object and an air pinch gesture (e.g., as previously described) is detected by hand 703a. For example, when hand 703a maintains the air pinch gesture in which the thumb and index finger are touching, objects 708-1 and 708-2 will be controlled by hand 703a.

[0124] While objects 708-1 and 708-2 are controlled by hand 703a, the user's second hand (hand 703b) is able to interact with other objects in three-dimensional environment 704. The response of computer system 101 to such interactions depends on the characteristics of the input provided by hand 703b. For example, if hand 703b performs a quick pinch air gesture (in which the thumb and index finger of hand 703b are brought together and touch within a time threshold of touch and then separated to release the pinch hand shape (e.g., as described in more detail with reference to method 800)) while gaze 720b is directed toward object 708-4, such as Figure 7B , then object 708-4 is optionally added to the set of objects controlled by hand 703a (e.g., objects 708-1 and 708-2). In this scenario, subsequent movement of hand 703a will cause computer system 101 to move not only objects 708-1 and 708-2 accordingly, but also object 708-4 accordingly. For example, in Figure 7C , hand 703a moves to the left, and in response, the computer system moves objects 708-1, 708-2, and 708-4 to the left according to the movement of hand 703a.

[0125] In contrast, Figure 7B , if computer system 101 instead detects hand 703b performing a long pinch air gesture (where the thumb and index finger of hand 703b are brought together and touching within a time threshold of touch and do not separate to release the pinch hand shape) while gaze 720b is directed toward object 708-4, followed by movement of hand 703b exceeding a movement threshold (e.g., as described in more detail with reference to method 800), object 708-4 will be controlled by hand 703b, while objects 708-1 and 708-2 will continue to be controlled by hand 703a. For example, from FIG. 7B to FIG. 7D, hand 703b has moved downward and computer system 101 has correspondingly moved object 708-4 downward, and hand 703a has moved upward and computer system 101 has correspondingly moved objects 708-1 and 708-2 upward. In some embodiments, in response to detecting that hands 703a and / or 703b release the hand pinch shape, the objects controlled by those hands remain selected. In some embodiments, in response to detecting that hands 703a and / or 703b release the hand pinch shape, the objects controlled by those hands become unselected.

[0126] Computer system 101 and / or object 710 also operates in a selection mode having one or more different responses to the above-described input types. Fig. 7A In response to detecting a selection of selectable option 714-5 (e.g., via a quick pinch air gesture performed by hand 703a while gaze 720b is directed toward selectable option 714-5), selection mode is activated, as shown in FIG. Fig. 7E In selection mode, element 712 is optionally updated to include selectable option 714-6, which can be selected to exit selection mode. When in selection mode, a long pinch and drag air gesture (e.g., such as reference Fig. 7A ) optionally causes scrolling through the object 710 instead of defining the selection area. Fig. 7E , computer system 101 detects gaze 720 directed to an empty portion of object 710 and a long pinch and drag air gesture by hand 703a while moving hand 703a to the left. It should be understood that hand 703a is optionally the same hand as hand 703b, or a different hand. In response, as Figure 7F , computer system 101 has scrolled right through object 710 to hide objects 708-1 through 708-6 and reveal objects 708-7 and 708-8. In some embodiments, even if gaze 720 is directed toward Fig. 7E , and even if the object (or objects) is selected, the computer system 101 will scroll through the objects 710 in response to a long pinch-and-drag air gesture by hand 703a, rather than moving the objects in response to dragging such as when in selection mode.

[0127] When in selection mode, a quick pinch air gesture directed at an object causes the object to become selected. Figure 7F In FIG. 7 , computer system 101 detects hand 703 a performing a quick pinch air gesture while gaze 720 is directed toward object 708 - 7 . In response, Figure 7F, object 708-7 has become selected. In addition, in some embodiments, computer system 101 updates selectable option 714-6 to indicate the number of currently selected objects within object 710. For example, in Figure 7F , the computer system 101 has updated the selectable option 714-6 to indicate that an object has been selected.

[0128] exist Figure 7G In the example of FIG. 1 , when object 708-7 is selected, computer system 101 detects hand 703a performing a quick pinch air gesture while gaze 720 is directed toward object 708-7. In response to input from hand 703a being directed toward object 708-7 when object 708-7 is selected, computer system 101 displays a context menu 722 associated with the currently selected object (e.g., object 708-7), such as Figure 7G . The context menu 722 optionally includes one or more selectable options for performing an operation corresponding to the selected option. For example, Figure 7G The context menu 722 in includes an option 722-1 that can be selected to copy the object 708-7 (e.g., for later pasting, as described in more detail with reference to method 1000); an option 722-2 that can be selected to cut the object 708-7 (e.g., for later pasting, as described in more detail with reference to method 1000); an option 722-3 that can be selected to copy the object 708-7 (e.g., as described in more detail with reference to method 1000); an option 722-4 that can be selected to flip (e.g., horizontally and / or vertically) the object 708-7; an option 722-5 that can be selected to open the object 708-7 (e.g., in a specified application that displays content associated with the object 708-7); and an option 722-6 that can be selected to delete the object 708-7.

[0129] FIG. 8A to FIG. 8K 800 is a flowchart illustrating an exemplary method 800 for performing various object selection related operations according to some embodiments. In some embodiments, the method 800 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet device, a smart phone, a wearable computer, or a head mounted device, the computer system including a display generating component (e.g., Figure 1 , Figure 3 and Figure 4In some embodiments, method 800 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1 Some operations in method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0130] In some embodiments, method 800 is performed at a computer system (e.g., 101) (e.g., a mobile device (e.g., a tablet device, a smart phone, a media player, or a wearable device) or a computer) that communicates with a display generation component (e.g., 120) and one or more input devices. In some embodiments, the display generation component is a display (optionally a touch screen display) integrated with the computer system, an external display such as a monitor, a projector, a television, or a hardware component (optionally integrated or external) for projecting a user interface or making a user interface visible to one or more users. In some embodiments, the one or more input devices include a computer system or component capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information associated with the user input to the computer system. Examples of input devices include a touch screen, a mouse (e.g., external), a trackpad (optionally integrated or external), a touchpad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the computer system), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, an eye tracking device, and / or a motion sensor (e.g., a hand tracking device, a hand motion sensor), etc. In some embodiments, the computer system communicates with a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touch screen, trackpad)). In some embodiments, the hand tracking device is a wearable device, such as a smart glove. In some embodiments, the hand tracking device is a handheld input device, such as a remote control or a stylus.

[0131] In some embodiments, when a first user interface object (e.g., 708-1) and a second user interface object (e.g., 708-4) are displayed via a display generation component (802a) (e.g., the first user interface object and the second user interface object are displayed on a display or in a three-dimensional environment. In some embodiments, the first user interface object and the second user interface object are user interfaces of an application (such as a word processing application, a photo management application, a spreadsheet application, a presentation application, and / or an email application) that can be accessed by a computer system. In some embodiments, the first user interface object and the second user interface object are user interfaces of the same application or correspond to user interfaces of different applications. In some embodiments, the first user interface object and / or the second user interface object are displayed at or in the first virtual object described in method 1200. In some embodiments, the three-dimensional environment is a computer generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment. In some embodiments, the three-dimensional environment includes virtual objects, such as application windows, operating system elements, representations of users other than users of the computer system, and physical environments of the computer system. In some embodiments, the representation of the physical object is a view of the physical object in the physical environment of the computer system that is visible through a transparent portion of the display generating component (e.g., a true or real passthrough). In some embodiments, the computer system captures one or more images of one or more physical objects in the physical environment of the computer system (e.g., via an image sensor of one or more input devices), and in response, displays corresponding virtual representations (e.g., a real representation, a simplified representation, or a cartoon representation) of the one or more physical objects in the three-dimensional environment. In some embodiments, the computer system displays the three-dimensional environment from the user's viewpoint at a position in the three-dimensional environment corresponding to the physical position of the computer system and / or the physical position of the user in the physical environment of the computer system. In some embodiments, displaying the three-dimensional environment from the user's viewpoint includes displaying the three-dimensional environment from a perspective associated with the position of the user's viewpoint in the three-dimensional environment. ), when a first group of one or more user interface objects including a first user interface object but not including a second user interface object is in a selected state and is controlled by a first part (e.g., a first hand) of a user of the computer system, such as by Figure 7BObjects 708-1 and 708-2 controlled by hand 703a in the three-dimensional environment (e.g., the first group of one or more user interface objects are selected / managed together so that an action (such as one of the actions / operations described in method 1000) performed by the first part of the user (e.g., the first hand) causes the action to be directed to a user interface object currently included in the first group of one or more user interface objects (rather than to a user interface object not currently included in the first group of one or more user interface objects). For example, when the first group of one or more user interface objects are being controlled by the first part of the user, the first group of one or more user interface objects are optionally moved simultaneously to one or more new locations in the three-dimensional environment based on the movement of the first part of the user. Thus, if the computer system detects that the first part of the user moves 1 cm, 2 cm, 5 cm, 10 cm, 20 cm, or 50 cm to the right, the computer system optionally moves the user interface objects included in the first group of one or more user interface objects to the right by 1 cm, 2 cm, 5 cm, 10 cm, 20 cm, or 50 cm. meters, 5 centimeters, 10 centimeters, 20 centimeters, or 50 centimeters (or amounts proportional to these distances). In some embodiments, the first group of one or more user interface objects are selected in a manner similar to the selection of the second user interface object and / or other user interface objects described below. In some embodiments, the first group of one or more user interface objects cease to be controlled by the first hand in response to the computer system detecting that the fingers of the first hand are no longer in a pinch hand shape (for example, when the computer system detects the release of the pinch gesture). ), the computer system detects (802b) via one or more input devices a first input directed to a second user interface object, the first input comprising a second part of the user of the computer system that is different from the first part (for example, a second hand, optionally while the gaze of the user of the computer system is directed to the second user interface object. In some embodiments, the first input and / or other inputs described with reference to method 800 are or include air gesture inputs.) that are different from the first part of the user of the computer system, such as from Figure 7B The hand 703b in FIG. 7 is pointing to the input of the object 708-4.

[0132] In some embodiments, the computer system detects that the second portion of the user performs or is performing a "quick" pinch or "long" pinch gesture while the user's gaze is directed toward the second user interface object. In some embodiments, the computer system detects that the second portion of the user performs a "quick" pinch gesture in response to detecting that the second portion of the user performs a "pinch" hand shape / posture (e.g., two or more fingers of the user's hand (such as the thumb and index finger) move together and touch each other or are within a threshold distance of each other (e.g., 0.1 cm, 0.2 cm, 0.3 cm, 0.5 cm, 1 cm, 2 cm, 3 cm, or 5 cm) for less than a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, or 20 seconds), and then move away from each other). Conversely, in some embodiments, the computer system detects that the second portion of the user performs or is performing a "long" pinch gesture in response to detecting that the second portion of the user has been (or is currently) in a "pinch" hand shape / posture for more than the above threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, or 20 seconds). In some embodiments, the first input is detected when the first portion of the user is currently in a “pinch” hand shape and / or is performing a “long” pinch gesture (e.g., the first portion of the user has been in the “pinch” hand shape for more than the threshold amount of time described above, and the fingers used to make the “pinch” shape have not moved away from each other). In some embodiments, the first input is detected after the first portion of the user performs a corresponding gesture that results in selection of one or more user interface objects of the first group (e.g., when the first input is detected, the first portion of the user is no longer in the pinch hand shape), as described in more detail below. In some embodiments, the gesture directed toward the second user interface corresponds to a gesture other than a “quick” pinch or a “long” pinch gesture, such as a forward pointing gesture (e.g., a forward movement of the user's hand as one or more fingers of the user's hand extend toward the first user interface object) or a tap gesture using fingers of the user's hand (e.g., a forward movement by fingers of the user's hand so that the fingers touch the first user interface object or are within a threshold distance of the first user interface object).

[0133] In some embodiments, in response to detecting a first input (802c), based on determining that the first input satisfies one or more first criteria, the one or more first criteria including a first criterion that is satisfied when an initiation of a corresponding gesture (e.g., when the computer system detects that the second part of the user is in a "pinch" hand shape / position, as previously described. In some embodiments, the initiation of the corresponding gesture is detected when two or more fingers of the second hand of the user move to contact each other.) and an end of the corresponding gesture (e.g., when the computer system detects that the fingers (e.g., the thumb and index finger) of the second part of the user that are making the "pinch" hand shape / position move away from each other) are performed within a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, or 20 seconds) of each other, the computer system adds (802d) a second user interface object to a first group of one or more user interface objects that are in a selected state and controlled by the first part of the user of the computer system, including moving the first user interface object and the second user interface object (e.g., in a three-dimensional environment) in accordance with movements of the first part of the user, such as by Figure 7C The hand 703a in the figure controls objects 708-1, 708-2 and 708-3 (for example, not controlling the second user interface object based on the second part of the user, including giving up moving the second user interface object in the three-dimensional environment according to the movement of the second part of the user).

[0134] In some embodiments, moving the first user interface object and the second user interface object in the three-dimensional environment according to the movement of the first part of the user includes moving those objects based on the magnitude and / or direction of the movement of the first part of the user. For example, if the first part of the user moves up and to the right, those objects are moved up and to the right, or if the first part of the user moves to the left, those objects are moved to the left. The movement amount / magnitude of those objects is optionally also controlled by the movement magnitude of the first part of the user. For example, if the movement amount of the first part of the user is 12cm, those objects are moved by a first amount (e.g., proportional to the movement of the first part of the user (e.g., 12cm)), and if the movement amount of the first part of the user is 24cm, those objects are moved by a second amount greater than the first amount (e.g., proportional to the movement of the first part of the user (e.g., 24cm)). For example, in response to detecting that the second part of the user performs a "fast" pinch gesture, while the user's gaze is optionally directed to the second user interface object, the second user interface object is added to the first group of one or more user interface objects. Thus, if after adding the second user interface object to the first group of one or more user interface objects, a "cut" or "copy" operation is directed to the first group of one or more user interface objects, the "cut" or "copy" operation will include a "copy" or "cut" of the second user interface object, because the second user interface object is now part of the first group of one or more user interface objects. Similarly, if the computer system detects a request to move the first group of one or more user interface objects in a three-dimensional environment, moving the first group of one or more user interface objects will include moving the second user interface object together with other user interface objects in the first group of one or more user interface objects, because the second user interface object is now part of the first group of one or more user interface objects. In some embodiments, after adding the second user interface object to the first group of one or more user interface objects via a "quick" pinch gesture, the computer system optionally detects one or more additional "quick" pinch gestures directed to one or more other user interface objects. In response, the computer system also adds one or more other user interface objects selected via the "quick" pinch gesture to the first group of one or more user interface objects. In some embodiments, when the second user interface object changes from being in a non-selected state to being in a selected state, the distance of the second user interface object from the user's viewpoint changes. For example, when the second user interface object is in the selected state, the second user interface object is optionally closer to the user's viewpoint than when the second user interface object is in the unselected state. In some embodiments, the user interface object in the selected state has a different visual appearance than when the user interface object is not in the selected state (e.g., in the unselected state), as will be described in more detail later.

[0135] In some embodiments, based on determining that the first input satisfies one or more second criteria that are different from the one or more first criteria, the one or more second criteria including a first criterion that is satisfied when the end of the corresponding gesture is not performed within a threshold amount of time of the initiation of the corresponding gesture (e.g., when the computer system is detecting that the second part of the user is in a "pinch" hand shape / posture and when the second part of the user has been in the "pinch" hand shape / posture for more than the above threshold amount of time) and a second criterion (802e) that is satisfied when the corresponding gesture includes movement of the second part of the user that exceeds a movement threshold (e.g., 0.1 cm, 5 cm, 10 cm, 15 cm, 20 cm, 30 cm, or 50 cm) after the initiation of the corresponding gesture (e.g., movement of the second part of the user is detected while the second part of the user is maintaining a "pinch" hand shape / posture. In some embodiments, the one or more second criteria also include a third criterion that is satisfied when the corresponding gesture includes movement of the second part of the user that exceeds the movement threshold after a threshold amount of time), the computer system updates the second user interface object (802f) to be in a selected state, such as Fig.7D , as shown in object 708-4 in . For example, because the second user interface object is selected via a "long" pinch gesture, the second user interface object is selected independently of the first set of one or more user interface objects (e.g., actions / operations directed to the first set of one or more user interfaces will not cause those actions / operations to be applied to the second user interface). In some embodiments, when the second user interface object is selected separately from the first set of one or more user interface objects, the computer system visually indicates that the second user interface object and the first set of one or more user interface objects are parts of different selections. In some embodiments, when the computer system updates the second user interface object from being in a non-selected state to being in a selected state, the computer system also updates the visual appearance of the second user interface object as described above.

[0136] In some embodiments, the computer system controls (802g) a second user interface object based on a second portion of the user, including moving the second user interface object (e.g., in a three-dimensional environment) in accordance with movement of the second portion of the user, while maintaining a selected state of a first set of user interface objects and control of one or more of the first set of user interface objects based on a first portion of the user of the computer system, such as based on Fig.7D The hand 703b in FIG. 708 moves the object 708-4, while the objects 708-1 and 708-2 are kept by Fig.7D The hand 703a in the three-dimensional environment controls the second user interface object (for example, not controlling the second user interface object based on the first part of the user, including giving up moving the second user interface object in the three-dimensional environment according to the movement of the first part of the user).

[0137] In some embodiments, moving the second user interface object in the three-dimensional environment according to the movement of the second part of the user includes moving the object based on the magnitude and / or direction of the movement of the second part of the user. For example, if the second part of the user moves up and to the right, the object is moved up and to the right, or if the second part of the user moves to the left, the object is moved to the left. The movement amount / magnitude of the object is optionally also controlled by the movement magnitude of the second part of the user. For example, if the movement amount of the second part of the user is 12cm, the object is moved by a first amount (e.g., proportional to the movement of the second part of the user (e.g., 12cm)), and if the movement amount of the second part of the user is 24cm, the object is moved by a second amount greater than the first amount (e.g., proportional to the movement of the second part of the user (e.g., 24cm)). For example, updating the second user interface object to be in a selected state does not modify / change the selection state of the first group of one or more user interface objects (e.g., continue to select the first group of one or more user interface objects). In addition, because the computer system detects movement of the second portion of the user when the second portion of the user performs the "long" pinch gesture, the second user interface object is moved to a new position in the three-dimensional environment based on the movement of the second portion of the user (e.g., if the computer system detects that the second portion of the user moves 1 cm, 2 cm, 5 cm, 10 cm, 20 cm, or 50 cm to the right, the computer system optionally moves the second user interface object 1 cm, 2 cm, 5 cm, 10 cm, 20 cm, or 50 cm to the right (or an amount proportional to those distances)). In some embodiments, the position of the first group of one or more user interface objects in the three-dimensional environment does not change in response to detecting movement of the second portion of the user because the second portion of the user does not control the first group of one or more user interface objects. In some embodiments, when the computer system is detecting movement of the second portion of the user, the computer system also detects movement of the first portion of the user. For example, when the computer system detects that the second portion of the user moves 1 cm, 2 cm, 5 cm, 10 cm, 20 cm, or 50 cm to the right, the computer system also detects that the first portion of the user moves 5 cm, 10 cm, 15 cm, 20 cm, or 25 cm to the left. In response, the computer system optionally moves a user interface object controlled by a first portion of the user (a first group of one or more user interface objects) to the left by 5 cm, 10 cm, 15 cm, 20 cm, or 25 cm (or amounts proportional to those distances) in the three-dimensional environment, and also moves a user interface object controlled by a second portion of the user (a second user interface object) to the right by 1 cm, 2 cm, 5 cm, 10 cm, 20 cm, or 50 cm (or amounts proportional to those distances) in the three-dimensional environment.Adding the corresponding user interface object to the set of user interface objects being selected and controlled by the first portion of the user or selecting the corresponding user interface object independently of the first set of user interface objects based on a gesture performed by the second portion of the user reduces the number of inputs required to select the user interface object.

[0138] In some embodiments, when a first user interface object and a second user interface object are displayed via a display generation component, neither of the user interface objects displayed via the display generation component is in a selected state, such as in Fig. 7E In the embodiment of the present invention, the computer system detects (804a) a second input directed to the first user interface object via one or more input devices, the second input comprising a corresponding gesture performed by a second portion of the user, such as by Figure 7B The hand 703b in Fig. 7E A gesture performed in the middle and directed toward object 708-2 (e.g., a quick or long pinch air gesture as previously described while the user's attention is directed toward the first user interface object. The second input optionally has one or more characteristics of the first input performed by the first or second portion of the user).

[0139] In some embodiments, in response to detecting the second input, and based on determining that the second input satisfies one or more first criteria, the computer system displays (804b) content associated with the first user interface object via the display generation component, such as in response to pointing to the second input. Fig. 7E In one embodiment, the first user interface object 708-2 is displayed in response to the above-mentioned input of object 708-2 in the selection state (e.g., without placing the first user interface object in a selected state). For example, when none of the selectable user interface objects being displayed are in a selected state, a quick pinch directed to a particular user interface object "opens" the user interface object. For example, if the first user interface object is an object corresponding to a photo application, the content displayed in response to the second input is optionally a user interface of the photo application, from which photos can be browsed and / or viewed. As another example, if the first user interface object is an object corresponding to a messaging application, the content displayed in response to the second input is optionally a user interface of the messaging application, from which messaging conversations with other users can be viewed and / or interacted with. Accessing content associated with the first user interface object based on input that satisfies one or more first criteria reduces the number of inputs required to access such content.

[0140] In some embodiments, in response to detecting a first input, the first input satisfies one or more third criteria that are different from the one or more first criteria and the one or more second criteria, the one or more third criteria including a first criterion that is satisfied when the end of the corresponding gesture is not performed within a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, or 20 seconds) of the initiation of the corresponding gesture and a third criterion (806a) that is satisfied when the corresponding gesture includes movement of a second portion of the user that is less than a movement threshold (e.g., 0.1 cm, 5 cm, 10 cm, 15 cm, 20 cm, 30 cm, or 50 cm) after the initiation of the corresponding gesture, such as pointing to Figure 7B In response to detecting an end of the corresponding gesture (e.g., when the computer system detects that the fingers (e.g., thumb and index finger) of the second portion of the user making the "pinch" hand shape / pose move away from each other), the computer system adds (806b) the second user interface object to the first set of one or more user interface objects that are in a selected state and controlled by the first portion of the user of the computer system, including moving the first user interface object and the second user interface object (e.g., in a three-dimensional environment) in accordance with the movement of the first portion of the user, such as adding the second user interface object to be controlled by the first portion of the computer system. FIG. 7B to FIG. 7C 708-4 controlled by hand 703a in the selected state, but at the end of the long air pinch gesture described above). Adding the second user interface object to the first group of one or more user interface objects in the selected state optionally has one or more of the characteristics described previously with reference to adding the second user interface object to the first group of one or more user interface objects in the selected state based on one or more first criteria being satisfied. Therefore, in some embodiments, the second object can be added to the first group of objects in response to a quick pinch or a long pinch with very little movement. Adding the second user interface object to the first group of user interface objects in response to a first input that satisfies one or more third criteria reduces the number of inputs required to select the user interface object.

[0141] In some embodiments, while a second user interface object is being controlled based on a second portion of the user (e.g., based on determining that the first input satisfies one or more second criteria), the second user interface object is moved based on movement of the second portion of the user (e.g., in a three-dimensional environment), while maintaining a selected state of a first group of user interface objects and control of one or more user interface objects of the first group based on a first portion of the user of the computer system, such as in Fig.7D In the example, the computer system detects (808a) the end of the corresponding gesture, such as from Fig.7DIn some embodiments, in response to detecting the end of the corresponding gesture, the computer system places (808b) a second user interface object at a location corresponding to the current location of the second portion of the user (e.g., in a three-dimensional environment), such as placing object 708-4 at the position corresponding to the current location of the second portion of the user. Fig.7D In some embodiments, the second user interface object is placed in a selected state after being placed in a position corresponding to the current position of the second part of the user. In some embodiments, the second user interface object is converted into an unselected state after being placed in a position corresponding to the current position of the second part of the user. Placing the second user interface object in response to detecting the end of the corresponding gesture reduces the number of inputs required to place the user interface object at the target location.

[0142] In some embodiments, after detecting the first input and when the first group of one or more user interface objects is in a selected state and controlled by a first part (e.g., a first hand) of a user of the computer system (e.g., when the second user interface object is included in the first group of user interface objects, or when the second user interface object is not included in the first group of user interface objects), the computer system detects (810a) via one or more input devices a second input pointing to a third user interface object (e.g., a second user interface object or a user interface object different from the first user interface object or the second user interface object), the second input including a corresponding gesture performed by a second part of the user of the computer system, such as in Figure 7B After the input pointing to object 708-4, the computer system 101 detects the input pointing to object 708-6 (e.g., the second input optionally has one or more of the characteristics of the first input, while the user's attention is directed to a third user interface object, such as an air pinch gesture performed by the corresponding part of the user, as previously described). In some embodiments, in response to detecting the second input (810b), based on determining that the second input satisfies one or more first criteria, the computer system adds (810c) the third user interface object to the first group of one or more user interface objects that are in a selected state and controlled by the first part of the user of the computer system, including moving the first user interface object and the third user interface object (e.g., in a three-dimensional environment) based on the movement of the first part of the user, such as based on, in addition to other objects controlled by the hand 703a. Figure 7C708-6. Adding the third user interface object to the first set of one or more user interface objects optionally has one or more of the characteristics of adding the second user interface object to the first set of one or more user interface objects previously described.

[0143] In some embodiments, based on determining that the second input satisfies one or more second criteria (810d), the computer system updates (810e) the third user interface object to be in a selected state. In some embodiments, the computer system controls (810f) the third user interface object based on the second portion of the user, including moving the third user interface object (e.g., in a three-dimensional environment) based on the movement of the second portion of the user, while maintaining the selected state of the first group of user interface objects and control of one or more user interface objects of the first group based on the first portion of the user of the computer system, such as controlling object 708-6 based on hand 703b, similar to the example of Fig.7D 708-4 described with reference to object 708-4. Updating the third user interface object and controlling the third user interface object optionally have one or more of the characteristics of updating the second user interface object and controlling the second user interface object previously described. Thus, as described above, in some embodiments, multiple different objects can be added to the first set of user interface objects, or multiple different objects can be individually controlled based on the characteristics of the input directed to those objects. Adding the corresponding user interface object to a set of user interface objects being selected and controlled by the first portion of the user or selecting the corresponding user interface object independently of the first set of user interface objects based on a gesture performed by the second portion of the user reduces the number of inputs required to select the user interface object.

[0144] In some embodiments, displaying the second user interface object in a selected state includes displaying a visual indication (812) that the second user interface object is in a selected state via a display generation component, such as FIG. 7A to FIG. 7G A selection box around the selected object and / or FIG. 7A to FIG. 7G For example, displaying a border around a second user interface object that is not displayed when the second user interface object is in an unselected state, changing one or more visual characteristics of the second object (e.g., size, brightness, color, and / or transparency) as compared to when the second object is in an unselected state, and / or displaying the second user interface object with a glow or highlight effect that is not present when the second object is in an unselected state. Displaying a visual indication that the second user interface object is in a selected state automatically communicates the selection state of the second user interface object.

[0145] In some embodiments, when the first group of objects is in a selected state and is controlled by the first portion of the user while the first portion of the user maintains the corresponding posture (e.g., when the first portion of the user is maintained in a pinched hand shape in which the index finger and thumb of the first portion of the user are touching), the computer system detects (814a) that the first portion of the user no longer maintains the corresponding posture, such as in Figure 7C In some embodiments, in response to detecting that the first portion of the user no longer maintains the corresponding posture, the computer system stops (814b) the control of the first portion of the user over the first group of objects while maintaining the first group of objects in the selected state, such as maintaining the control of the first portion of the user over the first group of objects in response to detecting that the hand 703a no longer maintains the pinch hand shape. Figure 7C , and 708-4 in the three-dimensional environment. For example, the first group of objects has been placed at its current location in the three-dimensional environment and will no longer be moved in response to movement of the first portion of the user. However, in response to detecting that the first portion of the user no longer maintains the pose and after the first group of objects is placed at the current location, the first group of objects is optionally maintained in the selected state (e.g., so that additional input directed to the first group of objects will be responded to by the computer system in the manner described herein with respect to the selected objects). Maintaining the first group of objects in the selected state reduces the number of inputs required to subsequently interact with the selected objects.

[0146] In some embodiments, when the first group of objects is in a selected state and is controlled by the first portion of the user while the first portion of the user maintains a corresponding posture (e.g., when the first portion of the user is maintained in a pinched hand shape in which the index finger and thumb of the first portion of the user are touching), the computer system detects (816a) via one or more input devices that the first portion of the user no longer maintains the corresponding posture, such as when Figure 7C In some embodiments, in response to detecting that the first portion of the user no longer maintains the corresponding posture, the computer system stops (816b) the control of the first portion of the user over the first group of objects and updates the first group of objects to be in an unselected state, such as in response to detecting that the hand 703a no longer maintains the pinch hand shape and does not maintain the control of the first portion of the user over the first group of objects. Figure 7C, and the selection of objects 708-1, 708-2, and 708-4 in the three-dimensional environment. For example, the first group of objects has been placed at its current position in the three-dimensional environment and will no longer be moved in response to the movement of the first part of the user. In addition, in response to detecting that the first part of the user no longer maintains the posture and after the first group of objects is placed at the current position, the first group of objects is optionally transformed into an unselected state (e.g., so that other inputs directed to the first group of objects will be responded to by the computer system in the manner described herein with respect to the unselected objects). In some embodiments, if two groups of one or more virtual objects are being controlled by different hands of the user, the selection state of the objects in the two groups of objects is optionally determined by the corresponding hands that are controlling the two groups of objects (e.g., the selection state of the objects in the first group of objects is controlled by the user's left hand in one or more of the above-mentioned ways, and the selection state of the objects in the second group of objects is controlled by the user's right hand in one or more of the above-mentioned ways). Stopping the selected state of the first group of objects reduces the number of inputs required to select a different group of one or more objects for subsequent interactions.

[0147] In some embodiments, the computer system detects (818a) via one or more input devices a second input directed to a location that does not include a user interface object, the second input comprising a first gesture performed by a first portion of the user, such as from hand 703a with a Fig. 7A In some embodiments, the second input has one or more of the characteristics of the first input. In some embodiments, the first gesture (e.g., an air gesture) has one or more of the characteristics of the corresponding gesture. In some embodiments, the user's attention (e.g., as part of the second input) is directed to an area and / or position in the three-dimensional environment that is empty (e.g., does not include any objects) or a position in the three-dimensional environment that includes an unselected object.

[0148] In some embodiments, in response to detecting a second input, the second input satisfies one or more third criteria, the one or more third criteria including a third criterion being satisfied when initiation of the first gesture (e.g., when the computer system detects that the first part of the user is in a “pinch” hand shape / position, as previously described. In some embodiments, initiation of the first gesture is detected when two or more fingers of the first hand of the user move to contact each other) is followed by movement of the first part of the user (e.g., optionally greater than the movement threshold described previously; optionally, regardless of whether the movement is greater than the movement threshold described previously), followed by an end of the first gesture (e.g., when the computer system detects that the fingers of the first part of the user that are making the “pinch” hand shape / position (e.g., the thumb and index finger) move away from each other), such as a gesture and / or movement of hand 703a from lower left to upper right, as previously described. Fig. 7A , where the user's first portion of the movement defines an area (818b) including a plurality of user interface objects, such as those represented by Fig. 7A the first portion of the user releasing the pinch hand shape), based on determining that a third user interface object in the plurality of user interface objects is in a selected state, the computer system updates (818c) the third user interface object to be in an unselected state, such as if when the definition Fig. 7A , object 708-2 is in a selected state, the computer system 101 will optionally place object 708-2 in an unselected state. In some embodiments, the position to which the user's attention is directed at the initiation of the first gesture defines a first vertex of the rectangle in the three-dimensional environment, and the movement of the first part of the user while maintaining the pinching hand shape defines / controls the position of a second vertex diagonally opposite the rectangle in the three-dimensional environment. The rectangle at the end of the first gesture is detected optionally defines the area mentioned above. Multiple user interface objects are optionally those objects that are contained within the boundaries of the rectangle at the end of the first gesture. Multiple user interface objects are not necessarily in the same plane as the rectangle, but are included in a representative volume, which will be generated by extruding the rectangle toward or away from the user's viewpoint in a direction perpendicular to the plane of the rectangle.

[0149] In some embodiments, based on determining that the third user interface object is in an unselected state, such as if Fig. 7AIn some embodiments, based on determining that a fourth user interface object among a plurality of user interface objects is in a selected state, the computer system updates (818e) the fourth user interface object to be in an unselected state. In some embodiments, based on determining that the fourth user interface object is in an unselected state, the computer system updates (818f) the fourth user interface object to be in a selected state. Therefore, the computer system optionally switches back and forth the selection states of a plurality of user interface objects targeted by the area defined by the second input. In some embodiments, after performing a selection based on the second input, individual object elements are displayed for each of the plurality of objects that have been converted to a selected state, and / or group elements are displayed for the plurality of objects in a selected state: corresponding elements that can be selected to manage their corresponding objects (e.g., to cut objects, copy objects, and / or duplicate objects). In some embodiments, detecting that attention is directed outside the area defined by the second input and pinching causes deselection of an object selected based on the second input, while detecting that attention is directed outside the area defined by the second input and pinching causes the computer system to display a context menu associated with the selected object (e.g., to cut the object, copy the object, and / or duplicate the object). Switching the selection state of multiple objects based on the second input reduces the number of inputs required to switch the selection state of multiple objects.

[0150] In some embodiments, when multiple user interface objects are in a selected state (such as in Figure 7B , objects 708-1 and 708-2 are in a selected state), the computer system detects (820a) via one or more input devices a third input directed to a third user interface object that is not included in the plurality of user interface objects and is in an unselected state, wherein the third input includes a second gesture performed by a corresponding portion of the user, such as a pointing gesture from hand 703b. Figure 7B In some embodiments, the third input and / or the second gesture (e.g., an air gesture) includes one or more characteristics of the first input, the second input, the first gesture, and / or the characteristics of the corresponding gesture. In some embodiments, when the third input is detected, the user's attention is directed to the third user interface object. In some embodiments, in response to detecting the third input, based on determining that the third input satisfies one or more first criteria (e.g., a quick pinch air gesture as previously described), the computer system updates (820b) the third user interface object to be in a selected state, while maintaining multiple user interface objects in a selected state, such as Figure 7B, where in addition to objects 708-1 and 708-2 being in a selected state, object 708-4 is also in a selected state. For example, the third user interface object is added to the group of selected objects selected based on the second input.

[0151] In some embodiments, when the third user interface object and the plurality of user interface objects are in the selected state, the computer system detects (820c) a fourth input corresponding to movement of the third user interface object and the plurality of user interface objects, such as from Figure 7B 708-1 via gaze 720a of hand 703a in the embodiment of the present invention. For example, a pinch and drag input directed to at least one of the selected objects includes the user's attention directed to at least one of the selected objects and a pinch and drag gesture performed by the first part or the second part of the user. The pinch and drag air gesture optionally has one or more characteristics of the previously described pinch and drag input.

[0152] In some embodiments, when the fourth input is detected, the computer system moves (820d) the third user interface object and the plurality of user interface objects according to the fourth input, such as according to Figure 7C Objects 708-1, 708-2, and 708-4 are moved according to the movement of the hand 703a in the three-dimensional environment (e.g., including the movement of the first part or the second part of the user). In some embodiments, moving the third user interface object and multiple user interface objects in the three-dimensional environment according to the movement of the first part of the user includes moving those objects based on the magnitude and / or direction of the movement of the first part of the user. For example, if the first part of the user moves up and to the right, those objects are moved up and to the right, or if the first part of the user moves to the left, those objects are moved to the left. The movement amount / magnitude of those objects is optionally also controlled by the movement magnitude of the first part of the user. For example, if the movement amount of the first part of the user is 12cm, those objects are moved a first amount (e.g., proportional to the movement of the first part of the user (e.g., 12cm)), and if the movement amount of the first part of the user is 24cm, those objects are moved a second amount greater than the first amount (e.g., proportional to the movement of the first part of the user (e.g., 24cm)).

[0153] In some embodiments, in response to detecting the end of the fourth input (e.g., when the computer system detects that the fingers (e.g., thumb and index finger) of the first portion or the second portion of the user that are making the "pinch" hand shape / pose move away from each other), the computer system updates (820e) the third user interface object to be in an unselected state while maintaining the multiple user interface objects in a selected state, such as if the computer system 101 detects a click from Figure 7CIn the embodiment of the present invention, if the input of hand 703a in the third user interface object is ended, this will optionally cause objects 708-1 and 708-2 to remain selected, while object 708-4 becomes unselected (e.g., and the plurality of user interface objects and the third user interface object are placed or positioned at their current positions in the three-dimensional environment). Returning the third user interface object to the unselected state in response to the end of the fourth input reduces the number of inputs required to interact with the third user interface object in the unselected state.

[0154] In some embodiments, the computer system detects (822a) via one or more input devices a second input directed to a location including a third user interface object in a selected state, wherein the second input includes a first gesture performed by a first portion of a user, such as from Figure 7B In some embodiments, the second input is directed to object 708-1 via gaze 720a from hand 703a (e.g., in some embodiments, the second input has one or more of the characteristics of the first input. In some embodiments, the first gesture (e.g., an air gesture) has one or more of the characteristics of the corresponding gesture. In some embodiments, the user's attention (e.g., as part of the second input) is directed to an area and / or location in the three-dimensional environment that includes a user interface object in a selected state), and when the second input is detected, one or more user interface objects including a third user interface object are in a selected state, such as when such input from hand 703a is detected. Figure 7B In the example, objects 708-1, 708-2, and 708-4 are in the selected state. For example, when the second input is detected, one or more user interface objects are already in the selected state, and the second input is directed to one of the user interface objects.

[0155] In some embodiments, in response to detecting the second input, the second input satisfies one or more third criteria, the one or more third criteria including a third criterion (822b) being satisfied when the initiation of the first gesture is followed by movement of the first part of the user and then by the end of the first gesture, such as the hand 703a moving from FIG. 7B to FIG. 7C The computer system moves (822c) one or more user interface objects based on the movement of the first portion of the user, such as based on the movement of hand 703a from the first portion of the user, and the movement of hand 703a from the second ... first portion of the user, FIG. 7B to FIG. 7CObjects 708-1, 708-2, and 708-4 are moved according to the movement of the first part of the user. For example, even if the input is directed to one of the selected objects, the other (or optionally, all) selected objects in the selected objects are moved according to the movement of the first part of the user. In some embodiments, moving one or more user interface objects in the three-dimensional environment according to the movement of the first part of the user includes moving those objects based on the magnitude and / or direction of the movement of the first part of the user. For example, if the first part of the user moves up and to the right, those objects are moved up and to the right, or if the first part of the user moves to the left, those objects are moved to the left. The movement amount / magnitude of those objects is optionally also controlled by the movement magnitude of the first part of the user. For example, if the movement amount of the first part of the user is 12cm, those objects are moved by a first amount (e.g., proportional to the movement of the first part of the user (e.g., 12cm)), and if the movement amount of the first part of the user is 24cm, those objects are moved by a second amount greater than the first amount (e.g., proportional to the movement of the first part of the user (e.g., 24cm)). Moving one or more selected user interface objects in response to the same second input reduces the number of inputs required to move the selected objects.

[0156] In some embodiments, the computer system detects (824a) via one or more input devices a second input directed toward a third user interface object in a selected state (e.g., the second input optionally has one or more of the characteristics of the first input), wherein the second input satisfies one or more second criteria (e.g., the second input is a quick pinch input, as previously described), while the user's attention (e.g., gaze, torso orientation, and / or shoulder orientation) is directed toward the third user interface object, such as input from hand 703a and / or pointing to the third user interface object. Figure 7F In some embodiments, in response to detecting the second input, the computer system displays (824b) a context menu associated with the third user interface object, such as a context menu associated with the third user interface object, via a display generation component. Figure 7G 722 shown in . For example, a menu including selectable options for cutting, copying, and / or duplicating a third user interface object is displayed. The menu optionally has one or more of the characteristics of the menus and context menus described with reference to method 1000. Displaying a context menu associated with the selected user interface object in response to the second input reduces the number of inputs required to perform one or more operations associated with the selected user interface object.

[0157] In some embodiments, a first input is detected while the computer system is operating in a first operating mode in which a user interface object is placed in a selected state or an unselected state in one or more first ways (826a) (e.g., based on one or more first criteria or one or more second criteria being satisfied, as previously described, in addition to or in alternative to other methods for selecting the above-mentioned objects). In some embodiments, the computer system displays (826b) a selectable option via a display generation component that can be selected to initiate a second operating mode of the computer system in which a user interface object is placed in a selected state or an unselected state in one or more second ways that are different from the one or more first ways, such as Fig. 7A 714-5 shown in . In some embodiments, in response to detecting the selection of the selectable option, the computer system changes from the first operating mode to the second operating mode. The selectable option is optionally displayed in a user interface element or an option board associated with an object (selected or unselected) displayed via the display generation component. In the second operating mode, the user interface object is optionally selected or deselected in a different manner from when in the first operating mode, as described below. The selectable option is optionally selected in response to detecting a quick pinch air gesture (as previously described) performed by either hand of the user, while the user's attention is directed to the selectable option. In some embodiments, the option board additionally includes one or more selectable options for performing an undo operation, a redo operation, or a cancel operation. In some embodiments, the option board additionally or alternatively includes a "completed" selectable option to leave the second mode and change to the first mode. In some embodiments, when in the second operating mode, the "completed" selectable option includes an indication of the number of user interface objects currently selected according to the second operating mode. Providing different operating modes for selecting objects in different ways provides a more robust and flexible selection of objects.

[0158] In some embodiments, the computer system detects (828a) via one or more input devices a second input directed to a third user interface object that is in an unselected state, wherein the second input satisfies one or more second criteria while the user's attention is directed to the third user interface object, such as from Figure 7B828a, and the computer system displays (828b) content associated with the third user interface object in response to detecting the second input, based on determining that the computer system is operating in the first operating mode, such as displaying content associated with object 708-4 in response to the second input (e.g., without placing the third user interface object in a selected state). For example, the third user interface object is "opened" to display a user interface of a photo application or a messaging application, as previously described.

[0159] In some embodiments, based on determining that the computer system is in a second operating mode (such as FIG. 7E to FIG. 7F The computer system updates (828d) the third user interface object to be in a selected state, such as via Figure 7F 708-7 in the second operation mode (e.g., without displaying content associated with the third user interface object). In some embodiments, in the second operation mode, a quick pinch input directed to different user interface objects toggles the selection state of those user interface objects back and forth (e.g., selected to unselected, or unselected to selected). Performing different operations on the third user interface object according to the operation mode enables selection of the object or display of content associated with the object to be performed without displaying additional controls.

[0160] In some embodiments, one or more user interface objects (830a) are displayed within a container user interface object, such as object 710 (e.g., a container or canvas object, such as described with reference to methods 1000 and / or 1200). In some embodiments, the computer system detects (830b) via one or more input devices a second input comprising movement of a corresponding part of the user (e.g., left or right hand), wherein the second input satisfies one or more second criteria, such as movement from hand 703a to the user's left hand. FIG. 7B to FIG. 7C Input / movement of the second input. For example, the second input is a pinch and drag input such as previously described. In some embodiments, when the second input is detected, the user's attention is directed to a user interface object (e.g., selected or unselected) in the one or more user interface objects. In some embodiments, when the second input is detected, the user's attention is not directed to a user interface object in the one or more user interface objects (e.g., to an empty portion of a container user interface object).

[0161] In some embodiments, in response to detecting the second input (830c), based on determining that the computer system is operating in the first operating mode, the computer system moves the corresponding part of the user (such as objects 708-1, 708-2, and 708-4 from the original position to the original position) according to the movement of the corresponding part of the user (such as objects 708-1, 708-2, and 708-4 from the original position to the original position). FIG. 7B to FIG. 7C The method of moving (830d) one or more user interface objects relative to the container user interface object (e.g., movement of the corresponding portion of the user) is performed. For example, one or more user interface objects (e.g., those user interface objects that are selected, including the selected user interface object to which the user's attention is directed) are moved within the container user interface object without scrolling through the container user interface object. In some embodiments, one or more user interface objects are moved so that the relative placement of those one or more user interface objects relative to other user interface objects contained within the container changes according to the movement of the corresponding portion of the user. In some embodiments, moving one or more user interface objects according to the movement of the corresponding portion of the user includes moving the object based on the magnitude and / or direction of the movement of the corresponding portion of the user. For example, if the corresponding portion of the user moves up and to the right, then those objects are moved up and to the right, or if the corresponding portion of the user moves to the left, then those objects are moved to the left. The amount / magnitude of movement of those objects is optionally also controlled by the magnitude of the movement of the corresponding portion of the user. For example, if the amount of movement of the corresponding part of the user is 12 cm, those objects are moved a first amount (e.g., proportional to the movement of the corresponding part of the user (e.g., 12 cm)), and if the amount of movement of the corresponding part of the user is 24 cm, those objects are moved a second amount greater than the first amount (e.g., proportional to the movement of the corresponding part of the user (e.g., 24 cm)).

[0162] In some embodiments, based on determining that the computer system is operating in the second operating mode, the computer system scrolls (830d) through the container user interface object without moving one or more user interface objects relative to the container user interface object, such as based on a movement from hand 703a. FIG. 7E to FIG. 7FScrolling through the container user interface object optionally reveals one or more user interface objects not previously displayed in the container (e.g., scrolling onto a visible portion of the container) and / or causes one or more user interface objects previously displayed to become hidden (e.g., scrolling off a visible portion of the container). In some embodiments, scrolling through the container user interface object does not cause the relative placement of the user interface objects contained within the container to change relative to each other. The container user interface object optionally displays a portion of the canvas that is larger than the container user interface object, and scrolling through the container user interface object optionally causes the container user interface object to display a portion of the canvas that is different from the portion previously displayed. In some embodiments, scrolling through the container according to movement of the corresponding portion of the user includes scrolling through the container based on a magnitude and / or direction of movement of the corresponding portion of the user. For example, if the corresponding portion of the user moves to the right, the container is scrolled in a rightward direction (e.g., to reveal a portion of the canvas on the left side of the container and hide a portion of the canvas on the right side of the container), or if the corresponding portion of the user moves to the left, the container is scrolled in a leftward direction (e.g., to reveal a portion of the canvas on the right side of the container and hide a portion of the canvas on the left side of the container). The scrolling amount / magnitude is optionally also controlled by the magnitude of the movement of the corresponding portion of the user. For example, if the amount of movement of the corresponding portion of the user is 12 cm, the container is scrolled a first amount (e.g., proportional to the movement of the corresponding portion of the user (e.g., 12 cm)), and if the amount of movement of the corresponding portion of the user is 24 cm, the container is scrolled a second amount greater than the first amount (e.g., proportional to the movement of the corresponding portion of the user (e.g., 24 cm)). Performing different operations on one or more user interface objects according to the operation mode enables movement of user interface objects or scrolling of containers without displaying additional controls.

[0163] It should be understood that the specific order in which the operations in method 800 are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0164] 9A to 9F An example of a computer system that places objects at locations in a display area based on a user's attention is shown according to some embodiments.

[0165] Fig. 9A The display generation components (eg, Figure 1101 ) is visible from a user's viewpoint 916 shown in a top view 914 (e.g., facing the back wall of the physical environment in which the computer system 101 is located and near the back left corner of the physical environment). Figures 1 to 6 As described above, the computer system 101 optionally includes a display generation component (e.g., a touch screen) and a plurality of image sensors (e.g., Figure 3 The image sensor 314 may include one or more of: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the computer system 101 can use to capture one or more images of a user or a portion of a user (e.g., one or more hands of a user) as the user interacts with the computer system 101. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display that includes a display generation component that displays a user interface or a three-dimensional environment to the user, as well as sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors that face outward from the user) and / or sensors that detect the user's gaze (e.g., internal sensors that face inward toward the user's face).

[0166] like Fig. 9A , computer system 101 captures one or more images of a physical environment 902 (e.g., operating environment 100) surrounding computer system 101 (including one or more objects in the physical environment surrounding computer system 101). In some embodiments, computer system 101 displays a representation of the physical environment in a three-dimensional environment 904 and / or physical environment 902 is visible in three-dimensional environment 904 via display generation component 120. For example, three-dimensional environment 904 visible via display generation component 120 includes a representation of the physical floor and back and side walls of the room in which computer system 101 is located. Three-dimensional environment 904 also includes table 906 (via display generation component 120 and shown in top view 914).

[0167] exist Fig. 9A, the three-dimensional environment 904 also includes virtual objects 908-1 to 908-4, virtual object 910, and virtual object 912. Virtual objects 908-1 to 908-4 are optionally one or more of user interfaces of applications (e.g., messaging user interfaces, content browsing user interfaces, etc.), three-dimensional objects (e.g., virtual clocks, virtual balls, virtual cars, etc.), or any other elements displayed by computer system 101 that are not included in the physical environment of computer system 101. Virtual object 910 is optionally a container or canvas object that contains virtual objects 908-1 to 908-4 in a composition area of ​​object 910, as described in more detail with reference to methods 800, 1000, and / or 1200. Virtual object 912 is optionally a tool palette object for performing operations related to virtual objects 908-1 to 908-4 and / or 910. For example, object 912 includes a selectable option 914-1 that can be selected to undo the last operation performed associated with virtual objects 908-1 to 908-4 and / or 910; a selectable option 914-2 that can be selected to redo the last operation canceled or undone associated with virtual objects 908-1 to 908-4 and / or 910; a selectable option 914-3 that can be selected to edit the currently selected object of virtual objects 908-1 to 908-4; a selectable option 914-4 that can be selected to add a new virtual object to object 910; and a selectable option 914-5 that can be selected to cause the computer system 101 and / or object 910 to transition to a selection mode, which is described in more detail with reference to method 800.

[0168] As described in more detail with reference to method 800, one or more of objects 908-1 through 908-4 can be selected and / or can be highlighted to designate those objects as targets for further operations (e.g., a move operation, a copy operation, or a cut operation). Fig. 9A, object 908-2 is selected, as indicated by the dashed border around it. The selected object is shown in the top view 914 as having a bold appearance to distinguish it from the unselected objects (which are shown as not having a bold appearance). In some embodiments, the selected two-dimensional object is separated from the object 910 (e.g., indicated by the projected shadow shown for the object 908-2), and the unselected two-dimensional objects are not separated from the object 910 or are less separated from the object 910 (e.g., indicated by the projected shadow not shown for the objects 908-1 and 908-3). In some embodiments, the three-dimensional object is separated from the object 910 regardless of whether the three-dimensional object is selected (e.g., indicated by the projected shadow shown for the object 908-4). In some embodiments, when the three-dimensional object is selected, the three-dimensional object is separated from the object 910 more than when the three-dimensional object is not selected; in some embodiments, the separation of the three-dimensional object from the object 910 does not change based on whether the three-dimensional object is selected.

[0169] exist Fig. 9A , when object 908-2 is selected, computer system 101 detects gaze 920 directed toward object 908-2 when hand 903 is in the ready state. In response, computer system 101 displays option 922 (e.g., on object 908-2), which can be selected to cause the computer system to display a context menu for object 908-2. In response to selection of option 922, computer system 101 displays Fig. 9B Context menu 924a in . Selection of option 922 optionally occurs in response to computer system 101 detecting the user's gaze 920 directed toward option 922 when hand 903 performs an air pinch gesture (in which the thumb and index finger of hand 903 are brought together and touching, and then moved away from each other).

[0170] like Fig. 9B As shown in , in response to selection of option 922, object 908-2 optionally remains selected. In addition, Fig. 9BThe context menu 924a in includes an option 922-1 that can be selected to copy object 908-2 (e.g., for later pasting, as described in more detail with reference to method 1000); an option 922-2 that can be selected to cut object 908-2 (e.g., for later pasting, as described in more detail with reference to method 1000); an option 922-3 that can be selected to copy object 908-2 (e.g., as described in more detail with reference to method 1000); an option 922-4 that can be selected to flip (e.g., horizontally and / or vertically) object 908-2; an option 922-5 that can be selected to open object 908-2 (e.g., in a specified application that displays content associated with object 908-2); and an option 922-6 that can be selected to delete object 908-2.

[0171] exist Fig. 9B , computer system 101 detects input selecting option 922-1. As described above, the selection of option 922-1 optionally occurs in response to computer system 101 detecting the user's gaze 920 directed toward option 922-1 when hand 903 performs an air pinch gesture.

[0172] exist Fig. 9B After the input is detected in Fig. 9C , computer system 101 detects input for displaying a different context menu associated with object 910, as will now be described. Fig. 9C The different inputs and resulting context menus shown in are optionally alternative scenes or scenes that appear in sequence. Specifically, in response to detecting an air pinch gesture performed by hand 903, as described above, when the user's gaze is directed to an empty space in object 910, the computer system 101 displays a context menu at a location corresponding to the location of the user's gaze when the air pinch gesture is detected. For example, in response to detecting an air pinch gesture from hand 903 when gaze 920a is directed to the upper left portion of object 910, the computer system displays context menu 924b at or near the location of gaze 920a. Context menu 924b includes a paste option that can be selected to paste a copy of object 908-2, which is selected in response to Fig. 9B . Computer system 101 optionally displays context menu 924c in response to detecting an air pinch gesture from hand 903 when gaze 920b is directed to the center portion of object 910, and displays context menu 924d in response to detecting an air pinch gesture from hand 903 when gaze 920c is directed to the middle right portion of object 910. Context menus 924c and 924d also include a paste option as described above.

[0173] In response to detecting selections of respective paste options in menus 924b, 924c, and 924d, computer system 101 optionally initiates a process of displaying a copy of object 908-2 at the location of the user's gaze when the paste option is selected. The paste option in menu 924b is optionally selected in response to detecting an air pinch gesture from hand 903 when gaze 920d is directed to the paste option, the paste option in menu 924c is optionally selected in response to detecting an air pinch gesture from hand 903 when gaze 920e is directed to the paste option, and the paste option in menu 924d is optionally selected in response to detecting an air pinch gesture from hand 903 when gaze 920f is directed to the paste option.

[0174] Fig.9D The computer system 101 is shown Fig. 9C For example, referring to gaze 920e, computer system 101 responds to the selection of the paste option in Fig. 9C 908-2) at the location of gaze 920e. In some embodiments, as shown, gaze 920e defines the center point of object 908-6. In addition, computer system 101 displays object 908-6 behind object 908-4 because object 908-4 is a three-dimensional object and a two-dimensional object (such as object 908-6) is pasted behind a three-dimensional object (such as object 908-4). In contrast, referring to gaze 920f, the computer system 101 displays object 908-6 (a copy of object 908-2) at the location of gaze 920e in response to the input in the figure. In some embodiments, as shown, gaze 920e defines the center point of object 908-6. In addition, computer system 101 displays object 908-6 behind object 908-4 because object 908-4 is a three-dimensional object and a two-dimensional object (such as object 908-6) is pasted behind a three-dimensional object (such as object 908-4). In contrast, referring to gaze 920f, the computer system 101 displays object 908-6 (a copy of object 908-2) at the location of gaze 920e. Fig. 9C In the input of the two-dimensional object, object 908-7 (a copy of object 908-2) is displayed in front of object 908-3 because object 908-7 is placed in object 910, and object 908-3 is already in object 910. In addition, because computer system 101 places object 908-7 at the location of gaze 920f and gaze 920f is relatively close to the right border of object 910, object 908-7 is displayed as partially extending beyond the right border of object 910. In some embodiments, computer system 101 allows such placement of objects that extend beyond the right border or left border of object 910 and does not automatically reposition such objects within object 910. In some embodiments, computer system 101 adjusts the visual appearance of portions of object 908-7 that extend beyond the right border or left border of object 910 (e.g., the further the portions extend beyond the border, the more the translucency of those portions is increased, and / or the further the portions extend beyond the border, the more the brightness of those portions is reduced).

[0175] Computer system 101 optionally does not allow an object to be placed in object 910 such that it extends beyond the top and bottom boundaries of object 910, as opposed to placing an object outside the left and right boundaries of object 910. For example, referring to gaze 920d, if the computer system has responded to Fig. 9C 920d, object 908-5 (a copy of object 908-2) is displayed centered at gaze 920d, then object 908-5 will optionally extend beyond the top boundary of object 910. Conversely, in response to Fig. 9C Based on the input in , the computer system 101 has automatically offset the position for placing the object 908-5 away from the top boundary toward the interior of the object 910, and placed the object 908-5 at this offset position, which is different from the position defined by the gaze 920d, so that when displayed at this position, the object 908-5 does not extend beyond the top boundary of the object 910. In some embodiments, the computer system 101 offsets the position of the object 908-5 to the nearest position (e.g., direction and / or distance) at which the object 908-5 will not extend beyond the top boundary of the object 910. The computer system 101 optionally performs similar operations to prevent the object from extending beyond the bottom boundary of the object 910.

[0176] Now refer to FIG. 9E to FIG. 9F Describes inserting a three-dimensional object into object 910. Fig.9E , a 3D object has been copied or cut, such as the reference FIG. 9A to FIG. 9B The object 908-2 in the description, for example, has been copied or cut Fig. 9B In addition, object 910 is being displayed with 9A to 9D different objects in the same 9A to 9D 800. In addition, the computer system 101 is displaying a menu 924e based on gaze 920a, a menu 924f based on gaze 920c, and a menu 924g based on gaze 920b (e.g., in response to the menu 924e being displayed in the computer system 101 in accordance with the present invention). Fig. 9C ), these menus are optionally alternatives to those previously described. Fig.9E As previously described, the computer system 101 detects mid-air pinch selection inputs for the paste options in menus 924e, 924f, and 924g based on gazes 920a, 920c, and 920e, respectively.

[0177] In response to Fig.9E The computer system 101 optionally places the pasted three-dimensional object in different ways, such as referring to Fig.9FFor example, referring to gaze 920f, computer system 101 responds to Fig.9E 920f. In some embodiments, as shown, gaze 920f defines the center point of object 908-22. In addition, computer system 101 displays object 908-22 in front of object 908-8 / in a manner that obscures the object because object 908-8 is a two-dimensional object and because object 908-22 is placed within object 910, while object 908-8 is already within object 910.

[0178] In contrast, computer system 101 optionally does not allow a three-dimensional object to be placed in object 910 such that the three-dimensional object occludes or conflicts with other three-dimensional objects. For example, referring to gaze 920d, if the computer system has responded to Fig.9E 920d is displayed with the gaze 920d as the center, then the object 908-24 will optionally occlude and / or conflict with the object 908-23 as a three-dimensional object. Fig.9E Based on the input in , the computer system 101 has automatically offset the position for placing object 908-24 from object 908-23, and placed object 908-24 at the offset position different from the position defined by gaze 920d, so that when displayed at this position, object 908-24 does not conflict with or obstruct object 908-23. In some embodiments, the computer system 101 offsets the position (e.g., direction and / or distance) of object 908-24 to the nearest position at which object 908-24 will not conflict with or obstruct object 908-23. In some embodiments, as previously described with reference to Fig.9D As depicted, computer system 101 additionally prevents object 908 - 24 from extending beyond the top or bottom boundaries of object 910 .

[0179] In some embodiments, if there is no three-dimensional object within a threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 20 cm, 50 cm, or 100 cm) of the user's gaze that will not conflict with or obstruct the position of another three-dimensional object, the computer system 101 does not offset the position of the inserted three-dimensional object. For example, with reference to the gaze 920 e positioned in the middle of the plurality of three-dimensional objects 908-9 to 908-20, the computer system 101 has responded to Fig.9E920e. In response to the input in the above-mentioned threshold distance of gaze 920e, object 908-21 (a copy of object 908-4) centered at gaze 920e is displayed. When displayed at the position of gaze 920e, object 908-21 optionally conflicts with and / or obscures one or more of objects 908-9 to 908-20. However, computer system 101 has not yet offset the position of displayed object 908-21 because there is no position within the above-mentioned threshold distance of gaze 920e where object 908-21 can be displayed without conflicting with or obscuring one or more of objects 908-9 to 908-20.

[0180] FIG. 10A to FIG. 10J 1 is a flowchart illustrating a method 1000 for placing an object at a location in a display area based on a user's attention according to some embodiments. In some embodiments, the method 1000 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet device, a smart phone, a wearable computer, or a head mounted device, the computer system including a display generating component (e.g., Figure 1 , Figure 3 and Figure 4 In some embodiments, method 1000 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1 Some operations in method 1000 are optionally combined, and / or the order of some operations is optionally changed.

[0181] In some embodiments, method 1000 is performed at a computer system (e.g., 101) in communication with a display generation component (e.g., 120) and one or more input devices. In some embodiments, the computer system is the same or similar to the computer system described above with reference to method 800. In some embodiments, the one or more input devices are the same or similar to the one or more input devices described above with reference to method 800. In some embodiments, the display generation component is the same or similar to the display generation component described above with reference to method 800.

[0182] In some embodiments, (e.g., when a three-dimensional environment (e.g., such as the three-dimensional environment described in methods 800 and / or 1200) is visible via a display generation component), when a composition area (e.g., an area of ​​object 910 (e.g., a three-dimensional environment and / or a two-dimensional or three-dimensional area of ​​a container or canvas user interface object as described in reference methods 800, 1000, and / or 1200)) is displayed via the display generation component, the computer system receives (1002a) via one or more input devices corresponding to a request to insert a first (e.g., two-dimensional or three-dimensional) virtual object into the composition area, such as reference methods 800, 1000, and / or 1200. Fig. 9C In some embodiments, the first input and / or other inputs described with reference to method 1000 are or include air gesture inputs. In some embodiments, the first input is detected after performing an operation associated with another virtual object (such as an operation for copying at least a portion of another virtual object that is then pasted as the first virtual object). For example, after "cutting" or "copying" another virtual object, the computer system receives a request to "paste" the first virtual object. In some embodiments, the first virtual object and / or other virtual objects described in method 1000 are displayed at or within the first virtual object described in method 1200. Similarly, in some embodiments, displaying the first virtual object includes displaying the first virtual object at or within the first virtual object described in method 1200. In some embodiments, the computer system stops displaying the other virtual object after performing the first operation (for example, if the first operation corresponds to a request to "cut" another virtual object). Alternatively, in some embodiments, the computer system maintains the display of another virtual object after performing the first operation (e.g., if the first operation corresponds to a request to "copy" another virtual object. In some embodiments, if the first operation corresponds to a request to "cut" another virtual object, performing the first operation optionally includes storing the other virtual object in a memory and stopping the display of the other virtual object. In some embodiments, if the first operation corresponds to a request to "copy" another virtual object, performing the first operation optionally includes creating a copy of the other virtual object and storing the copy of the other virtual object in a memory without modifying / changing the display and / or position of the other virtual object in the three-dimensional environment. In some embodiments, the first virtual object (e.g., the newly displayed virtual object) corresponds to the user interface object that was last "cut" or "copied" by the computer system (e.g., has the same characteristics as the user interface object that was last "cut" or "copied" by the computer system, such as size, shape, color, area, and / or volume).

[0183] In some embodiments, in response to receiving the first input (1002b), the user of the computer system determines that attention is directed to a first location in the composition area (e.g., in a three-dimensional environment) when the first input is received, such as by Fig. 9C 920e in the user's gaze (e.g., in some embodiments, an eye tracking device of one or more input devices determines that when the first input is received, the user's gaze is directed to the first location (e.g., in a three-dimensional environment)), the computer system displays (1002c) the first virtual object at the first location, such as via Fig.9D 908-6 in the 3D environment (e.g., in the 3D environment). For example, the first virtual object is displayed at a first position in the 3D environment (e.g., in the lower right portion of the 3D environment relative to the user's viewpoint) because when the first input is received, the user's attention is directed to the first position in the 3D environment (e.g., when the first input is received, the user's gaze is directed to the lower right portion of the 3D environment relative to the user's viewpoint). In some embodiments, the first position in the 3D environment corresponds to an X position (e.g., a horizontal position relative to the user's viewpoint), a Y position (e.g., a vertical position relative to the user's viewpoint), and / or a Z position (e.g., a distance relative to the user's viewpoint) in the 3D environment. In some embodiments, if the user's gaze is directed to a first location in the three-dimensional environment when the first input is received, a second virtual object is displayed in the three-dimensional environment at an X location that is offset (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm) from the first location in the three-dimensional environment, at a Y location in the three-dimensional environment that is offset (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm) from the first location in the three-dimensional environment, and / or at a Z location in the three-dimensional environment that is offset (e.g., 1 cm, 3 cm, 5 cm, 7 cm, 10 cm, 20 cm, or 30 cm) from the first location in the three-dimensional environment. In some embodiments, displaying the first virtual object at the first location includes displaying a center of the first virtual object at the first location in the three-dimensional environment.

[0184] In some embodiments, based on determining, upon receiving the first input, that the attention of the user of the computer system is directed to a second location in the composition area that is different from the first location (e.g., in a three-dimensional environment), such as by Fig. 9C The computer system displays (1002d) the first virtual object at the second location, such as via Fig.9D908-7 in the three-dimensional environment (e.g., in the three-dimensional environment). For example, the first virtual object is displayed at a second position in the three-dimensional environment (e.g., in the upper left portion of the three-dimensional environment relative to the user's viewpoint) because when the first input is received, the user's attention is directed to the second position in the three-dimensional environment (e.g., when the first input is received, the user's gaze is directed to the upper left portion of the three-dimensional environment relative to the user's viewpoint). In some embodiments, displaying the first virtual object at the second position includes displaying the center of the first virtual object at the second position in the three-dimensional environment. In some embodiments, displaying the first virtual object at the second position has one or more of the characteristics described above relative to displaying the first virtual object at the first position. Adding the first virtual object to the three-dimensional environment based on the user's attention enables the computer system to automatically determine the appropriate position of the first virtual object in the three-dimensional environment.

[0185] In some embodiments, while the composition area is displayed, the computer system detects (1004a) a second input via one or more input devices, the second input comprising a corresponding portion of a user performing a corresponding gesture when the user's attention is directed to an empty portion of the composition area, such as a reference to Fig. 9C In some embodiments, the second input is a pinch air gesture input (e.g., a long pinch or a quick pinch) as described with reference to method 800. In some embodiments, in response to detecting the second input, the computer system displays (1004b) a context menu associated with the composition area, such as menu 924a, 924b, 924c, or 924d, via a display generation component. The context menu optionally includes one or more selectable options for performing operations relative to the composition area as described below. Displaying the context menu in response to the second input reduces the number of inputs required to access one or more operations associated with the composition area.

[0186] In some embodiments, the context menu includes a selectable option (1006) that can be selected to paste the virtual object into the composition area, such as shown in menus 924b, 924c and 924d. In some embodiments, the first input includes the selection of the selectable option. In some embodiments, the selection of the selectable option occurs in response to the computer system detecting a pinch gesture in the air (e.g., a quick pinch or a long pinch) (as previously described with reference to method 800), while the user's attention is directed to the selectable option. In some embodiments, the selection of the selection option occurs in response to the computer system detecting a direct input (e.g., the user's finger seeks and touches the selectable option) to select the selectable option. In some embodiments, the virtual object to be pasted into the composition area was previously copied or cut. The option of pasting the virtual object in the context menu reduces the number of inputs required to paste the virtual object into the composition area.

[0187] In some embodiments, the context menu includes one or more selectable options (1008) in addition to the selectable option that can be selected to paste the virtual object into the composition area, such as Fig. 9B In some embodiments, the selection of selectable options occurs as previously described with reference to the selection of selectable options for pasting virtual objects into the composition area. In some embodiments, one or more selectable options include one or more of the following: a selectable option for cutting one or more selected virtual objects (e.g., as described with reference to method 800), a selectable option for copying one or more selected virtual objects (e.g., as described with reference to method 800), a selectable option for rotating one or more selected virtual objects, a selectable option for horizontally flipping one or more selected virtual objects, a selectable option for vertically flipping one or more selected virtual objects, a selectable option for grouping one or more selected virtual objects together, a selectable option for specifying an application in which to open one or more selected virtual objects, a selectable option for deleting one or more selected virtual objects, and / or a selectable option for temporarily displaying content associated with one or more selected virtual objects. Displaying various options for performing one or more various operations associated with the composition area in the context menu reduces the number of inputs required for performing various operations associated with the composition area.

[0188] In some embodiments, when displaying the second virtual object in the composition area via the display generation component, the computer system detects (1010a) the user's attention directed toward the second virtual object, such as directed toward Fig. 9AIn some embodiments, in response to detecting the user's attention directed toward the second virtual object, the user's gaze 920 (e.g., the user's gaze, head, and / or torso) of the object 908-2 in the second virtual object is detected. In some embodiments, in response to determining that the corresponding part of the user (e.g., the user's hand) is in a predefined posture (such as relative to the second virtual object), the user's gaze 920 is detected. Fig. 9A 903 in the hand) (e.g., the ready state of the user's hand, as previously described), the computer system displays (1010b) via the display generation component selectable options that can be selected to display a menu associated with the second virtual object, such as Fig. 9A Option 922 in. For example, in response to the computer system detecting the user's attention directed to the second virtual object and the user's hand in a ready state, a button that can be selected to display a context menu associated with the second virtual object is displayed. In some embodiments, the button is selected in one or more of the above-mentioned ways of selecting selectable options. In some embodiments, the button is not displayed unless the ready state and the user's attention are detected. In some embodiments, the button is displayed as an overlay on the second virtual object. Displaying selectable options for accessing a menu associated with the second virtual object reduces display clutter.

[0189] In some embodiments, when displaying a selectable option that can be selected to display a menu associated with a second virtual object, the computer system detects (1012a) a second input that includes a corresponding portion of the user (e.g., the user's hand) performing a corresponding gesture when the user's attention is directed to the selectable option, such as a pointing gesture from hand 903. Fig. 9A In some embodiments, the second input is a pinch air gesture input (e.g., a long pinch or a quick pinch) as described with reference to method 800. In some embodiments, in response to detecting the second input, the computer system displays (1012b) a menu associated with the second virtual object via the display generation component, wherein the menu associated with the second virtual object includes one or more selectable options that can be selected to perform one or more operations associated with the second virtual object, such as Fig. 9B In some embodiments, the selectable options included in the menu are selected in one or more of the above-described ways of selecting selectable options. In some embodiments, the one or more selectable options include one or more of the following: a selectable option to cut the second virtual object, a selectable option to copy the second virtual object, and / or a selectable option to paste the previously copied or cut virtual object at the location of the second virtual object. Displaying selectable options for performing one or more operations associated with the second virtual object reduces the number of inputs required to access those one or more operations.

[0190] In some embodiments, displaying a menu associated with the second virtual object includes (1014a) based on determining that the computer system has previously detected an input to copy at least a portion of the corresponding virtual object (e.g., before the second input - in some embodiments, within a time threshold (such as 1 minute, 3 minutes, 5 minutes, 20 minutes, 45 minutes, 60 minutes, 120 minutes, 240 minutes, or 480 minutes) before the second input. In some embodiments, the input to copy at least a portion of the corresponding virtual object is an input to cut at least a portion of the corresponding virtual object), displaying a first selectable option in the menu that can be selected to paste at least a portion of the corresponding virtual object in the composition area (1014b), such as, in Fig. 9B In some embodiments, based on determining that the computer system has not previously detected an input to copy at least a portion of the corresponding virtual object, the first selectable option (1014c) is abandoned in the menu, such as Fig. 9B The context menu for the second virtual object does not include a paste option 924. Thus, in some embodiments, the context menu for the second virtual object only includes an option to paste if the computer system has previously (recently) detected an input to cut or copy the virtual object. Displaying the first selectable option based on whether the computer system has previously detected an input to copy the virtual object causes the computer system to automatically include or exclude the relevant operation to be performed in the context menu.

[0191] In some embodiments, while displaying a composition area (e.g., a two-dimensional or three-dimensional area of ​​a three-dimensional environment and / or a container or canvas user interface object as described with reference to methods 800, 1000, and / or 1200) via a display generation component, the computer system receives (1016a) via one or more input devices a second input corresponding to a request to insert a plurality of virtual objects into the composition area, such as in Fig.9E The second input is optionally a selection of a paste option to paste the previously copied or cut multiple virtual objects. In some embodiments, the multiple virtual objects have been previously grouped together (e.g., in response to a grouping operation as previously described). In some embodiments, the multiple virtual objects have previously been in a selected state together without being grouped together (e.g., as described with reference to method 800).

[0192] In some embodiments, in response to receiving the second input (1016b), based on determining that the attention of the user of the computer system was directed to a third location in the composition area (e.g., in a three-dimensional environment) when the second input was received, the computer system displays (1016c) multiple virtual objects in a spatial arrangement in the composition area (e.g., the spatial arrangement of the multiple virtual objects is optionally the same spatial arrangement of corresponding multiple virtual objects that are copied and based on which the multiple virtual objects are based), wherein a particular portion of the spatial arrangement is at the third location in the composition area, such as inserting objects 908-9 to 908-20 into object 910, thereby causing objects 908-9 to 908-20 (e.g., Fig.9E In some embodiments, the specific portion of the spatial arrangement is the center (and / or centroid) of the spatial arrangement. In some embodiments, the specific portion of the spatial arrangement is a different portion of the spatial arrangement. Placing multiple virtual objects in a composition area based on user attention optionally has one or more of the characteristics of placing a first virtual object in a composition area based on user attention.

[0193] In some embodiments, based on determining that the attention of the user of the computer system is directed to a fourth position in the composition area that is different from the third position (e.g., in a three-dimensional environment) when the second input is received, the computer system displays (1016d) multiple virtual objects in a spatial arrangement in the composition area (e.g., the same spatial arrangement as when the user's attention was directed to the third position), wherein a specific portion of the spatial arrangement is at the fourth position in the composition area, such as inserting objects 908-9 to 908-20 into object 910, thereby causing objects 908-9 to 908-20 (e.g., Fig.9E In some embodiments, the user's attention controls the position of a particular portion of the spatial arrangement of the plurality of virtual objects. Adding the plurality of virtual objects to the composition area based on the user's attention causes the computer system to automatically determine appropriate positions for the plurality of virtual objects.

[0194] In some embodiments, when the composition area is displayed via the display generation component, the computer system receives (1018a) via one or more input devices a second input corresponding to a request to insert the second virtual object into the composition area, such as Fig. 9CIn some embodiments, the second input has one or more of the characteristics of the first input, as previously described. In some embodiments, in response to receiving the second input (1018b), the user of the computer system determines that the attention of the user is directed to a third location in the composition area (e.g., in a three-dimensional environment) when the second input is received, wherein the third location is within a threshold distance (e.g., 0.1 cm, 0.2 cm, 0.5 cm, 1 cm, 2 cm, 3 cm, 5 cm, 10 cm, or 20 cm) of the lower boundary of the composition area, such as if Fig. 9C In some embodiments, the computer system displays (1018c) a second virtual object at a fourth position in the composition area, which is farther from the lower boundary of the composition area than the threshold distance, similar to the manner in which the computer system 101 displays object 908-5 at a position farther from the upper boundary of object 910 than the threshold distance when the gaze 920d in the composition area is within a threshold distance of the lower boundary of the object 910 (e.g., the lower edge of the composition area, optionally corresponding to a boundary that the composition area does not extend downwardly across), and when the gaze 920d in the composition area is within a threshold distance of the lower boundary of the object 910 (e.g., the lower edge of the composition area, optionally corresponding to a boundary that the composition area does not extend downwardly across), the computer system displays (1018c) a second virtual object at a fourth position in the composition area, which is farther from the lower boundary of the composition area than the threshold distance, similar to the manner in which the computer system 101 displays object 908-5 at a position farther from the upper boundary of object 910 than the threshold distance when the gaze 920d is within a threshold distance of the upper boundary. For example, the computer system optionally displays the second virtual object at a position that is more upward than the position of the user's attention (e.g., the computer system offsets the second virtual object upward from the lower boundary so that the second virtual object does not extend beyond the lower boundary). In some embodiments, the threshold distance is based on the size of the second virtual object (e.g., is different for virtual objects of different sizes). In some embodiments, the threshold distance corresponds to the vertical size of the second virtual object, is based on the vertical size of the second virtual object, or is the same as the vertical size of the second virtual object. In some embodiments, the horizontal positioning of the fourth position is the same as the horizontal positioning of the third position (e.g., the second virtual object is merely offset vertically from the position of attention).

[0195] In some embodiments, based on determining, when receiving the second input, that the attention of the user of the computer system is directed to a fifth location in the composition area (e.g., in a three-dimensional environment), wherein the fifth location is outside a threshold distance from a lower boundary of the composition area, such as Fig. 9C In response to an input associated with gaze 920e in FIG. 1 , the computer system displays ( 1018d ) a second virtual object at a fifth position in the composition area, including displaying the second virtual object within a lower boundary of the composition area without displaying a portion of the second virtual object outside the lower boundary of the composition area, such as Fig.9D In some embodiments, when displayed at the fifth position, the second virtual object is completely within the composition area, rather than partially outside the composition area (e.g., outside the lower boundary of the composition area). Offsetting the position of the second virtual object causes the computer system to automatically position the second virtual object at an appropriate location in the composition area.

[0196] In some embodiments, in response to receiving the second input (1019a), the user's attention of the computer system is directed to a sixth position in the composition area (e.g., in a three-dimensional environment) based on determining when the second input is received, where the sixth position is within a threshold distance (e.g., 0.1 cm, 0.2 cm, 0.5 cm, 1 cm, 2 cm, 3 cm, 5 cm, 10 cm, or 20 cm) of an upper boundary of the composition area, such as Fig. 9C 920d in the composition area (e.g., an upper edge of the composition area, optionally corresponding to a boundary across which the composition area does not extend upward), the computer system displays (1019b) a second virtual object at a seventh position in the composition area that is farther from the upper boundary of the composition area than a threshold distance, such as via Fig.9D 908-5 in the composition area. For example, the computer system optionally displays the second virtual object at a position further downward than the position of the user's attention (e.g., the computer system offsets the second virtual object downward from the upper boundary so that the second virtual object does not extend beyond the upper boundary). In some embodiments, the horizontal positioning of the seventh position is the same as the horizontal positioning of the sixth position (e.g., the second virtual object is only offset vertically from the position of attention). Offsetting the position of the second virtual object enables the computer system to automatically position the second virtual object at an appropriate position in the composition area.

[0197] In some embodiments, while a composition area (e.g., a three-dimensional environment and / or a two-dimensional or three-dimensional area of ​​a container or canvas user interface object as described with reference to methods 800, 1000, and / or 1200) is displayed via a display generation component, a computer system receives (1020a) via one or more input devices a second input corresponding to a request to insert a second virtual object into the composition area. In some embodiments, as previously described, the second input has one or more of the characteristics of the first input. In some embodiments, in response to receiving the second input (1020b), a user of the computer system determines that attention is directed to a third location in the composition area (e.g., in a three-dimensional environment) when the second input is received, wherein the third location is within a threshold distance (e.g., 0.1 cm, 0.2 cm, 0.5 cm, 1 cm, 2 cm, 3 cm, 5 cm, 10 cm, or 20 cm) of a lateral (e.g., left or right) boundary of the composition area, such as with Fig. 9C In response to an input associated with gaze 920f in the composition area, the computer system displays (1020c) a second virtual object at a third location in the composition area, including displaying at least a portion of the second virtual object outside a lateral boundary of the composition area, such as via Fig.9DIn some embodiments, the threshold distance is based on the size of the second virtual object (e.g., is different for virtual objects of different sizes). In some embodiments, the threshold distance corresponds to, is based on, or is the same as the horizontal or lateral size of the second virtual object. In some embodiments, when displayed at the third position, the second virtual object is partially within the composition area and partially outside the composition area (e.g., outside the left or right border of the composition area).

[0198] In some embodiments, based on determining, when receiving the second input, that the attention of the user of the computer system is directed to a fourth location in the composition area (e.g., in a three-dimensional environment), where the fourth location is outside a threshold distance from a lateral boundary of the composition area, such as Fig. 9C The computer system displays (1020d) the second virtual object at a fourth position in the composition area, including displaying the second virtual object within a lateral boundary of the composition area without displaying a portion of the second virtual object outside the lateral boundary of the composition area, such as via Fig.9D 908-6 in the composition area. In some embodiments, when displayed at the fourth position, the second virtual object is completely within the composition area, rather than partially outside the composition area (e.g., outside the left or right border of the composition area). Therefore, in some embodiments, even if displaying the second virtual object at the attention position would cause the second virtual object to extend beyond the lateral border of the composition area, the computer system does not offset the second virtual object from the attention position. This is optionally in contrast to the upper or lower border of the composition area as previously described. Not offsetting the position of the second virtual object from the lateral border of the composition area provides feedback to the user that the composition area can be scrolled horizontally to scroll to other parts of the composition area, as described with reference to method 800.

[0199] In some embodiments, displaying the first virtual object at the first position in the composition area includes (1022a) determining that the first position includes a three-dimensional virtual object and determining that the first virtual object is a two-dimensional virtual object when the first input is received, displaying the first virtual object behind the three-dimensional virtual object at the first position in the composition area (1022b), such as via Fig.9D 908-6 and 908-7 in FIG. Therefore, in some embodiments, the three-dimensional object is always displayed in front of the two-dimensional object. Displaying different types of virtual objects in different display orders provides feedback to the user about the type of object being displayed.

[0200] In some embodiments, when the composition area is displayed via the display generation component, the computer system receives (1024a) via one or more input devices a second input corresponding to a request to insert the second virtual object into a third position in the composition area, such as Fig.9E In some embodiments, the second input has one or more of the characteristics of the first input, as previously described. In some embodiments, when the second input is detected, the user's attention is directed to the third location. In some embodiments, in response to receiving the second input (1024b), according to determining that the second virtual object is the first three-dimensional object and the third location in the composition area includes the second three-dimensional object when the second input is received, such as Fig.9E The computer system displays (1024c) a second virtual object at a fourth position in the composition area that is different from the third position, wherein when displayed at the fourth position, the second virtual object does not overlap with the second three-dimensional object, such as via Fig.9F 908-24 in the object. For example, the computer system optionally shifts the position of the second virtual object from the attention position to avoid or reduce the overlap and / or conflict between the second virtual object and the second three-dimensional object. In some embodiments, the fourth position is the point closest to the third position in the composition area, at which the overlap and / or conflict between the second virtual object and the second three-dimensional object is avoided or reduced. In some embodiments, the distance and / or relative position between the third position and the fourth position is based on the size of the second virtual object and the second three-dimensional object. Shifting the position of the second virtual object enables the computer system to automatically position the second virtual object at an appropriate position in the composition area.

[0201] In some embodiments, in response to receiving the second input (1026a), according to determining that the second virtual object is a two-dimensional object and the corresponding position in the composition area includes a second three-dimensional object when the second input is received, such as Fig. 9C The computer system displays (1026b) a second virtual object at a third location in the composition area, wherein when displayed at the third location, the second virtual object overlaps the second three-dimensional object, such as via Fig.9D 908-4 in the composition area. Therefore, the computer system optionally does not offset the position of the two-dimensional object pasted into the composition area from the user's attention position. As previously described, the two-dimensional object is optionally pasted behind the three-dimensional object. Not offsetting the position of the two-dimensional object avoids excessive and / or unnecessary movement of the two-dimensional object when the user does not provide the movement.

[0202] In some embodiments, when the composition area is displayed via the display generation component, the computer system receives (1028a) via one or more input devices a second input corresponding to a request to insert the second virtual object into a third position in the composition area, such as Fig.9E In some embodiments, the user's attention is directed to a third location in response to receiving the second input and in accordance with determining that the third location in the composition area includes the second three-dimensional object (1028b) when the second input is received, such as Fig.9E In the embodiment of the present invention, the computer system displays (1028c) the second virtual object at the fourth position in the composition area, such as via Fig.9F In some embodiments, the fourth position is the position closest to the third position, at which the display of the second virtual object will not cause a conflict and / or overlap between the second virtual object and another three-dimensional object in the composition area.

[0203] In some embodiments, based on determining that no position within a threshold distance of the third position does not include a three-dimensional object, such as, Fig.9E As in the case of gaze 920e in the composition area (e.g., there are no positions in the composition area within the threshold distance of the user's attention position that can be used to display the second virtual object, because each position within the threshold distance will result in a conflict / overlap between the second virtual object and another three-dimensional object in the composition area), the computer system displays (1028d) the second virtual object at a third position in the composition area, such as via Fig.9F In some embodiments, the second virtual object is displayed in front of the second three-dimensional object. Shifting the position of the second virtual object enables the computer system to automatically position the second virtual object at an appropriate location in the composition area while avoiding shifting the second virtual object by a relatively large amount.

[0204] In some embodiments, the first input is such as via Fig. 9B 922-3 in menu 924a in the composition area to copy the input (1030) of the second virtual object included in the composition area. In some embodiments, the input for copying the second virtual object is an input for selecting the "copy" selectable option displayed in the context menu for the second virtual object. The first virtual object (e.g., which is a copy of the second virtual object) is optionally placed in the composition area at a position to which the user's attention is directed when the "copy" selectable option is selected, such as previously described with reference to the first input. Therefore, in some embodiments, copying a virtual object and pasting a virtual object (e.g., different from copying a virtual object) optionally place the inserted virtual object into the composition area at a position based on the user's attention. Copying a virtual object to a position based on the user's attention enables the computer system to automatically determine the appropriate position of the copied virtual object.

[0205] In some embodiments, the first input is an input to paste a previously copied virtual object into the composition area (1032a), such as Fig. 9C In some embodiments, when the composition area including the second virtual object is displayed at a third location via the display generation component, the computer system receives (1032b) a second input corresponding to a request to copy the second virtual object, such as via a control panel, and / or a control panel. Fig. 9B ) selection of option 922-3 in menu 924a in (eg, selection of a “copy” option for a second virtual object as previously described).

[0206] In some embodiments, in response to receiving the second input, the computer system displays (1032c) a third virtual object (a copy of the second virtual object) at a fourth position in the composition area, where the fourth position is not based on the user's attention when the second input was detected, such as if the computer system 101 copied the third virtual object. Fig. 9BThe fourth position is a predetermined distance or orientation from the position of the virtual object being copied (e.g., 5 cm away from the virtual object being copied, diagonally downward and to the right). In some embodiments, the amount and / or direction of the fourth position separated from the virtual object to be copied is based on the size and / or shape of the virtual object to be copied. Therefore, in some embodiments, the computer system places objects in different ways for the paste operation and the copy operation. Copying the virtual object to a position not based on the user's attention causes the computer system to automatically place the copied virtual object near the virtual object being copied for easy access.

[0207] It should be understood that the specific order in which the operations in method 1000 are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0208] FIG. 11A to FIG. 11G An example of a computer system that displays a container virtual object having curvature in a three-dimensional environment is shown according to some embodiments.

[0209] Fig.11A The display generation components (eg, Figure 1 101 ) is visible from a user's viewpoint 1116 shown in the top view 1114 (e.g., facing the back wall of the physical environment in which the computer system 101 is located and near the back left corner of the physical environment). Figures 1 to 6 As described above, the computer system 101 optionally includes a display generation component (e.g., a touch screen) and a plurality of image sensors (e.g., Figure 3 The image sensor 314 may include one or more of: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the computer system 101 can use to capture one or more images of a user or a portion of a user (e.g., one or more hands of a user) as the user interacts with the computer system 101. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display that includes a display generation component that displays a user interface or a three-dimensional environment to the user, as well as sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors that face outward from the user) and / or sensors that detect the user's gaze (e.g., internal sensors that face inward toward the user's face).

[0210] like Fig.11A , computer system 101 captures one or more images of a physical environment 1102 (e.g., operating environment 100) surrounding computer system 101 (including one or more objects in the physical environment surrounding computer system 101). In some embodiments, computer system 101 displays a representation of the physical environment in a three-dimensional environment 1104 and / or physical environment 1102 is visible in three-dimensional environment 1104 via display generation component 120. For example, three-dimensional environment 1104 visible via display generation component 120 includes a representation of the physical floor and back and side walls of the room in which computer system 101 is located. Three-dimensional environment 1104 also includes table 1106 (shown via display generation component 120 and in top view 1114).

[0211] exist Fig.11A , the three-dimensional environment 1104 also includes virtual objects 1108-1 to 1108-7, virtual object 1110, and virtual object 1112. Virtual objects 1108-1 to 1108-7 are optionally one or more of user interfaces of applications (e.g., messaging user interfaces, content browsing user interfaces, etc.), three-dimensional objects (e.g., virtual clocks, virtual balls, virtual cars, etc.), or any other elements displayed by computer system 101 that are not included in the physical environment of computer system 101. Virtual object 1110 is optionally a container or canvas object that contains virtual objects 1108-1 to 1108-7, as described in more detail with reference to methods 800, 1000, and / or 1200. Virtual object 1110 is optionally scrollable horizontally (e.g., right / left), as will be described in more detail later. In addition, Fig.11A The virtual object 1110 in the scroll indicator 1126 includes a scroll indicator 1126 (e.g., a scroll bar), which includes a scroll position indicator 1128; the position of the scroll position indicator 1128 within the scroll indicator 1126 optionally corresponds to the current scroll position within the object 1110.

[0212] Virtual object 1112 is, optionally, a tool palette object for performing operations associated with virtual objects 1108 - 1 to 1108 - 7 and / or 1110 . For example, object 1112 includes a selectable option 1114-1 that can be selected to undo the last operation performed associated with virtual objects 1108-1 to 1108-7 and / or 1110; a selectable option 1114-2 that can be selected to redo the last operation canceled or undone associated with virtual objects 1108-1 to 1108-7 and / or 1110; a selectable option 1114-3 that can be selected to edit the currently selected object of virtual objects 1108-1 to 1108-7; a selectable option 1114-4 that can be selected to add a new virtual object to object 1110; and a selectable option 1114-5 that can be selected to cause the computer system 101 and / or object 1110 to transition to a selection mode, which is described in more detail with reference to method 800.

[0213] like Fig.11A As shown in , in some embodiments, object 1110 is translucent such that virtual and / or physical objects behind object 1110 are at least partially visible through object 1110 (e.g., the back wall and floor of the physical room). In addition, in some embodiments, two-dimensional objects contained within object 1110 have a curvature that is based on the curvature of object 1110 (e.g., has the same curvature as the curvature of object 1110). For example, when included in object 1110, object 1108-5 optionally has the curvature of object 1110 and / or a portion of object 1110 on which object 1108-5 is located. Fig.11A Objects 1108-1, 1108-2, 1108-3, and 1108-4 in are also two-dimensional objects, and therefore also have a curvature based on the object 1110. Scroll indicator 1126 optionally also has a curvature based on the curvature of object 1110.

[0214] On the other hand, three-dimensional objects optionally do have a shape, curvature, outline, or other dimensional properties that are based on the curvature of object 1110. For example, when three-dimensional objects are added or removed from object 1110, they optionally do not change in shape, curvature, outline, volume, or other dimensional properties. Fig.11A , objects 1108 - 6 and 1108 - 7 are three-dimensional objects, and the shapes, curvatures, contours, volumes, or other dimensional properties of these objects are not based on the curvature of object 1110 .

[0215] As previously mentioned, object 1112 is optionally a control palette for object 1110. Object 1112 is optionally displayed separately from object 1110; for example, closer to the user's viewpoint than object 1110 (e.g., visually separated from object 1110), and partially overlapping and / or obscuring a portion (e.g., a bottom portion) of object 1110 from the user's viewpoint. In some embodiments, computer system 101 displays object 1112 according to the above-described rules for displaying two-dimensional objects (e.g., object 1110 is displayed with a curvature based on the curvature of object 1112). In some embodiments, computer system 101 displays object 1112 according to the above-described rules for displaying three-dimensional objects (e.g., object 1110 is flat and / or is not displayed with a curvature based on the curvature of object 1112).

[0216] In some embodiments, objects cannot be placed at locations within object 1110 such that those objects would extend beyond the top or bottom border of object 1110 (e.g., as described in more detail with reference to method 1000). In some embodiments, objects can be placed at such locations, but those objects will be clipped off at the top or bottom border of object 1110 such that portions of those objects that would otherwise extend beyond the top or bottom border of object 1110 are not displayed by computer system 101. In contrast, in some embodiments, objects can be placed at locations within object 1110 such that those objects would extend beyond the left or right border of object 1110 (e.g., as described in more detail with reference to method 1000). In addition, computer system 101 will optionally display those portions of such objects that are extending beyond the left or right border of object 1110 as hanging or extending beyond those borders. For example, in Fig.11A , object 1108 - 7 has been placed in object 1110 by hand 1103 b (e.g., in the manner described with reference to method 800 and / or 1000 ), and a right portion of object 1108 - 7 is extending beyond the right boundary of object 1110 and is displayed by computer system 101 .

[0217] In some embodiments, the curvature of the object 1110 is optionally based on the number and / or positioning of viewpoints of users simultaneously sharing the three-dimensional environment 1104, whether computer system 101 alone, or computer system 101 and one or more other computer systems (e.g., having one or more of the characteristics of computer system 101). For example, computer system 101 is optionally included in a communication session with one or more other computer systems, as described in more detail in method 1200. Computer system 101 optionally displays object 1110 with a greater curvature as fewer viewpoints are present, the viewpoints are closer together, and / or the viewpoints are closer to object 1110, and displays object 1110 with a smaller curvature as more viewpoints are present, the viewpoints are farther from each other, and / or the viewpoints are farther from object 1110, as described in more detail with reference to method 1200. In this manner, the visibility of object 1110 and / or objects included in object 1110 is improved for one or more viewpoints.

[0218] For example, in Fig.11A In FIG. 1 , the three-dimensional environment 1104 is shared simultaneously by the viewpoint 1116 and the viewpoint 1120 at the position shown in the top view 1114 and has Fig.11A The curvature shown in Fig. 11B In FIG. 1 , in addition to viewpoints 1120 and 1116 at the position shown in top view 1114, object 1110 is now also simultaneously shared by viewpoint 1122. Because three-dimensional environment 1104 including object 1110 is now shared by viewpoint 1122, Fig.11A Because more viewpoints are shared simultaneously and / or because viewpoints 1120 and 1122 are relatively spaced apart from each other, computer system 101 can Fig.11A As previously described, the curvature of the object based on the curvature of the object 1110 also optionally decreases due to the decrease in the curvature of the object 1110. The position of the object 1110 in the three-dimensional environment 1104 (e.g., the position of a particular point of the object 1110, such as the center point) optionally does not change in response to the change in the curvature of the object 1110.

[0219] Return to Fig.11A, object 1110 is displayed in association with selectable option 1121. Option 1121 is optionally separate from object 1110, displayed outside of object 1110, and / or displayed adjacent to a corner of object 1110. Option 1121 can optionally be selected to initiate a process of resizing object 1110 without changing the curvature of object 1110. For example, an air pinch-and-drag input (e.g., thumb and index finger of hand 1103a brought together and touching, followed by movement of hand 1103a while remaining in the pinching hand shape) directed to option 1121 optionally initiates a process of resizing object 1110. Horizontal movement of hand 1103a optionally increases or decreases the lateral size of object 1110 without changing the vertical size of object 1110. For example, horizontal movement of hand 1103a toward the interior of object 1110 (e.g., to the left) optionally decreases the horizontal size of object 1110; optionally equally from both the left and right sides of object 1110, optionally only from the right side of object 1110. Horizontal movement of hand 1103a away from the interior of object 1110 (e.g., to the right) optionally similarly increases the horizontal size of object 1110. Such resizing optionally does not change the curvature of object 1110.

[0220] Vertical movement of hand 1103a optionally increases or decreases both the lateral and vertical dimensions of object 1110, optionally while maintaining the aspect ratio of object 1110. For example, upward movement of hand 1103a optionally decreases both the lateral and vertical dimensions of object 1110; optionally equally from the left and right sides and the top and bottom sides of object 1110, optionally only from the sides or corners of object 1110 adjacent to option 1121 (e.g., Fig.11A The downward movement of hand 1103a optionally similarly increases the lateral and vertical dimensions of object 1110. This resizing optionally does not change the curvature of object 1110.

[0221] For example, in Fig.11A , the computer system 101 optionally detects that the hand 1103a performs a horizontal and left air pinch and drag gesture pointing to the option 1121. In response, as Fig.11G As shown in , the computer system 101 reduces the horizontal size of the object 1110 according to the movement of the hand 1103a (eg, from both the left and right sides of the object 1110) without adjusting the vertical size of the object 1110. Fig.11G The curvature of the object 1110 in is optionally related to Fig.11A The curvature of the object 1110 in is the same. Fig.11GAs shown in , the resizing of object 1110 optionally does not result in resizing of objects included in object 1110. In addition, objects included in object 1110 that are no longer within the updated boundaries of object 1110 are optionally no longer displayed by computer system 101 (e.g., objects 1108-1, 1108-2, 1108-7, 1108-3, and 1108-5). Fig.11G , objects 1108-4 and 1108-6 are within the update boundary of object 1110 and therefore remain as they were in Fig.11A 1104 (eg, at the same location, in the same orientation, and / or at the same size in three-dimensional environment 1104).

[0222] Return to Fig. 11B In some embodiments, an air pinch-and-drag gesture directed to an empty area of ​​object 1110 (e.g., as previously described) causes computer system 101 to scroll through object 1110 in accordance with the movement of the user's hand, as described in more detail with reference to method 1200. For example, in Fig. 11B , the computer system 101 detects that the hand 1103a performs a left-to-air pinch-and-drag gesture while the user's gaze 1124 is directed to an empty area of ​​the object 1110. In response, Fig. 11C , computer system 101 scrolls left through object 1110, causing objects 1108-1 to 1108-7 to scroll left and off the currently visible portion of object 1110, and objects 1108-9 to 1108-12 to scroll from the right onto the currently visible portion of object 1110. Computer system 101 optionally updates scroll indicator 1126 to indicate the updated current scroll position accordingly.

[0223] In some embodiments, during scrolling of object 1110, computer system 101 changes the visual appearance of objects included in object 1110. For example, Fig. 11CAs shown in , the computer system 101 optionally displays the objects included in the object 1110 with increased translucency, reduced brightness, and / or reduced color saturation during scrolling (e.g., when the object is displayed as moving across the object 1110, moving onto the object, and / or moving away from the object). In some embodiments, the magnitude of the above-mentioned change in the visual appearance of the objects included in the object 1110 is based on the magnitude of the scroll (e.g., speed and / or amount). For example, the greater the magnitude of the scroll, the more the computer system 101 optionally changes the visual appearance of the object, and the smaller the magnitude of the scroll, the less the computer system changes the visual appearance of the object. In some embodiments, upon detecting the end of the scroll (e.g., the movement of the hand 1103a stops or the hand 1103a releases the air pinch gesture, such as the thumb and index finger of the hand 1103a are separated from each other), the computer system 101 optionally restores the visual appearance of the objects included in the object 1110 to the visual appearance they had before the scroll input.

[0224] In some embodiments, such as Fig. 11C As shown in , in response to detecting a user's gaze 1124 directed toward a scroll indicator 1126, the computer system 101 updates the display of the scroll indicator 1126 to include a snapshot or preview or map of the content (e.g., the object) included in the object 1110 (e.g., a snapshot or preview of the entire canvas of the object 1110), including a portion of the object 1110 corresponding to a scroll position other than the current scroll position in the object 1110 (e.g., a portion of the canvas of the object 1110 that is not currently displayed). In addition, the scroll indicator 1126 includes a current scroll position indicator 1130 that indicates which portion of the snapshot or preview or map is currently being displayed within the object 1110. The current scroll position indicator 1130 is, optionally, any visual indication indicating a particular portion of the snapshot or preview or map, such as an outline, glow, and / or highlighting of the particular portion.

[0225] The object 1110 may optionally also scroll in response to an air pinch and drag input directed to the current scroll position indicator 1130, which moves the current scroll position indicator 1130 within the scroll indicator 1126 in accordance with the movement of the user's hand, and scrolls the object 1110 to a scroll position corresponding to the current position of the current scroll position indicator 1130 within the scroll indicator 1126. The object 1110 may also optionally scroll in response to an air pinch and release input directed to a specific portion of the scroll indicator 1126 (e.g., without movement of the user's hand), which causes the computer system 101 to scroll the object 1110 to a scroll position corresponding to the specific portion of the scroll indicator 1126. Additional details of the above scrolling operations are described with reference to the method 1200.

[0226] from FIG. 11C to FIG. 11D , the computer system 101 detects further leftward movement of the hand 1103a while in the pinching hand shape, and therefore, as Fig.11D The object 1110 is further scrolled to the left as shown in FIG. Fig.11D As shown in , the computer system 101 also updates the position of the current scroll position indicator 1130 within the scroll indicator 1126 accordingly.

[0227] In some embodiments, the computer system 101 moves the object 1110 closer to or further away from the viewpoint 1116 in response to a particular input. For example, an air pinch-and-drag input directed to an empty area of ​​the object 1110 optionally causes the object 1110 to move toward or away from the viewpoint 1116 in the three-dimensional environment 1104 in accordance with the dragging input. Fig.11D In the example of FIG. 1 , the computer system 101 detects that the hand 1103a performs an air pinch input pointing to an empty area of ​​the object 1110, followed by the hand 1103a moving toward the user in a pinching hand shape. In response, as shown in FIG. Fig.11E As shown in , computer system 101 moves object 1110 toward viewpoint 1116 by an amount corresponding to the magnitude of the movement of hand 1103a toward the user. Computer system 101 optionally does not respond to Fig.11D 110 in the three-dimensional environment 1104. However, for the reasons previously described, because the object 1110 has now moved closer to the viewpoints 1116, 1120, and / or 1122, the computer system 101 Fig.11E In Fig.11D The object 1110 is displayed with a greater curvature in FIG. Fig.11D As a result of the input detected in , the curvature of objects 1108-14 and 1108-15 has increased, the shape / contour / curvature of object 1108-13 has not changed, and the orientation of objects 1108-14 and / or 1108-15 relative to viewpoint 1116 has changed (e.g., has become more parallel to the normal of the viewpoint).

[0228] In some embodiments, computer system 101 snaps object 1110 to another object or surface (e.g., whether physical or virtual) in response to object 1110 being moved within a threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm) of the object or surface. When snapping object 1110 to another object or surface, computer system 101 optionally displays object 1110 as having no curvature (e.g., flat) or having curvature (or flat) based on the curvature (or lack thereof) of the object or surface. For example, in Fig.11EIn the example of FIG. 1 , the computer system 101 detects that the hand 1103a performs an air pinch input pointing to an empty area of ​​the object 1110, followed by the hand 1103a moving away from the user in a pinching hand shape. In response, as shown in FIG. Fig.11F As shown in , computer system 101 moves object 1110 away from viewpoint 1116 by an amount corresponding to the magnitude of the movement of hand 1103a away from the user. Computer system 101 optionally does not respond to Fig.11E The hand 1103a is moved from the FIG. 11E to FIG. 11F The movement of away from the user is sufficient to move object 1110 to within a threshold distance of the back wall of the physical room in which computer system 101 is located. Fig.11F As shown in , computer system 101 has aligned object 1110 to the back wall of the physical room and is displaying object 1110 with no curvature and / or with a curvature based on the curvature of the wall. Fig.11E As a result of the input detected in , the curvature of objects 1108-14 and 1108-15 has decreased (e.g., decreased to zero), the shape / contour / curvature of object 1108-13 has not changed, and the orientation of objects 1108-14 and / or 1108-15 relative to viewpoint 1116 has changed (e.g., has become more perpendicular relative to the normal of the viewpoint). Fig.11F The curvature of the object 1110 in is optionally less than Fig.11D and Fig.11E The curvature of object 1110 in FIG.

[0229] FIG. 12A to FIG. 12L 1 is a flowchart illustrating a method 1200 for displaying a container virtual object having curvature in a three-dimensional environment according to some embodiments. In some embodiments, the method 1200 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet device, a smart phone, a wearable computer, or a head mounted device, the computer system including a display generating component (e.g., Figure 1 , Figure 3 and Figure 4 In some embodiments, method 1200 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1Some operations in method 1200 are optionally combined, and / or the order of some operations is optionally changed.

[0230] In some embodiments, method 1200 is performed at a computer system (e.g., 101) in communication with a display generation component (e.g., 120) and one or more input devices. In some embodiments, the computer system is the same or similar to the computer system described above with reference to method 800 and / or 1000. In some embodiments, the one or more input devices are the same or similar to the one or more input devices described above with reference to method 800 and / or 1000. In some embodiments, the display generation component is the same or similar to the display generation component described above with reference to method 800 and / or 1000.

[0231] In some embodiments, the computer system displays (1202a) via a display generation component a first virtual object, such as object 1110, which is a virtual object at a first location in a three-dimensional environment, the virtual object including a second object, such as other objects (e.g., different virtual objects or physical objects) in the three-dimensional environment 1104. In some embodiments, the three-dimensional environment has one or more of the characteristics of the three-dimensional environment of method 800 and / or 1000. In some embodiments, the first object includes or comprises one or more objects, such as virtual objects or user interface objects described with reference to method 800 and / or 1000. In some embodiments, the first virtual object is a virtual object having dynamic curvature as described below.

[0232] In some embodiments, while displaying the first virtual object in the three-dimensional environment, the computer system receives (1202b) via one or more input devices a first input corresponding to a request to move the first virtual object from a first position in the three-dimensional environment to a second position different from the first position, such as from Fig.11D1103a in the hand 1103a (in some embodiments, the first input and / or other inputs described with reference to method 800 are or include air gesture inputs). In some embodiments, the computer system detects a request to move the first virtual object from the first position to the second position in response to the computer system detecting that the user's hand performs a gesture to select the first virtual object and detects that the user's hand moves to a position in the physical environment corresponding to the second position, while the first virtual object is still being selected by the user's hand. For example, the computer system receives a request to move the first virtual object from the first position to the second position in response to the computer system detecting that the user's hand selects the first virtual object by performing a pinch gesture as described above (optionally, while the user's attention is directed to the first virtual object) and detects that the user's hand moves to a position in the physical environment corresponding to the second position, while maintaining the pinch gesture. In some embodiments, the computer system receives a request to move the first virtual object from the first position to the second position in response to the computer system detecting that the user's hand is moving the first virtual object via a drag / move gesture on the touch screen of the computer system (e.g., if the display generating component is a touch screen).

[0233] In some embodiments, in response to receiving the first input (1202c), the computer system displays (1202d) the first virtual object at the second location in the three-dimensional environment (e.g., and stops displaying the first virtual object at the first location), including determining that the second location in the three-dimensional environment corresponds to a second object in the three-dimensional environment (e.g., the first virtual object is within a threshold distance from the second object, is aligned to the second object, and / or has a predefined pose relative to the second object), such as moving the object 1110 to the second location. Fig.11F The first virtual object is configured to be displayed in a manner that is consistent with the user's physical environment 1104 and that is close to the user's physical environment 1104 (e.g., the first virtual object is within a threshold distance from a first portion (e.g., a surface) of a second object, such as anchored / aligned to a surface of a physical object in the user's physical environment (e.g., a surface of a wall or table) and / or a surface of a virtual object in the three-dimensional environment), and displaying the first virtual object (1202e) with a first amount of curvature, such as Fig.11FFor example, if the computer system is displaying a first virtual object that is “aligned” or “anchored” to a first portion of a corresponding object, the first virtual object is being displayed with a curvature that corresponds to the curvature of the first portion of the corresponding object (e.g., if the first portion of the corresponding object has no curvature (e.g., is flat), the first virtual object has no curvature (e.g., is flat), and / or if the first portion of the corresponding object has a curvature of 1 degree, 2 degrees, 3 degrees, 4 degrees, or 10 degrees, the first virtual object has a curvature of 1 degree, 2 degrees, 3 degrees, 4 degrees, or 10 degrees). In some embodiments, the corresponding object includes the above-mentioned first portion (e.g., top surface), second portion (e.g., bottom surface), and third portion (e.g., side surface), and in response to the computer system detecting a request to move the first virtual object within a threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 20 cm, 30 cm, or 50 cm) of the first portion, second portion, or third portion of the corresponding object, the first virtual object becomes anchored to any of the first portion, second portion, or third portion. When the first virtual object is anchored to the second portion of the corresponding object, if the corresponding first portion and second portion correspond to different amounts of curvature, the first virtual object is optionally displayed with a different amount of curvature than when the first virtual object is anchored to the first portion of the corresponding object. Similarly, when the first virtual object is anchored to the third portion of the corresponding object, if the third portion of the corresponding object has a curvature that is different from the curvature of the first portion and / or the second portion of the corresponding object, the first virtual object is optionally displayed with a different amount of curvature than when the first virtual object is anchored to the first portion and / or the second portion of the corresponding object. In some embodiments, if the first virtual object is "aligned" or "anchored" to the first portion of the corresponding object, the orientation of the first virtual object corresponds to the orientation of the first portion of the corresponding object.

[0234] In some embodiments, based on determining that a second position in the three-dimensional environment does not correspond to a second object in the three-dimensional environment (e.g., the first object is not within a threshold distance from any object in the three-dimensional environment and / or the physical environment, is not aligned to any object in the three-dimensional environment and / or the physical environment, and / or does not have a predefined pose relative to any object in the three-dimensional environment and / or the physical environment - optionally, the second position does not correspond to any object in the three-dimensional environment and / or the physical environment), such as object 1110 is located in the three-dimensional environment. Fig.11E The first virtual object is displayed via the display generation component with a second curvature amount different from the first curvature amount (1202f), such as in Fig.11E110 in the three-dimensional environment. For example, if the movement of the first virtual object to the second position in the three-dimensional environment does not cause the first virtual object to be "aligned" or "anchored" to the corresponding object or any other object in the three-dimensional or physical environment of the computer system (e.g., not moved within a threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 20 cm, 30 cm, or 50 cm) of the second object), the computer system displays the first virtual object with a different amount of curvature than if the first virtual object had been anchored / aligned to the corresponding object in the three-dimensional or physical environment of the computer system. In some embodiments, when the first virtual object is being displayed with the second amount of curvature, the first virtual object has a non-zero amount of curvature (e.g., is not flat). In some embodiments, when the movement of the first virtual object to the second position in the three-dimensional environment does not cause the first virtual object to be anchored / aligned to a surface in the three-dimensional or physical environment of the computer system, the first virtual object has an amount of curvature that is optionally based on the number of users in the three-dimensional environment and / or the positions of the users in the three-dimensional environment, as described in more detail below. In some embodiments, when the first virtual object is not anchored / aligned to a particular object in the three-dimensional or physical environment of the computer system, the computer system displays the first virtual object with a higher degree of curvature than when the first virtual object is anchored / aligned to a particular object in the three-dimensional or physical environment of the computer system. Displaying the first virtual object with different amounts of curvature based on whether the first virtual object is aligned to an object in the three-dimensional or physical environment of the computer system enables the computer system to automatically display the first virtual object with an appropriate amount of curvature.

[0235] In some embodiments, the first location corresponds to a third object in the three-dimensional environment (e.g., in the same or similar manner as the second location corresponds to the second object), such as if Fig.11D In the embodiment of the present invention, the object 1110 has been aligned to another wall of the three-dimensional environment 1104, and a first virtual object is displayed (1204) at a first location with a first amount of curvature when a first input is detected, such as if in Fig.11D In one embodiment, the object 1110 is displayed without curvature (e.g., the first virtual object is displayed at the second position in the same or similar manner when the second position corresponds to the second object). For example, when the first input is detected, the first virtual object is aligned to the surface and / or object. Displaying the first virtual object at the first position with the first amount of curvature provides feedback that the first virtual object is aligned to another object in the three-dimensional environment.

[0236] In some embodiments, when the first input is received and when the first virtual object is displayed at a third location other than the first location and the second location (e.g., the third location does not correspond to any object in the three-dimensional environment), the first virtual object is displayed with a second amount of curvature (1206), such as if the object 1110 is at a location where it is being viewed. Fig.11D and Fig.11E between and / or Fig.11E and Fig.11F The computer system optionally updates the curvature of the first virtual object to the second curvature when the first virtual object is moved away from the first position. Displaying the first virtual object at the third position with the second amount of curvature provides feedback that the first virtual object is not aligned with another object in the three-dimensional environment when it is moved.

[0237] In some embodiments, the first location does not correspond to an object in the three-dimensional environment (e.g., the first location is empty space in the three-dimensional environment), such as in Fig.11D or Fig.11E and displaying a first virtual object (1208) at a first location with a second amount of curvature when a first input is detected, such as in Fig.11D or Fig.11E In one embodiment, the first virtual object is aligned with the second object in the three-dimensional environment. For example, the first virtual object optionally maintains the second curvature until the second position is reached, and if the second position corresponds to the second object, the computer system aligns the first virtual object to the second object and updates the curvature of the first virtual object to the second curvature. Displaying the first virtual object with the second curvature when the first input is detected provides feedback that the first virtual object is not aligned with another object in the three-dimensional environment when the first input is received.

[0238] In some embodiments, the first location is a first distance from a viewpoint of a user from which the three-dimensional environment is visible (e.g., a viewpoint from which a computer system displays the three-dimensional environment and / or makes visible the three-dimensional environment, including the first virtual object and the second object), and the second location is a first distance from the viewpoint of the user (1210), such as if object 1110 is a distance from the user. Fig.11F1104 to the back wall of the room in the three-dimensional environment 1104. Thus, in some embodiments, both the first position and the second position are at the same distance from the user's viewpoint. In some embodiments, the curvature of the first virtual object (other factors described below being equal) is based on the distance of the first virtual object from the user's viewpoint, and is the same curvature for the same distance, even if the relative position of the first virtual object relative to the user's viewpoint is different (e.g., the relative left side of the viewpoint compared to the relative right side of the viewpoint). Thus, for a given distance from the user's viewpoint, the difference in curvature of the first virtual object optionally indicates other relevant factors related to the first virtual object (e.g., whether the first virtual object is aligned to another object, the number of users viewing the first virtual object, and / or the number of users to whom the first virtual object is accessible, and / or the number of users with whom the three-dimensional environment / first virtual object is shared simultaneously, etc., as described below). Displaying the first virtual object with different curvatures for a given distance from the user's viewpoint based on whether the position of the first virtual object corresponds to another object provides feedback on whether the first virtual object is aligned to another object in the three-dimensional environment.

[0239] In some embodiments, when the first virtual object is at the corresponding position in the three-dimensional environment, displaying the first virtual object at the corresponding position includes (1212a) displaying the first virtual object at the corresponding position based on determining that the first virtual object is in a three-dimensional environment that is simultaneously shared by a first number of users including the user of the computer system (such as a shared Fig.11A In some embodiments, the first virtual object is displayed or is displayable by the computer system and one or more other computer systems as part of the communication session. In some embodiments, the first virtual object is interactively displayed by the user of the computer system and one or more other users associated with the other computer systems as part of the communication session), the first virtual object is displayed (1212b) with a third amount of curvature, such as Fig.11A The curvature of the object 1110 in FIG. 110 is determined by the user's computer readable medium. ...

[0240] In some embodiments, based on determining that the first virtual object is in a three-dimensional environment that is simultaneously shared by a second number of users (such as a shared Fig. 11B ), displaying the first virtual object (1212c) with a fourth amount of curvature different from the third amount of curvature, such as Fig. 11BIn some embodiments, the curvature of the object 1110 in the image is displayed. In some embodiments, when the other factors described herein are the same, the more users the first virtual object and / or the three-dimensional environment are shared with simultaneously, the smaller the curvature of the first virtual object displayed by the computer system, thereby facilitating more users to view the first virtual object and / or interact with the first virtual object. Similarly, when the other factors described herein are the same, the fewer users the first virtual object and / or the three-dimensional environment are shared with simultaneously, the larger the curvature of the first virtual object displayed by the computer system, thereby concentrating the viewing of the first virtual object and / or the interaction with the first virtual object to a smaller number of users. Displaying the first virtual object with different curvatures based on the number of users with which the three-dimensional environment / first virtual object is shared enables the computer system to automatically facilitate the relevant users to view the first virtual object or interact with the first virtual object.

[0241] In some embodiments, when the first virtual object is at the corresponding position in the three-dimensional environment, displaying the first virtual object at the corresponding position includes (1214a) determining that a viewpoint of a first user (e.g., a user of the computer system, or a user of another computer system in the communication session) who is simultaneously sharing the three-dimensional environment (e.g., according to a communication session as previously described) is the first viewpoint, such as Fig. 11B , the first virtual object (1214b) is displayed with a third amount of curvature, such as from a user's viewpoint shown in FIG. 1214 (e.g., at a particular location in the three-dimensional environment, such as a particular location relative to a location of the first virtual object, including one or more of a relative orientation or a relative distance), such as Fig. 11B In some embodiments, based on determining that the viewpoint of the first user who is concurrently sharing the three-dimensional environment is a second viewpoint that is different from the first viewpoint (e.g., at a different specific location or relative location in the three-dimensional environment), the curvature of the object 1110 in the three-dimensional environment is the same as the first curvature amount, the same as the second curvature amount, or different from the first curvature amount and the second curvature amount. Fig. 11B , the first virtual object (1214c) is displayed with a fourth curvature amount different from the third curvature amount, such as the object 1110 having a fourth curvature amount different from the third curvature amount. Fig. 11BIn some embodiments, the positions of users viewing and / or interacting with the first virtual object contribute to different curvatures of the first virtual object, so as to facilitate those users to view and / or interact with the first virtual object, provided that other factors described herein are the same. For example, if the user is usually centered on the first virtual object, the computer system optionally displays the first virtual object with a larger curvature, and if the user is usually away from the center of the first virtual object, the computer system optionally displays the first virtual object with a smaller curvature. In some embodiments, the more the user's position deviates from the center, the smaller the curvature of the first virtual object is displayed. In some embodiments, when placing and / or moving the first virtual object in a three-dimensional environment, the curvature of the first virtual object is set based on the average position of the users who are simultaneously sharing the first virtual object when placing and / or moving the first virtual object. In some embodiments, the curvature of the first virtual object does not change after such curvature is set for the first virtual object, even if those users subsequently move relative to the first virtual object, unless and / or until the first virtual object is placed or moved again. In some embodiments, when (optionally, whenever) a user joins the sharing of the first virtual object or the user stops participating in the sharing of the first virtual object, the curvature of the first virtual object is updated based on the (e.g., average) position of the users with which the first virtual object is shared. Displaying the first virtual object with different curvatures based on the three-dimensional environment / position of the users with which the first virtual object is shared enables the computer system to automatically facilitate the relevant users to view or interact with the first virtual object.

[0242] In some embodiments, when the first virtual object is at the corresponding position in the three-dimensional environment, displaying the first virtual object at the corresponding position includes (1216a) determining that a viewpoint of a first user (e.g., a user of the computer system, or a user of another computer system in the communication session) who is simultaneously sharing the three-dimensional environment (e.g., according to a communication session as previously described) is a first distance (such as a distance from the first virtual object). Fig.11D The first virtual object (1216b) is displayed with a third amount of curvature, such as Fig.11D In some embodiments, the curvature of the object 1110 in the three-dimensional environment is determined to be the same as the first curvature amount, the same as the second curvature amount, or different from the first curvature amount and the second curvature amount. Fig.11E The first virtual object (1216c) is displayed with a fourth amount of curvature different from the third amount of curvature, such as Fig.11EIn some embodiments, the distance of users viewing and / or interacting with the first virtual object contributes to different curvatures of the first virtual object, thereby facilitating those users to view and / or interact with the first virtual object, with the other factors described herein being the same. For example, the closer the user is to the first virtual object, the computer system optionally displays the first virtual object with a greater curvature, and the farther the user is from the first virtual object, the computer system optionally displays the first virtual object with a smaller curvature. In some embodiments, when placing and / or moving the first virtual object in a three-dimensional environment, the curvature of the first virtual object is set based on the average distance of users who are simultaneously sharing the first virtual object when the first virtual object is placed and / or moved. In some embodiments, the curvature of the first virtual object does not change after such curvature is set for the first virtual object, even if those users subsequently move relative to the first virtual object, unless and / or until the first virtual object is placed or moved again. In some embodiments, when (optionally, whenever) a user joins the sharing of the first virtual object or the user stops participating in the sharing of the first virtual object, the curvature of the first virtual object is updated based on the (e.g., average) distance of the first virtual object with the users it is shared with. Displaying the first virtual object with different curvatures based on the distance of the first virtual object with the users it is shared with enables the computer system to automatically facilitate the relevant users to view or interact with the first virtual object.

[0243] In some embodiments, the second object is a physical object (e.g., a wall, table, door, window, dresser, or any other physical object) (1218) in the physical environment of the computer system, such as the back wall of the three-dimensional environment 1104. Thus, in some embodiments, the first virtual object may snap to (the surface of) the physical object in the three-dimensional environment, which causes the curvature of the first virtual object to change (e.g., to match or correspond to the curvature of (the surface of) the physical object. As previously described, in some embodiments, the physical object is visible via a display generation component in the three-dimensional environment (e.g., via a semi-transparent or transparent display generation component) via passive see-through. In some embodiments, the physical object is visible and / or displayed via a display generation component in the three-dimensional environment via active see-through (e.g., the computer system captures one or more images of the physical object and displays a real or other representation of the physical object in the three-dimensional environment). Allowing the first virtual object to snap to the physical object provides the user with consistent interaction with objects in the three-dimensional environment, regardless of whether the object is a physical object or a virtual object.

[0244] In some embodiments, the first virtual object includes a second virtual object (e.g., the second virtual object is contained within the first virtual object and / or has been aligned to the first virtual object in the same or similar manner as described with reference to aligning the first virtual object to the second object) that is a two-dimensional object (e.g., an object that has no thickness or has a thickness less than a threshold amount (such as 0.01 cm, 0.05 cm, 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm) in a three-dimensional environment to distinguish it from a three-dimensional object (such as a three-dimensional model of a car or a building, as referenced below), and when the first virtual object has a first curvature, the second virtual object has a third curvature (1220) corresponding to the first curvature, such as Fig.11D In some embodiments, the curvature of the two-dimensional object included in the first virtual object is based on the curvature of the first virtual object. In some embodiments, if the computer system detects an input to remove the second virtual object from the first virtual object, the computer system will remove the second virtual object from the first virtual object, and the curvature of the second virtual object will be different from the third curvature (for example, will have no curvature, or will have a curvature inherent to the second virtual object that is not based on the curvature of the first virtual object). Basing the curvature of the two-dimensional object on the curvature of the first virtual object provides feedback that the two-dimensional object is included in the first virtual object.

[0245] In some embodiments, when the first virtual object has a second curvature, the second virtual object has a fourth curvature different from the third curvature, wherein the fourth curvature corresponds to the second curvature (1222), such as Fig.11E In some embodiments, the curvature of the two-dimensional object included in the first virtual object changes as the curvature of the first virtual object changes. For example, in response to an input that increases or decreases the curvature of the first virtual object, the curvature of the first virtual object and the curvature of the two-dimensional object included in the first virtual object increase or decrease, respectively, according to the input. In some embodiments, the curvature of the two-dimensional object is the same as or corresponds to the curvature of the first virtual object. Changing the curvature of the two-dimensional object as the curvature of the first virtual object changes provides feedback that the two-dimensional object is included in the first virtual object.

[0246] In some embodiments, the first virtual object includes a second virtual object (e.g., the second virtual object is contained within the first virtual object and / or has been aligned to the first virtual object in the same or similar manner as described with reference to the alignment of the first virtual object to the second object) as a three-dimensional object (e.g., an object having a thickness greater than a threshold amount (such as 0.01 cm, 0.05 cm, 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm) in a three-dimensional environment to be distinguished from a two-dimensional object (such as the two-dimensional object described above)), and the second virtual object (1224) is not displayed with a curvature based on the curvature of the first virtual object, such as the shape, outline, etc. of objects 1108-13 is not based on the curvature of object 1110 and / or is not Fig.11D and Fig.11E In some embodiments, the shape or curvature of the three-dimensional object included in the first virtual object is not based on the curvature of the first virtual object. For example, the shape of the three-dimensional object (e.g., a model of a car, a model of a building, etc.) is optionally inherent to those objects and does not change based on the curvature of the first virtual object (e.g., does not change with the change of the curvature of the first virtual object). In some embodiments, the three-dimensional object included in the first virtual object is visually separated from the first virtual object by a threshold amount (e.g., 0.1cm, 0.3cm, 0.5cm, 1cm, 3cm, 5cm, 10cm, or 20cm). In some embodiments, if the computer system detects an input to remove the second virtual object from the first virtual object, the computer system will remove the second virtual object from the first virtual object, and the shape and / or curvature of the second virtual object will remain the same. Maintaining the three-dimensional object shape as a curvature independent of the first virtual object provides feedback that the object is a three-dimensional object.

[0247] In some embodiments, the computer system simultaneously displays (1226) a user interface element (e.g., a tool palette) with the first virtual object, the user interface element including one or more selectable options that can be selected to perform an operation associated with the first virtual object, such as element 1112 (e.g., one or more selectable options that can be selected to perform an operation such as a cut operation, a copy operation, a paste operation, a resize operation, or a selection operation, such as described with reference to methods 800 and / or 1000), wherein the user interface element is not displayed with a curvature that is based on the curvature of the first virtual object, such as via FIG. 11A to FIG. 11G110 is a diagram showing an element 1112 in FIG. 1113 . For example, similar to the three-dimensional object described above, the shape or curvature of the tool palette is not based on the curvature of the first virtual object. In some embodiments, the tool palette is included in the first virtual object (e.g., similar to the two-dimensional and three-dimensional objects previously described). In some embodiments, the tool palette is displayed separately from the first virtual object and at a location separate from the tool palette in the three-dimensional environment. Maintaining the tool palette shape independent of the curvature of the first virtual object promotes consistent interaction with the tool palette regardless of the curvature of the first virtual object.

[0248] In some embodiments, the computer system displays (1228) a user interface element (e.g., the tool palette described above) simultaneously with the first virtual object, the user interface element including one or more selectable options that can be selected to perform an operation associated with the first virtual object, such as element 1112 (e.g., the operation described above associated with the tool palette), wherein the user interface element is displayed with a curvature based on the curvature of the first virtual object, such as if element 1112 is displayed with a curvature based on the curvature of object 1110. For example, similar to the two-dimensional objects described above, the shape or curvature of the tool palette changes based on the curvature of the first virtual object and / or based on a change in the curvature of the first virtual object. In some embodiments, the tool palette is included in the first virtual object (e.g., similar to the two-dimensional and three-dimensional objects described previously). In some embodiments, the tool palette is displayed separately from the first virtual object, at a location separate from the tool palette in the three-dimensional environment. Displaying the tool palette with a curvature based on the curvature of the first virtual object indicates that the tool palette and / or the selectable options included in the tool palette are associated with the first virtual object.

[0249] In some embodiments, the user interface element is not contained within the first virtual object (1230), such as via FIG. 11A to FIG. 11G 1112 in . For example, the tool palette is optionally displayed at a location different from the location of the first virtual object in the three-dimensional environment. In some embodiments, the tool palette can be moved separately from the first virtual object in the three-dimensional environment. In some embodiments, the tool palette is displayed at a different spacing and / or relative position to the first virtual object compared to the two-dimensional or three-dimensional virtual objects included in the first virtual object. Displaying the tool palette separately from the first virtual object facilitates personalized placement or interaction with the tool palette.

[0250] In some embodiments, the first virtual object is associated with a selectable option (such as Fig.11A, or a plurality of other options are displayed in association with the option 1121 shown in the computer system (e.g., a selectable option displayed in the lower right corner of the first virtual object, within the boundaries of the first virtual object, or outside the boundaries of the first virtual object), which can be selected to initiate a process of changing the size of the first virtual object. In some embodiments, when a first virtual object and a selectable option are displayed, wherein the first virtual object has a first size (e.g., a first area, a first width, and / or a first height in the case of a first virtual object that is rectangular in shape (whether or not curved), the computer system detects (1232b) a second input via one or more input devices, the second input comprising a corresponding gesture performed by a corresponding part of a user of the computer system (e.g., a hand), while the user's attention (e.g., gaze) is directed to the selectable option, followed by movement of the corresponding part of the user while in a corresponding posture, such as from Fig.11A For example, the second input is the pinch and drag input (eg, air gesture) described with reference to method 800 .

[0251] In some embodiments, in response to detecting the second input, the computer system displays (1232c) the first virtual object having a second size different from the first size based on the second input, such as Fig.11G. For example, movement of the selectable option toward the horizontal and / or vertical center of the first virtual object optionally causes the first virtual object to decrease in size based on the amount of this movement (e.g., the more the movement, the greater the decrease in size), and movement of the selectable option away from the horizontal and / or vertical center of the first virtual object optionally causes the first virtual object to increase in size based on the amount of this movement (e.g., the more the movement, the greater the increase in size). In some embodiments, the movement of the selectable option is associated with the movement of the corresponding part of the user, such as described with reference to moving objects based on the movement of the user's hand in method 800. In some embodiments, resizing the first virtual object as described does not cause the virtual objects included in the first virtual object to change in size (e.g., they remain at their current size). When the end of the pinch and drag input is detected (e.g., when the fingers making the pinch hand shape are detected to move away from each other), the size of the first virtual object is no longer controlled by the movement of the user's hand. In some embodiments, the size of the first virtual object referenced here is the size of the first virtual object in the three-dimensional environment (e.g., the size of the first virtual object relative to the three-dimensional environment). The size of the first virtual object displayed via the display generation component (e.g., the amount of the user's field of view consumed by the first virtual object) is optionally based on the size of the first virtual object relative to the three-dimensional environment, and changes based on a change in the relative position of the first virtual object relative to the user's viewpoint (e.g., as the viewpoint moves an...

Claims

1. A method, comprising: At a computer system in communication with a display generating component and one or more input devices: displaying, via the display generation component, a first virtual object at a first position in a three-dimensional environment; while displaying the first virtual object in the three-dimensional environment, receiving, via the one or more input devices, a first input corresponding to a request to move the first virtual object from the first position in the three-dimensional environment to a second position different from the first position; as well as In response to receiving the first input: Displaying the first virtual object at the second position in the three-dimensional environment includes: displaying the first virtual object with a first amount of curvature via the display generation component based on determining that the second position in the three-dimensional environment is a first distance from a current viewpoint of a user of the computer system; as well as Based on determining that the second position in the three-dimensional environment is a second distance different from the first distance from a current viewpoint of a user of the computer system, the first virtual object is displayed via the display generation component with a second amount of curvature different from the first amount of curvature.

2. The method according to claim 1, displaying the first virtual object at the second position in the three-dimensional environment further comprises: displaying, via the display generation component, the first virtual object with the third amount of curvature based on determining that the second position in the three-dimensional environment corresponds to the second object in the three-dimensional environment; as well as Based on determining that the second position in the three-dimensional environment does not correspond to the second object in the three-dimensional environment, the first virtual object is displayed via the display generation component with a fourth amount of curvature different from the third amount of curvature. 3 . The method of claim 2 , wherein the first position corresponds to a third object in the three-dimensional environment, and when the first input is detected, the first virtual object is displayed at the first position with the third amount of curvature. 4 . The method of claim 3 , wherein when the first input is received and when the first virtual object is displayed at a third position other than the first position and the second position, the first virtual object is displayed with the fourth amount of curvature. 5 . The method of claim 2 , wherein the first location does not correspond to an object in the three-dimensional environment, and when the first input is detected, the first virtual object is displayed at the first location with the fourth amount of curvature. The method of claim 2 , wherein the second object is a physical object in a physical environment of the computer system.

7. The method according to claim 1, wherein when the first virtual object is at a corresponding position in the three-dimensional environment, displaying the first virtual object at the corresponding position comprises: displaying the first virtual object with a third amount of curvature based on determining that the first virtual object is in a three-dimensional environment shared simultaneously by a first number of users including a user of the computer system; as well as Based on determining that the first virtual object is in a three-dimensional environment simultaneously shared by a second number of users including the user of the computer system that is different from the first number of users, the first virtual object is displayed with a fourth amount of curvature different from the third amount of curvature.

8. The method according to claim 1, wherein when the first virtual object is at a corresponding position in the three-dimensional environment, displaying the first virtual object at the corresponding position comprises: displaying the first virtual object with a third amount of curvature in response to determining that the viewpoint of a first user who is concurrently sharing the three-dimensional environment is a first viewpoint; as well as Based on determining that the viewpoint of the first user who is concurrently sharing the three-dimensional environment is a second viewpoint different from the first viewpoint, the first virtual object is displayed with a fourth amount of curvature different from the third amount of curvature. 9 . The method of claim 1 , wherein the first virtual object includes a second virtual object that is a two-dimensional object, and when the first virtual object has a first amount of curvature, the second virtual object has a third amount of curvature corresponding to the first amount of curvature. 10 . The method of claim 9 , wherein when the first virtual object has a second amount of curvature, the second virtual object has a fourth amount of curvature different from the third amount of curvature, wherein the fourth amount of curvature corresponds to the second amount of curvature. 11 . The method of claim 1 , wherein the first virtual object includes a second virtual object that is a three-dimensional object, and the second virtual object is not displayed with a curvature based on a curvature of the first virtual object.

12. The method according to claim 1, further comprising: Displaying a user interface element simultaneously with the first virtual object, the user interface element comprising one or more selectable options that can be selected to perform an operation associated with the first virtual object, wherein the user interface element is not displayed with a curvature that is based on the curvature of the first virtual object.

13. The method according to claim 1, further comprising: A user interface element is displayed simultaneously with the first virtual object, the user interface element comprising one or more selectable options that can be selected to perform an operation associated with the first virtual object, wherein the user interface element is displayed with a curvature that is based on the curvature of the first virtual object. The method of claim 13 , wherein the user interface element is not contained in the first virtual object.

15. The method of claim 1, wherein the first virtual object is displayed in association with a selectable option that can be selected to initiate a process of changing a size of the first virtual object, the method further comprising: while displaying the first virtual object and the selectable option, wherein the first virtual object has a first size, detecting a second input via the one or more input devices, the second input comprising a respective gesture performed by a respective portion of a user of the computer system while the user's attention is directed to the selectable option, followed by movement of the respective portion of the user while in a respective pose; as well as In response to detecting the second input, the first virtual object is displayed having a second size different from the first size based on the second input.

16. The method of claim 15, wherein the first virtual object having the first size has a first size along a first direction and a second size along a second direction, and in response to detecting the second input: Based on determining that the movement of the corresponding part of the user is in the second direction, the first virtual object having the second size has the first size along the first direction and a third size along the second direction that is different from the second size.

17. The method of claim 16, wherein in response to detecting the second input: Based on determining that the movement of the corresponding part of the user is in the first direction, the first virtual object having the second size has a fourth size different from the first size along the first direction and a fifth size different from the second size along the second direction.

18. The method of claim 1, wherein the first virtual object includes a second virtual object, and: Based on determining that the first part of the second virtual object has a first spatial relationship with the first boundary of the first virtual object, the first part of the second virtual object is displayed via the display generation component with a corresponding visual characteristic having a first value, and based on determining that the first part of the second virtual object has the first spatial relationship with a second boundary of the first virtual object that is different from the first boundary, the first part of the second virtual object is displayed via the display generation component with the corresponding visual characteristic having a second value that is different from the first value.

19. The method according to claim 1, further comprising: detecting, via the one or more input devices, a second input corresponding to a request to place a second virtual object at a corresponding position in the first virtual object; as well as In response to detecting the second input: displaying the second virtual object at a second corresponding position in the first virtual object that is different from the corresponding position based on determining that placement of the second virtual object at the corresponding position will cause at least a portion of the second virtual object to extend beyond a first boundary of the first virtual object, wherein when displayed at the second corresponding position, the second virtual object does not extend beyond the first boundary of the first virtual object; as well as The second virtual object is displayed at the corresponding position based on determining that placement of the second virtual object at the corresponding position will cause at least a portion of the second virtual object to extend beyond a second boundary of the first virtual object that is different from the first boundary of the first virtual object.

20. The method according to claim 1, further comprising: while displaying the first virtual object, detecting, via the one or more input devices, a second input, the second input comprising a respective gesture performed by a respective portion of a user of the computer system followed by movement of the respective portion of the user while in a respective pose; as well as In response to detecting the second input, the first virtual object is scrolled through according to the movement of the corresponding portion of the user.

21. The method according to claim 1, further comprising: while displaying the first virtual object, wherein the first virtual object includes a second virtual object having a first visual characteristic, the first visual characteristic having a first value, detecting, via the one or more input devices, a second input corresponding to a request to scroll through the first virtual object; as well as Upon detecting the second input: scrolling through the first virtual object based on the second input; and The second virtual object is displayed with the first visual characteristic having a corresponding value different from the first value.

22. The method of claim 21 , wherein displaying the second virtual object with the first visual characteristic having the corresponding value different from the first value comprises: displaying the second virtual object with the first visual characteristic having a second value based on determining that the second input corresponds to a first scroll amount; as well as Based on determining that the second input corresponds to a second scroll amount, the second virtual object is displayed with the first visual characteristic having a third value different from the second value.

23. The method of claim 21, wherein the second virtual object is displayed in front of a background element of the first virtual object, and: prior to detecting the second input, displaying the background element with a second visual characteristic having a third value; and When the second input is detected, the background element is displayed with the second visual characteristic having the third value.

24. The method of claim 23, wherein the second visual characteristic having the third value corresponds to the background element being displayed with at least some semi-transparency.

25. The method according to claim 21, further comprising: When the second input is detected, a scroll indicator is displayed via the display generation component, the scroll indicator indicating the progress of scrolling through the first virtual object.

26. The method according to claim 25, further comprising: while displaying the scroll indicator, wherein the scroll indicator is displayed in a first presentation mode, detecting a third input via the one or more input devices, the third input comprising attention of the user directed toward the scroll indicator; as well as In response to detecting the third input, displaying the scroll indicator in a second presentation mode includes displaying within the scroll indicator: a first representation of content, the first representation of the content being included in the first virtual object at a current scroll position in the first virtual object, the first representation of the content corresponding to content currently displayed in the first virtual object; and A second representation of content, wherein the second representation of the content is included in the first virtual object at a different scroll position in the first virtual object, the different scroll position is different from the current scroll position, and the second representation of the content corresponds to content that is not currently displayed in the first virtual object.

27. The method of claim 26, wherein the third input comprises a respective portion of a user of the computer system being in a respective state associated with displaying the scroll indicator in the second rendering mode.

28. The method of claim 25, wherein displaying the scroll indicator comprises: Based on determining that the current scrolling position in the first virtual object is a first scrolling position, displaying a position indicator indicating the first scrolling position in the scroll indicator; as well as Based on determining that the current scroll position in the first virtual object is a second scroll position different from the first scroll position, the position indicator indicating the second scroll position is displayed in the scroll indicator.

29. The method according to claim 25, further comprising: while displaying the scroll indicator, detecting, via the one or more input devices, a third input directed to the scroll indicator, the third input comprising a respective gesture performed by a respective portion of a user of the computer system followed by movement of the respective portion of the user while in a respective posture; as well as In response to detecting the third input, the first virtual object is scrolled through according to the third input.

30. The method of claim 25, further comprising: while displaying the scroll indicator, detecting, via the one or more input devices, a third input directed to the scroll indicator, the third input comprising a respective gesture performed by a respective portion of a user of the computer system; as well as In response to detecting the third input: scrolling to a first scroll position in the first virtual object corresponding to the first position based on determining that the third input is directed to a first position in the scroll indicator; as well as Based on determining that the third input is directed to a second position in the scroll indicator different from the first position in the scroll indicator, scrolling to a second scroll position in the first virtual object different from the first scroll position and corresponding to the second position.

31. The method of claim 1, further comprising: when the first virtual object is displayed at the second position in the three-dimensional environment from a first viewpoint of the user with a respective amount of curvature, detecting that a current viewpoint of the user changes from the first viewpoint to a second viewpoint different from the first viewpoint; as well as In response to detecting a change in the user's current viewpoint, the first virtual object is displayed at the second position with a respective amount of curvature from a second viewpoint of the user in a three-dimensional environment via the display generation component.

32. The method of claim 31 further comprising: while displaying the first virtual object at the second position in the three-dimensional environment from a second viewpoint of the user with the respective amount of curvature, detecting a second input directed toward the first virtual object in the three-dimensional environment; as well as In response to detecting the second input, the first virtual object is displayed via the display generation component with a third amount of curvature different from the respective amount of curvature, the third amount of curvature being based on the respective distance between the first virtual object and a second viewpoint of the user in the three-dimensional environment.

33. The method of claim 1, further comprising: while displaying the first virtual object at the respective position in the three-dimensional environment in a state without curvature, receiving, via the one or more input devices, a second input corresponding to a request to move the first virtual object from the respective position in the three-dimensional environment; as well as In response to detecting the second input, the first virtual object is displayed in a three-dimensional environment in a state with curvature.

34. The method of claim 1, wherein: The first distance is greater than the second distance, and the first amount of curvature is less than the second amount of curvature.

35. The method of claim 1, wherein: When the first virtual objects are located at respective positions in the three-dimensional environment, displaying the virtual objects at the respective positions includes: Based on determining that the first virtual object is in a three-dimensional environment shared simultaneously by a group of users, and the group of users are a first average distance from the first virtual object, displaying the first virtual object with a third amount of curvature; and Based on determining that the first virtual object is in a three-dimensional environment shared simultaneously by the group of users, and the group of users are at a second average distance from the first virtual object that is different from the first average distance, the first virtual object is displayed with a fourth curvature amount that is different from the third curvature amount.

36. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generating component and one or more input devices, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 35.

37. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 35.

Images