Method and device for managing user interface-directed interactions with physical objects - Patents.com

Abstract

The method includes displaying a first graphical element associated with a first plurality of output modalities in an XR environment; detecting a movement of a physical object while displaying the first graphical element; and in response to detecting the movement of the physical object, selecting a first output modality associated with the first graphical element as a current output modality for the physical object in accordance with a determination that the movement of the physical object causes the physical object to cross a distance threshold for a first graphical element of the first graphical elements; and selecting a second output modality associated with the second graphical element as the current output modality for the physical object in accordance with a determination that the movement of the physical object causes the physical object to cross a distance threshold for a second graphical element of the first graphical elements.

Description

[Technical Field]

[0001] The present disclosure relates generally to interacting with and manipulating user interfaces, and more particularly to systems, methods, and methods for managing user interface-directed interactions with physical objects. [Background technology]

[0002] Typically, a user can interact with a user interface through a variety of input modalities, such as touch input, voice, input, stylus / peripheral input, etc. However, the workflow for performing actions within the user interface may remain the same regardless of the input modality. This misses opportunities to enhance the user experience based on input modality, etc.

[0003] The present disclosure may have a more detailed description, as can be understood by those skilled in the art, by reference to aspects of several exemplary implementations, some of which are illustrated in the accompanying drawings. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of an example operational architecture according to some implementations.

[0005] [Figure 2] FIG. 2 is a block diagram of an example controller according to some implementations.

[0006] [Figure 3] FIG. 1 is a block diagram of an exemplary electronic device according to some implementations.

[0007] [Figure 4] FIG. 1 is a block diagram of an exemplary control device according to some implementations.

[0008] [Figure 5A] FIG. 1 is a block diagram of a first portion of an exemplary content distribution architecture according to some implementations.

[0009] [Figure 5B] 1 illustrates an example data structure according to some implementations.

[0010] [Figure 5C] FIG. 1 is a block diagram of a second portion of an exemplary content distribution architecture according to some implementations.

[0011] [Figure 6A] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6B] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6C] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6D] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6E] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6F] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6G] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6H] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6I] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6J]1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6K] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6L] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6M] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6N] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6O] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations. [Figure 6P] 1 illustrates a sequence of instances for a first content delivery scenario according to some implementations.

[0012] [Figure 7A] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7B] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7C] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7D] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7E] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7F] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7G]10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7H] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7I] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7J] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7K] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7L] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7M] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations. [Figure 7N] 10 illustrates a sequence of instances for a second content delivery scenario according to some implementations.

[0013] [Figure 8A] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8B] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8C] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8D] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8E] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8F]10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8G] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8H] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8I] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8J] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8K] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8L] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations. [Figure 8M] 10 illustrates a sequence of instances for a third content delivery scenario according to some implementations.

[0014] [Figure 9A] 1 illustrates a flowchart representation of a method for selecting an output modality for a physical object when interacting with or manipulating an XR environment, according to some implementations. [Figure 9B] 1 illustrates a flowchart representation of a method for selecting an output modality for a physical object when interacting with or manipulating an XR environment, according to some implementations. [Figure 9C] 1 illustrates a flowchart representation of a method for selecting an output modality for a physical object when interacting with or manipulating an XR environment, according to some implementations.

[0015] [Figure 10A]1 shows a flowchart representation of a method for modifying parameters of a mark based on a first input (pressure) value during direct marking on a physical surface or based on a second input (pressure) value during indirect marking, according to some implementations. [Figure 10B] 1 shows a flowchart representation of a method for modifying parameters of a mark based on a first input (pressure) value during direct marking on a physical surface or based on a second input (pressure) value during indirect marking, according to some implementations.

[0016] [Figure 11A] 1 illustrates a flowchart representation of a method for changing a selection modality based on whether a user is currently grasping a physical object, according to some implementations. [Figure 11B] 1 illustrates a flowchart representation of a method for changing a selection modality based on whether a user is currently grasping a physical object, according to some implementations. [Figure 11C] 1 illustrates a flowchart representation of a method for changing a selection modality based on whether a user is currently grasping a physical object, according to some implementations. Summary of the Invention

[0017] According to common practice, the various features illustrated in the figures may not be drawn to scale. Accordingly, dimensions of various features may be arbitrarily increased or decreased for clarity. In addition, some drawings may not depict all of the components of a given system, method, or device. Finally, like reference numerals may be used throughout the specification and figures to indicate like features.

[0018] Various implementations disclosed herein include devices, systems, and methods for selecting an output modality of a physical object when interacting with or manipulating an XR environment. According to some implementations, the methods are performed in a computing system including non-transitory memory and one or more processors, the computing system being communicatively coupled to a display device and one or more input devices. The method includes displaying, via a display device, a first plurality of graphical elements associated with a first plurality of output modalities in an augmented reality (XR) environment; detecting a first movement of a physical object while displaying the first plurality of graphical elements; and, in response to detecting the first movement of the physical object, selecting a first output modality associated with the first graphical element as a current output modality for the physical object in accordance with a determination that the first movement of the physical object has caused the physical object to cross a distance threshold for a first graphical element of the first plurality of graphical elements; and selecting a second output modality associated with the second graphical element as a current output modality for the physical object in accordance with a determination that the first movement of the physical object has caused the physical object to cross a distance threshold for a second graphical element of the first plurality of graphical elements.

[0019] Various implementations disclosed herein include devices, systems, and methods for modifying parameters of a mark based on a first input (pressure) value during direct marking on a physical surface or based on a second input (pressure) value during indirect marking. According to some implementations, the method is executed in a computing system including a non-transitory memory and one or more processors, the computing system being communicatively coupled to a display device and one or more input devices. The method includes displaying a user interface via the display device; detecting a marking input by a physical object while displaying the user interface; pursuant to determining that the marking input is directed to the physical surface, displaying a mark in the user interface based on the marking input via the display device, wherein a parameter of the mark displayed based on the marking input is determined based on how firmly the physical object is pressed against the physical surface; and pursuant to determining that the marking input is not directed to the physical surface, displaying a mark in the user interface based on the marking input via the display device, wherein a parameter of the mark displayed based on the marking input is determined based on how firmly the physical object is gripped by the user.

[0020] Various implementations disclosed herein include devices, systems, and methods for changing a selection modality based on whether a user is currently gripping a physical object. According to some implementations, the method is executed in a computing system including a non-transitory memory and one or more processors, the computing system being communicatively coupled to a display device and one or more input devices. The method includes: displaying content via the display device; detecting a selection input while displaying the content and while the physical object is being held by the user; and performing an action corresponding to the selection input in response to detecting the selection input, the action including: performing a selection action on a first portion of the content in accordance with a determination that a grip posture associated with the manner in which the physical object is being held by the user corresponds to a first grip, the first portion of the content being selected based on a direction in which a portion of the physical object is facing; and performing a selection action on a second portion of the content that is different from the first portion of the content in accordance with a determination that the grip posture associated with the manner in which the physical object is being held by the user does not correspond to the first grip, the second portion of the content being selected based on a gaze direction of the user.

[0021] In some implementations, an electronic device includes one or more displays, one or more processors, a non-transitory memory, and one or more programs, the one or more programs stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions to perform or cause to be performed any of the methods described herein. According to some implementations, a non-transitory computer-readable storage medium has instructions stored therein that, when executed by one or more processors of the device, cause the device to perform or cause to be performed any of the operations of the methods described herein. According to some implementations, a device includes one or more displays, one or more processors, a non-transitory memory, and means for performing or causing to be performed any of the methods described herein.

[0022] According to some implementations, a computing system includes one or more processors, a non-transitory memory, an interface for communicating with a display device and one or more input devices, and one or more programs, the one or more programs being stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing or causing to be performed any of the operations of the methods described herein. According to some implementations, a non-transitory computer-readable storage medium stores instructions therein that, when executed by one or more processors of a computing system having an interface for communicating with a display device and one or more input devices, cause the computing system to perform or cause to be performed any of the operations of the methods described herein. According to some implementations, a computing system includes one or more processors, a non-transitory memory, an interface for communicating with a display device and one or more input devices, and means for performing or causing to be performed any of the operations of the methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0023] Numerous details have been described to provide a thorough understanding of the exemplary implementations shown in the drawings. However, the drawings merely illustrate some exemplary aspects of the present disclosure and therefore should not be considered limiting. Those skilled in the art will understand that other useful aspects and / or variations do not include all of the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the exemplary implementations described herein.

[0024] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment may include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through sight, touch, hearing, taste, smell, and the like. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and / or interact with through electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more properties of one or more virtual objects simulated within the XR environment are adjusted to behave according to at least one law of physics. As one example, an XR system can detect a person's head rotation and, in response, adjust the graphical content and sound fields presented to that person in a manner similar to how such views and sounds would change in a physical environment. As another example, an XR system can detect movement of an electronic device presenting the XR environment (e.g., a mobile phone, tablet, laptop, etc.) and, in response, adjust the graphical content and sound fields presented to that person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), an XR system can adjust a characteristic(s) of the graphical content within the XR environment in response to an expression of body movement (e.g., a voice command).

[0025] A wide variety of electronic systems exist that enable people to sense and / or interact with various XR environments. Examples include head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mountable system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head-mountable system may be configured to accept an external opaque display (e.g., a smartphone). A head-mountable system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mountable system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display may utilize digital light projection, OLED, LED, μLED, liquid crystal on silicon, laser-scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects a graphical image onto a person's retina. A projection system may also be configured to project virtual objects into a physical environment, for example, as a hologram or onto a physical surface.

[0026] 1 is a block diagram of an exemplary operations architecture 100 according to some implementations. While relevant features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more pertinent aspects of the exemplary implementations disclosed herein. To that end, by way of non-limiting example, operations architecture 100 includes an optional controller 110 and an electronic device 120 (e.g., a tablet, a mobile phone, a laptop, a near-eye system, a wearable computing device, etc.).

[0027] In some implementations, controller 110 is configured to manage and coordinate an XR experience (sometimes referred to herein as an "XR environment" or "virtual environment" or "graphical environment") for user 149, having left hand 150 and right hand 152, and optionally other users. In some implementations, controller 110 includes a suitable combination of software, firmware, and / or hardware. Controller 110 is described in more detail below with reference to FIG. 2. In some implementations, controller 110 is a computing device that is local or remote to physical environment 105. For example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside physical environment 105. In some implementations, the controller 110 is communicatively coupled to the electronic device 120 via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In some implementations, the functionality of the controller 110 is provided by the electronic device 120. Thus, in some implementations, components of the controller 110 are integrated into the electronic device 120.

[0028] As shown in FIG. 1 , user 149 grasps control device 130 in right hand 152. As shown in FIG. 1 , control device 130 includes first end 176 and second end 177. In various embodiments, first end 176 corresponds to the tip of control device 130 (e.g., the tip of a pencil), and second end 177 corresponds to the opposite or bottom end of control device 130 (e.g., the eraser of a pencil). As shown in FIG. 1 , control device 130 includes a touch-sensitive surface 175 for receiving touch input from user 149. In some implementations, control device 130 includes a suitable combination of software, firmware, and / or hardware. Control device 130 is described in more detail below with reference to FIG. 4 . In some implementations, control device 130 corresponds to an electronic device having a wired or wireless communication channel to controller 110. For example, control device 130 corresponds to a stylus, a finger-worn device, a handheld device, etc. In some implementations, the controller 110 is communicatively coupled to the control device 130 via one or more wired or wireless communication channels 146 (e.g., BLUETOOTH, IEEE802.11x, IEEE802.16x, IEEE802.3x, etc.).

[0029] In some implementations, electronic device 120 is configured to present audio and / or video (A / V) content to user 149. In some implementations, electronic device 120 is configured to present a user interface (UI) and / or an XR environment 128 to user 149. In some implementations, electronic device 120 includes a suitable combination of software, firmware, and / or hardware. Electronic device 120 is described in more detail below with reference to FIG. 3.

[0030] According to some implementations, the electronic device 120 presents an XR experience to the user 149 while the user 149 is physically present in a physical environment 105 that includes a table 107 within a field of view (FOV) 111 of the electronic device 120. Thus, in some implementations, the user 149 holds the electronic device 120 in their hand(s). In some implementations, the electronic device 120 is configured to present XR content (sometimes referred to herein as “graphical content” or “virtual content”) including the XR cylinder 109 while presenting the XR experience, and to enable video pass-through of the physical environment 105 (e.g., including the table 107 or a representation thereof) on the display 122. For example, the XR environment 128 including the XR cylinder 109 is stereoscopic or three-dimensional (3D).

[0031] In one example, the XR cylinder 109 corresponds to display-locked content such that the XR cylinder 109 remains displayed at the same location on the display 122 when the FOV 111 changes due to translational and / or rotational movement of the electronic device 120. As another example, the XR cylinder 109 corresponds to world-locked content such that the XR cylinder 109 remains displayed at its original location when the FOV 111 changes due to translational and / or rotational movement of the electronic device 120. Thus, in this example, if the FOV 111 does not include the origin location, the XR environment 128 does not include the XR cylinder 109. For example, the electronic device 120 corresponds to a near-eye system, a mobile phone, a tablet, a laptop, a wearable computing device, etc.

[0032] In some implementations, the display 122 corresponds to an additional display that allows optical see-through of the physical environment 105, including the table 107. For example, the display 122 corresponds to a transparent lens, and the electronic device 120 corresponds to glasses worn by the user 149. Thus, in some implementations, the electronic device 120 presents a user interface by projecting XR content (e.g., the XR cylinder 109) onto the additional display, and the user interface is then overlaid on the physical environment 105 from the perspective of the user 149. In some implementations, the electronic device 120 presents a user interface by displaying XR content (e.g., the XR cylinder 109) on the additional display, and the user interface is then overlaid on the physical environment 105 from the perspective of the user 149.

[0033] In some implementations, the user 149 wears the electronic device 120, such as a near-eye system. Thus, the electronic device 120 includes one or more displays (e.g., a single display or one for each eye) provided for displaying XR content. For example, the electronic device 120 encompasses the FOV of the user 149. In such implementations, the electronic device 120 presents the XR environment 128 by displaying data corresponding to the XR environment 128 on one or more displays or by projecting data corresponding to the XR environment 128 onto the retina of the user 149.

[0034] In some implementations, the electronic device 120 includes an integrated display (e.g., a built-in display) that displays the XR environment 128. In some implementations, the electronic device 120 includes a head-mounted enclosure. In various implementations, the head-mounted enclosure includes a mounting area to which another device having a display can be attached. For example, in some implementations, the electronic device 120 can be mounted to the head-mounted enclosure. In various implementations, the head-mounted enclosure is shaped to form a receptacle for receiving another device (e.g., the electronic device 120) that includes a display. For example, in some implementations, the electronic device 120 slides / snaps into or is otherwise attached to the head-mounted enclosure. In some implementations, a display of a device mounted in the head-mounted enclosure presents (e.g., displays) the XR environment 128. In some implementations, the electronic device 120 is replaced with an XR chamber, enclosure, or room configured to present XR content without the user 149 wearing or holding the electronic device 120.

[0035] In some implementations, the controller 110 and / or the electronic device 120 move the XR representation of the user 149 within the XR environment 128 based on movement information (e.g., torso pose data, gaze tracking data, hand / limb / finger / limb tracking data, etc.) from the electronic device 120 and / or optional remote input devices within the physical environment 105. In some implementations, the optional remote input devices correspond to fixed or movable sensing equipment (e.g., image sensors, depth sensors, infrared (IR) sensors, event cameras, microphones, etc.) within the physical environment 105. In some implementations, each of the remote input devices is configured to collect / capture input data and provide the input data to the controller 110 and / or the electronic device 120 while the user 149 is physically present within the physical environment 105. In some implementations, the remote input device includes a microphone, and the input data includes audio data (e.g., speech samples) associated with the user 149. In some implementations, the remote input device includes an image sensor (e.g., a camera), and the input data includes an image of the user 149. In some implementations, the input data characterizes torso poses of the user 149 at different times. In some implementations, the input data characterizes head poses of the user 149 at different times. In some implementations, the input data characterizes hand tracking information associated with the hands of the user 149 at different times. In some implementations, the input data characterizes the velocity and / or acceleration of a torso part of the user, such as the hands of the user 149. In some implementations, the input data indicates joint positions and / or joint orientations of the user 149. In some implementations, the remote input device includes a feedback device, such as a speaker, a light, or the like.

[0036] 2 is a block diagram of an example controller 110 according to some implementations. While certain features are shown, those skilled in the art will understand from this disclosure that for the sake of brevity, various other features are not shown so as not to obscure more pertinent aspects of the implementations disclosed herein. Thus, by way of non-limiting example, in some implementations, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0037] In some implementations, one or more communication buses 204 include circuitry that interconnects and controls communication between system components. In some implementations, one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a touchscreen, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0038] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some implementations, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some implementations, memory 220 or its non-transitory computer-readable storage medium stores the following programs, modules, and data structures, or a subset thereof, described below with reference to FIG. 2 :

[0039] Operating system 230 includes procedures for handling various basic system services and for performing hardware-dependent tasks.

[0040] In some implementations, the data acquirer 242 is configured to acquire data (e.g., captured image frames of the physical environment 105, presentation data, input data, user interaction data, camera pose tracking information, eye gaze tracking information, head / torso pose tracking information, hand / limb / finger / limb tracking information, sensor data, location data, etc.) from at least one of the I / O devices 206 of the controller 110, the I / O devices and sensors 306 of the electronic device 120, and optional remote input devices. To that end, in various implementations, the data acquirer 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0041] In some implementations, the mapper and locator engine 244 is configured to map the physical environment 105 and track the position / location of at least the electronic device 120 or the user 149 relative to the physical environment 105. To that end, in various implementations, the mapper and locator engine 244 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0042] In some implementations, data sender 246 is configured to send data (e.g., presentation data such as rendered image frames associated with the XR environment, location data, etc.) to at least electronic device 120 and optionally one or more other devices. To that end, in various implementations, data sender 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0043] In some implementations, the privacy architecture 508 is configured to ingest data and filter user information and / or identifying information within the data based on one or more privacy filters. The privacy architecture 508 is described in more detail below with reference to FIG. 5A. To that end, in various implementations, the privacy architecture 508 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0044] In some implementations, the object tracking engine 510 is configured to determine / generate an object tracking vector 511 for tracking a physics object (e.g., the control device 130 or a proxy object) based on the tracking data and update the object tracking vector 511 over time. For example, as shown in FIG. 5B , the object tracking vector 511 includes a physics object translation value 572 (e.g., associated with x, y, and z coordinates relative to the physical environment 105), a physics object rotation value 574 (e.g., roll, pitch, and yaw), one or more pressure values ​​576 associated with the physics object, optional touch input information 578 associated with the physics object, etc. The object tracking engine 510 is described in more detail below with respect to FIG. 5A . To that end, in various implementations, the object tracking engine 510 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0045] In some implementations, the eye tracking engine 512 is configured to determine / generate an eye tracking vector 513 (e.g., along with a gaze direction) as shown in FIG. 5B based on the input data and update the eye tracking vector 513 over time. For example, the gaze direction indicates a point, physical object, or region of interest (ROI) within the physical environment 105 that the user 149 is currently viewing (e.g., associated with x, y, and z coordinates relative to the physical environment 105 or the world at large). As another example, the gaze direction indicates a point, XR object, or region of interest (ROI) within the XR environment 128 that the user 149 is currently viewing (e.g., associated with x, y, and z coordinates relative to the XR environment 128). The eye tracking engine 512 is described in more detail below with reference to FIG. 5A. To that end, in various implementations, the eye tracking engine 512 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0046] In some implementations, the torso / head pose tracking engine 514 is configured to determine / generate a pose characterization vector 515 based on input data and update the pose characterization vector 515 over time. For example, as shown in FIG. 5B , the pose characterization vector 515 includes a head pose descriptor 592A (e.g., upright, down, neutral, etc.), a translation value for the head pose 592B, a rotation value for the head pose 592C, a torso pose descriptor 594A (e.g., standing, sitting, prone, etc.), a translation value for a torso segment / limb / leg / joint 594B, a rotation value for a torso segment / limb / leg / joint 594C, etc. The torso / head pose tracking engine 514 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, the torso / head pose tracking engine 514 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some implementations, the object tracking engine 510 , gaze tracking engine 512 , and torso / head pose tracking engine 514 may be located on the electronic device 120 in addition to or instead of the controller 110 .

[0047] In some implementations, the content selector 542 is configured to select XR content (sometimes referred to herein as "graphical content" or "virtual content") from the content library 545 based on one or more user requests and / or inputs (e.g., voice commands, selections from a user interface (UI) menu of XR content items, etc.). The content selector 542 is described in more detail below with reference to FIG. 5A. To that end, in various implementations, the content selector 542 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0048] In some implementations, the content library 545 includes multiple content items, such as audio / visual (A / V) content, virtual agents (VAs), and / or XR content, objects, items, scenery, etc. As an example, the XR content includes 3D reconstructions of user-captured videos, movies, TV episodes, and / or other XR content. In some implementations, the content library 545 is pre-populated or manually created by the user 149. In some implementations, the content library 545 is located locally with respect to the controller 110. In some implementations, the content library 545 is located remotely from the controller 110 (e.g., on a remote server, a cloud server, etc.).

[0049] In some implementations, the input manager 520 is configured to capture and analyze input data from various input sensors. The input manager 520 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, the input manager 520 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some implementations, the input manager 520 includes a data aggregator 521, a content selection engine 522, a grip posture evaluator 524, an output modality selector 526, and a parameter adjuster 528.

[0050] In some implementations, data aggregator 521 is configured to aggregate object tracking vector 511, gaze tracking vector 513, and pose characterization vector 515 and determine / generate characterization vector 531 (shown in FIG. 5A ) therefrom for subsequent downstream use. Data aggregator 521 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, data aggregator 521 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0051] In some implementations, the content selection engine 522 is configured to determine a selected content portion 523 (shown in FIG. 5A ) within the XR environment 128 based on the characterization vector 531 (or a portion thereof). The content selection engine 522 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, the content selection engine 522 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0052] In some implementations, the grip posture evaluator 524 is configured to determine, based on the characterization vector 531 (or a portion thereof), a grip posture 525 (as shown in FIG. 5A ) associated with the current manner in which the physics object is being held by the user 149. For example, the grip posture 525 indicates the manner in which the user 149 grasps the physics object (e.g., a proxy object, the control device 130, etc.). For example, the grip posture 525 may correspond to one of a remote-control-like grip, a pointing / wand-like grip, a writing grip, a reverse writing grip, a handle grip, a thumb-top grip, a level-like grip, a gamepad-like grip, a flute-like grip, or a fire starter-like grip, etc. The grip posture evaluator 524 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, the grip posture evaluator 524 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0053] In some implementations, the output modality selector 526 is configured to select a current output modality 527 (shown in FIG. 5A ) associated with how a physical object interacts with or manipulates the XR environment 128. For example, a first output modality corresponds to selecting / manipulating an object / content within the XR environment 128, and a second output modality corresponds to sketching, drawing, writing, etc. within the XR environment 128. The output modality selector 526 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, the output modality selector 526 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0054] In some implementations, the parameter adjuster 528 is configured to adjust a parameter value (e.g., thickness, brightness, color, texture, etc.) associated with the marking input (as shown in FIG. 5A ) directed at the XR environment 128 based on either a first input (pressure) value or a second input (pressure) value associated with the physics object. The parameter adjuster 528 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, the parameter adjuster 528 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0055] In some implementations, the content manager 530 is configured to manage and update the layout, settings, structure, etc. of the XR environment 128, including one or more of the VAs, the XR content, one or more user interface (UI) elements associated with the XR content, etc. The content manager 530 is described in more detail below with reference to FIG. 5C . To that end, in various implementations, the content manager 530 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some implementations, the content manager 530 includes a buffer 534, a content updater 536, and a feedback engine 538. In some implementations, the buffer 534 includes XR content, rendered image frames, etc., for one or more past instances and / or frames.

[0056] In some implementations, the content updater 536 is configured to modify the XR environment 128 over time based on translational or rotational movement of the electronic device 120 or physical objects in the physical environment 105, user input (e.g., hand / limb tracking input, eye tracking input, touch input, voice commands, physical object manipulation input, etc.), etc. To that end, in various implementations, the content updater 536 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0057] In some implementations, the feedback engine 538 is configured to generate sensory feedback (e.g., visual feedback such as changes in text or lighting, audio feedback, haptic feedback, etc.) associated with the XR environment 128. To that end, in various implementations, the feedback engine 538 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0058] In some implementations, the rendering engine 550 is configured to render the XR environment 128 (sometimes referred to herein as a “graphical environment” or a “virtual environment”) or image frames associated therewith, as well as the VA, the XR content, one or more UI elements associated with the XR content, and the like. To that end, in various implementations, the rendering engine 550 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some implementations, the rendering engine 550 includes a pose determination unit 552, a rendering unit 554, an optional image processing architecture 562, and an optional compositing unit 564. Those skilled in the art will understand that the optional image processing architecture 562 and the optional compositing unit 564 may be present for a video pass-through configuration, but may be removed for a full VR or optical see-through configuration.

[0059] In some implementations, the pose determiner 552 is configured to determine a current camera pose of the electronic device 120 and / or the user 149 relative to the A / V and / or XR content. The pose determiner 552 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, the pose determiner 552 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0060] In some implementations, the renderer 554 is configured to render the A / V and / or XR content according to a current camera pose relative thereto. The renderer 554 is described in more detail below with reference to FIG. 5A. To that end, in various implementations, the renderer 554 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0061] In some implementations, image processing architecture 562 is configured to obtain (e.g., receive, retrieve, or capture) an image stream including one or more images of physical environment 105 from a current camera pose of electronic device 120 and / or user 149. In some implementations, image processing architecture 562 is also configured to perform one or more image processing operations on the image stream, such as warping, color correction, gamma correction, sharpening, noise reduction, white balancing, etc. Image processing architecture 562 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, image processing architecture 562 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0062] In some implementations, the compositor 564 is configured to composite the rendered A / V and / or XR content with the processed image stream of the physical environment 105 from the image processing architecture 562 to generate rendered image frames of the XR environment 128 for display. The compositor 564 is described in more detail below with reference to FIG. 5A . To that end, in various implementations, the compositor 564 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0063] Although the data acquisition unit 242, the mapper and locator engine 244, the data sender 246, the privacy architecture 508, the object tracking engine 510, the gaze tracking engine 512, the torso / head pose tracking engine 514, the content selector 542, the content manager 530, the motion modality manager 540, and the rendering engine 550 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other implementations, any combination of the data acquisition unit 242, the mapper and locator engine 244, the data sender 246, the privacy architecture 508, the object tracking engine 510, the gaze tracking engine 512, the torso / head pose tracking engine 514, the content selector 542, the content manager 530, the motion modality manager 540, and the rendering engine 550 may be located in separate computing devices.

[0064] In some implementations, functionality and / or components of controller 110 are combined with or provided by electronic device 120, shown below in FIG. 3. Furthermore, FIG. 2 is not intended as a structural overview of the implementations described herein, but rather as a description of the functionality of various features present in particular implementations. As will be recognized by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in FIG. 2 can be implemented within a single module, and various functions of a single functional block can be implemented by one or more functional blocks in various implementations. The actual number of modules, as well as the division of specific functionality and how functions are allocated among them, will vary from implementation to implementation and, in some implementations, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0065] 3 is a block diagram of an example of an electronic device 120 (e.g., a mobile phone, a tablet, a laptop, a near-eye system, a wearable computing device, etc.) according to some implementations. While certain features are shown, those skilled in the art will understand from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, by way of non-limiting example, in some implementations, electronic device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, image capture devices 370 (e.g., one or more optional inward-facing and / or outward-facing image sensors), memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0066] In some implementations, the one or more communication buses 304 include circuitry that interconnects and controls communication between system components. In some implementations, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a magnetometer, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oximetry monitor, a blood glucose monitor, etc.), one or more microphones, one or more speakers, a haptic engine, a heating and / or cooling unit, a skin shear engine, one or more depth sensors (e.g., structured light, time-of-flight, LiDAR, etc.), a localization and mapping engine, an eye gaze tracking engine, a torso / head pose tracking engine, a hand / limb / finger / limb tracking engine, a camera pose tracking engine, etc.

[0067] In some implementations, one or more displays 312 are configured to present the XR environment to the user. In some implementations, one or more displays 312 are also configured to present flat video content to the user (e.g., two-dimensional or "flat" AVI, FLV, WMV, MOV, MP4, etc. files associated with TV episodes or movies, or live video pass-through of the physical environment 105). In some implementations, one or more displays 312 correspond to touchscreen displays. In some implementations, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more displays 312 correspond to a waveguide display, such as a diffractive, reflective, polarized, holographic, etc. For example, electronic device 120 includes a single display. In another example, electronic device 120 includes a display for each eye of the user. In some implementations, one or more displays 312 can present AR and VR content. In some implementations, one or more displays 312 can present AR or VR content.

[0068] In some implementations, image capture device 370 includes one or more RGB cameras (e.g., with a complementary metal-oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), IR image sensors, event-based cameras, etc. In some implementations, image capture device 370 includes a lens assembly, a photodiode, and a front-end architecture. In some implementations, image capture device 370 includes outward-facing and / or inward-facing image sensors.

[0069] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some implementations, memory 320, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and presentation engine 340:

[0070] The operating system 330 includes procedures for handling various basic system services and for performing hardware-dependent tasks. In some implementations, the presentation engine 340 is configured to present media items and / or XR content to a user via one or more displays 312. To that end, in various implementations, the presentation engine 340 includes a data acquisition unit 342, a presentation unit 570, an interaction processor 540, and a data transmission unit 350.

[0071] In some implementations, the data acquirer 342 is configured to acquire data (e.g., presentation data such as rendered image frames related to a user interface or an XR environment, input data, user interaction data, head tracking information, camera pose tracking information, gaze tracking information, hand / limb / finger / limb tracking information, sensor data, location data, etc.) from at least one of the I / O devices and sensors 306 of the electronic device 120, the controller 110, and a remote input device. To that end, in various implementations, the data acquirer 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0072] In some implementations, the interaction processor 540 is configured to detect user interactions with the presented A / V and / or XR content (e.g., gesture input detected via hand / limb tracking, gaze input detected via eye tracking, voice commands, etc.) To that end, in various implementations, the interaction processor 540 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0073] In some implementations, the presenter 570 is configured to present and update A / V content and / or XR content (e.g., rendered image frames associated with a user interface or XR environment 128, including VA, XR content, one or more UI elements associated with the XR content, etc.) via one or more displays 312. To that end, in various implementations, the presenter 570 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0074] In some implementations, the data sender 350 is configured to send data (e.g., presentation data, location data, user interaction data, head tracking information, camera pose tracking information, gaze tracking information, hand / limb / finger / limb tracking information, etc.) to at least the controller 110. To that end, in various implementations, the data sender 350 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0075] Although the data acquisition unit 342, the interaction processing unit 540, the presentation unit 570, and the data transmission unit 350 are shown as residing on a single device (e.g., the electronic device 120), it should be understood that in other implementations, any combination of the data acquisition unit 342, the interaction processing unit 540, the presentation unit 570, and the data transmission unit 350 may be located within separate computing devices.

[0076] Furthermore, Figure 3 is not intended as a structural overview of the implementations described herein, but rather as a description of the functionality of various features that may be present in particular implementations. As will be recognized by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in Figure 3 can be implemented within a single module, and various functions of a single functional block can be implemented by one or more functional blocks in various implementations. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation and, in some implementations, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0077] 4 is a block diagram of an exemplary control device 130 according to some implementations. The control device 130 may also be referred to simply as a stylus. The control device 130 includes non-transitory memory 402 (optionally including one or more computer-readable storage media), a memory controller 422, one or more processing units (CPUs) 420, a peripherals interface 418, RF circuitry 408, an input / output (I / O) subsystem 406, and other input or control devices 416. The control device 130 optionally includes an external port 424 and one or more optical sensors 464. The control device 130 optionally includes one or more contact intensity sensors 465 for detecting the intensity of a contact of the control device 130 on the electronic device 100 (e.g., when the control device 130 is used with a touch-sensitive surface such as the display system 122 of the electronic device 120) or on another surface (e.g., a desk surface). Control device 130 optionally includes one or more tactile output generators 463 for generating tactile outputs on control device 130. These components optionally communicate via one or more communication buses or signal lines 403.

[0078] It should be understood that control device 130 is merely one example of an electronic stylus, and that control device 130 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of those components. The various components shown in FIG. 4 are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application-specific integrated circuits. In some implementations, some functions and / or operations of control device 130 (e.g., touch interpretation module 477) are provided by controller 110 and / or electronic device 120. Thus, in some implementations, some components of control device 130 are integrated into controller 110 and / or electronic device 120.

[0079] 1, control device 130 includes a first end 176 and a second end 177. In various embodiments, first end 176 corresponds to the tip of control device 130 (e.g., the tip of a pencil) and second end 177 corresponds to the opposite or bottom end of control device 130 (e.g., the eraser of a pencil).

[0080] As shown in FIG. 1 , control device 130 includes touch-sensitive surface 175 for receiving touch input from user 149. In some implementations, touch-sensitive surface 175 corresponds to a capacitive touch element. Control device 130 includes a sensor or set of sensors that detects input from the user based on haptic and / or tactile contact with touch-sensitive surface 175. In some implementations, control device 130 detects contact and its movement or breaking using any of a number of now-known or later-developed touch-sensing technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements that determine one or more points of contact with touch-sensitive surface 175. Because control device 130 includes a variety of sensors and types of sensors, control device 130 can detect a variety of different inputs from user 149. In some implementations, the one or more sensors can detect a single touch input or a sequence of touch inputs in response to a user tapping touch-sensitive surface 175 one or more times. In some implementations, one or more sensors can detect a swipe input on control device 130 in response to a user stroking one or more fingers along touch-sensitive surface 175. In some implementations, if the speed at which the user strokes along touch-sensitive surface 175 exceeds a threshold, the one or more sensors detect a flick input rather than a swipe input.

[0081] The control device 130 also includes one or more sensors that detect the orientation (e.g., angular position) and / or movement of the control device 130, such as one or more accelerometers 467, one or more gyroscopes 468, one or more magnetometers 469, etc. The one or more sensors can detect various rotational movements of the control device 130 by a user, including the type and direction of rotation. For example, the one or more sensors can detect that a user is rolling and / or rotating the control device 130 and can detect the direction of the roll / rotation (e.g., clockwise or counterclockwise). In some implementations, the detected input depends on the angular positions of the first end 176 and second end 177 of the control device 130 relative to the electronic device. For example, in some implementations, when the control device 130 is substantially perpendicular to the electronic device and the second end 177 (e.g., an eraser) is closer to the electronic device, an erase operation is performed by contacting a surface of the electronic device with the second end 177. On the other hand, when the control device 130 is substantially perpendicular to the electronic device and the first end 176 (e.g., the tip) is closer to the electronic device, the marking operation is performed by contacting the surface of the electronic device with the first end 176.

[0082] Memory 402 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more flash memory devices or other non-volatile solid-state memory devices. Access to memory 402 by other components of control device 130, such as CPU(s) 420 and peripherals interface 418, is optionally controlled by memory controller 422.

[0083] A peripheral interface 418 can be used to couple the stylus input and output peripherals to CPU(s) 420 and memory 402. The one or more processors 420 operate or execute various software programs and / or instruction sets stored in memory 402 to perform various functions and process data for control device 130. In some implementations, peripheral interface 418, CPU(s) 420, and memory controller 422 are optionally implemented on a single chip, such as chip 404. In some other embodiments, they are optionally implemented on separate chips.

[0084] RF (radio frequency) circuitry 408 transmits and receives RF signals, also referred to as electromagnetic signals. RF circuitry 408 converts electrical signals to or from electromagnetic signals and communicates with communication networks and / or other communication devices, such as controller 110, electronic device 120, etc. RF circuitry 408 optionally includes well-known circuits for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, etc. RF circuitry 408 optionally communicates via wireless communication with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet, and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs), and with other devices. The wireless communication optionally includes a communication protocol such as Global System for Mobile Communications (GSM)®, Enhanced Data GSM Environment (EDGE)®, High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), BLUETOOTH, Wireless Fidelity (Wi-Fi)® (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE IEEE 802.11g, and / or IEEE 802.11n), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0085] I / O subsystem 406 couples input / output peripherals on control device 130, such as other input or control devices 416, to peripheral interface 418. I / O subsystem 406 optionally includes one or more input controllers 460 for light sensor controller 458, intensity sensor controller 459, haptic feedback controller 461, and other input or control devices. One or more input controllers 460 receive electrical signals from and send electrical signals to other input or control devices 416. Other input or control devices 416 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, input controller(s) 460 are optionally coupled to either (or neither) an infrared port and / or a USB port.

[0086] The control device 130 also includes a power system 462 that provides power to the various components. The power system 462 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, power failure detection circuitry, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with generating, managing, and distributing power in the portable device and / or portable accessory.

[0087] The control device 130 also optionally includes one or more optical sensors 464. Figure 4 shows an optical sensor coupled to an optical sensor controller 458 in the I / O subsystem 406. The one or more optical sensors 464 optionally include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The one or more optical sensors 464 receive light from the projected environment through one or more lenses and convert the light into data representing an image.

[0088] Control device 130 also optionally includes one or more contact intensity sensors 465. FIG. 4 shows a contact intensity sensor coupled to intensity sensor controller 459 in I / O subsystem 406. Contact intensity sensor 465 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact against a surface or against the grasp of user 149). Contact intensity sensor 465 receives contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some implementations, at least one contact intensity sensor is juxtaposed to or proximate to the tip of control device 130. In some implementations, at least one contact intensity sensor is juxtaposed to or proximate to the body of control device 130.

[0089] The control device 130 also optionally includes one or more proximity sensors 466. Figure 4 shows the one or more proximity sensors 466 coupled to the peripherals interface 418. Alternatively, the one or more proximity sensors 466 are optionally coupled to an input controller 460 in the I / O subsystem 406. In some implementations, the one or more proximity sensors 466 determine the proximity of the control device 130 to an electronic device (e.g., the electronic device 120).

[0090] Control device 130 also optionally includes one or more tactile output generators 463. FIG. 4 shows a tactile output generator coupled to haptic feedback controller 461 in I / O subsystem 406. One or more tactile output generator(s) 463 optionally include one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile output on an electronic device). One or more tactile output generator(s) 463 receive tactile feedback generation instructions from haptic feedback module 433 and generate tactile outputs on control device 130 that can be sensed by a user of control device 130. In some implementations, at least one tactile output generator is juxtaposed with or adjacent to the length (e.g., body or housing) of the control device 130 and, optionally, generates a tactile output by moving the control device 130 vertically (e.g., parallel to the length of the control device 130) or laterally (e.g., normal to the length of the control device 130).

[0091] Control device 130 also optionally includes one or more accelerometers 467, one or more gyroscopes 468, and / or one or more magnetometers 469 (e.g., as part of an inertial measurement unit (IMU)) for obtaining information regarding the location and positional state of control device 130. FIG. 4 shows sensors 467, 468, and 469 coupled to peripherals interface 418. Alternatively, sensors 467, 468, and 469 are optionally coupled to input controller 460 in I / O subsystem 406. Control device 130 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the location of control device 130.

[0092] Control device 130 includes touch sensing system 432. Touch sensing system 432 detects inputs received on touch-sensitive surface 175. These inputs include those described herein with respect to touch-sensitive surface 175 of control device 130. For example, touch sensing system 432 can detect tap inputs, rotate inputs, roll inputs, flick inputs, swipe inputs, etc. Touch sensing system 432 cooperates with touch interpretation module 477 to decipher the particular type of touch input (e.g., rotate / roll / flick / swipe, etc.) received on touch-sensitive surface 175.

[0093] In some implementations, the software components stored in memory 402 include an operating system 426, a communications module (or instruction set) 428, a contact / motion module (or instruction set) 430, a position module (or instruction set) 431, and a global positioning system (GPS) module (or instruction set) 435. Additionally, in some implementations, memory 402 stores device / global internal state 457, as shown in FIG. 4 . Additionally, memory 402 includes a touch interpretation module 477. Device / global internal state 457 includes one or more of sensor state, including information obtained from various sensors of the stylus and other input or control devices 416, information regarding the position and / or orientation of control device 130 (e.g., translation and / or rotation values), and location information regarding the location of control device 130 (e.g., as determined by GPS module 435).

[0094] Operating system 426 (e.g., an embedded operating system such as iOS, Darwin®, RTXC®, LINUX®, UNIX®, OS X®, WINDOWS®, or VxWorks®) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, power management, etc.) and facilitating communication between various hardware and software components. Communications module 428 optionally also includes various software components for facilitating communication with other devices via one or more external ports 424 and for processing data received by RF circuitry 408 and / or external ports 424. External ports 424 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) are adapted to couple to other devices directly or indirectly via a network (e.g., the Internet, wireless LAN, etc.).

[0095] The contact / motion module 430 optionally detects contact with the control device 130 and other touch-sensitive devices of the control device 130 (e.g., buttons or other touch-sensitive components of the control device 130). The contact / motion module 430 includes software components for performing various operations related to detecting contact (e.g., detecting the tip of a stylus against a touch-sensitive display, such as the display 122 of the electronic device 120, or against another surface, such as a desk surface), such as determining whether contact has occurred (e.g., detecting a touch-down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact (e.g., across the display 122 of the electronic device 120) and tracking of that movement, and determining whether the contact has ceased (e.g., detecting a lift-off event or interruption of contact). In some implementations, the contact / motion module 430 receives contact data from the I / O subsystem 406. Determining the movement of the contact point, as represented by the series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. As mentioned above, in some implementations, one or more of these operations related to detecting a contact are performed by electronic device 120 or controller 110 (in addition to or instead of a stylus using contact / motion module 430).

[0096] The contact / motion module 430 optionally detects gesture input by the control device 130. Different gestures by the control device 130 have different contact patterns (e.g., different movements, timing, and / or strength of the detected contact). Thus, gestures are optionally detected by detecting a particular contact pattern. For example, detecting a single tap gesture includes detecting a touch-down event, followed by detecting a lift-off event at the same position (or substantially the same position) as the touch-down event (e.g., at the position of an icon). As another example, detecting a swipe gesture includes detecting a touch-down event, followed by detecting one or more stylus drag events, and then detecting a lift-off event. As mentioned above, in some implementations, gesture detection is performed by an electronic device using the contact / motion module 430 (in addition to or instead of a stylus using the contact / motion module 430).

[0097] The position module 431, together with one or more accelerometers 467, one or more gyroscopes 468, and / or one or more magnetometers 469, optionally detects position information related to the stylus, such as the attitude (e.g., roll, pitch, and / or yaw) of the control device 130 in a particular frame of reference. The position module 431, in conjunction with the one or more accelerometers 467, one or more gyroscopes 468, and / or one or more magnetometers 469, optionally detects movement gestures, such as flicks, taps, and rolls of the control device 130. The position module 431 includes software components for performing various operations related to detecting the position of the stylus within a particular coordinate system and detecting changes in the position of the stylus. In some implementations, the position module 431 detects the positional state of the control device 130 relative to the physical environment 105 or the world at large, and detects changes in the positional state of the control device 130.

[0098] The haptic feedback module 433 includes various software components that generate instructions used by one or more tactile output generators 463 to generate tactile outputs at one or more locations on the control device 130 in response to user interactions with the control device 130. The GPS module 435 determines the location of the control device 130 and provides this information for use in various applications (e.g., applications that provide location-based services, such as applications for finding lost devices and / or accessories).

[0099] Touch interpretation module 477 cooperates with touch sensing system 432 to determine (e.g., decode or identify) the type of touch input received on touch-sensitive surface 175 of control device 130. For example, touch interpretation module 477 determines that the touch input corresponds to a swipe input (as opposed to a tap input) if the user strokes a sufficient distance across touch-sensitive surface 175 of control device 130 in a sufficiently short amount of time. As another example, touch interpretation module 477 determines that the touch input corresponds to a flick input (as opposed to a swipe input) if the speed at which the user strokes touch-sensitive surface 175 of control device 130 is sufficiently faster than the speed corresponding to a swipe input. The threshold speed of the stroke can be preset and can be changed. In various embodiments, the pressure and / or force with which the touch is received on the touch-sensitive surface determines the type of input. For example, a light touch can correspond to a first type of input, and a stronger touch can correspond to a second type of input.

[0100] Each of the above-identified modules and applications corresponds to executable instruction sets that perform one or more of the functions described above and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). The modules (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules; thus, various subsets of the modules are optionally combined or otherwise rearranged in various embodiments. In some implementations, memory 402 optionally stores a subset of the above-identified modules and data structures. Additionally, memory 402 optionally stores additional modules and data structures not described above.

[0101] 5A is a block diagram of a first portion 500A of an exemplary content distribution architecture according to some implementations. While relevant features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for brevity's sake so as not to obscure more pertinent aspects of the exemplary implementations disclosed herein. To that end, by way of non-limiting example, the content distribution architecture may be rendered and presented by a computing system such as the controller 110 shown in FIGS. 1 and 2, the electronic device 120 shown in FIGS. 1 and 3, and / or a suitable combination thereof.

[0102] 5A , one or more local sensors 502 of the controller 110, the electronic device 120, and / or a combination thereof acquire local sensor data 503 associated with the physical environment 105. For example, the local sensor data 503 may include an image or stream of an image of the physical environment 105, simultaneous location and mapping (SLAM) information about the physical environment 105, a location of the electronic device 120 or the user 149 relative to the physical environment 105, ambient lighting information about the physical environment 105, ambient sound information about the physical environment 105, acoustic information about the physical environment 105, dimensional information about the physical environment 105, semantic labels for objects in the physical environment 105, etc. In some implementations, the local sensor data 503 includes raw or post-processed information.

[0103] 5A , one or more remote sensors 504 associated with the physical environment 105, the control device 130, and / or optional remote input devices, etc., acquire remote sensor data 505 associated with the physical environment 105. For example, the remote sensor data 505 may include an image of the physical environment 105 or a stream thereof, SLAM information of the physical environment 105 and the location of the electronic device 120 or the user 149 relative to the physical environment 105, ambient lighting information of the physical environment 105, ambient sound information of the physical environment 105, acoustic information of the physical environment 105, dimensional information of the physical environment 105, semantic labels of objects in the physical environment 105, and / or the like. In some implementations, the remote sensor data 505 includes raw or post-processed information.

[0104] 5A , tracking data 506 is acquired by at least one of the controller 110, the electronic device 120, or the control device 130 to locate and track the control device 130. As one example, the tracking data 506 includes images or streams of images of the physical environment 105 captured by an outward-facing image sensor of the electronic device 120, including the control device 130. As another example, the tracking data 506 corresponds to IMU information, accelerometer information, gyroscope information, magnetometer information, and / or the like from integrated sensors of the control device 130.

[0105] According to some implementations, the privacy architecture 508 captures local sensor data 503, remote sensor data 505, and tracking data 506. In some implementations, the privacy architecture 508 includes one or more privacy filters associated with user information and / or identification information. In some implementations, the privacy architecture 508 includes an opt-in feature whereby the electronic device 120 notifies the user 149 regarding what user information and / or identification information is being monitored and how the user information and / or identification information will be used. In some implementations, the privacy architecture 508 selectively prevents and / or restricts the content delivery architecture 500A / 500B, or portions thereof, from obtaining and / or transmitting user information. To this end, the privacy architecture 508 receives user preferences and / or selections from the user 149 in response to prompting the user 149 for the user preferences and / or selections. In some implementations, the privacy architecture 508 prevents the content delivery architecture 500A / 500B from obtaining and / or transmitting user information unless and until the privacy architecture 508 obtains informed consent from the user 149. In some implementations, the privacy architecture 508 anonymizes (e.g., scrambles, obscures, encrypts, etc.) certain types of user information. For example, the privacy architecture 508 receives user input specifying which types of user information the privacy architecture 508 anonymizes. As another example, the privacy architecture 508 anonymizes (e.g., automatically) certain types of user information that are likely to include sensitive and / or identifying information, regardless of user specification.

[0106] According to some implementations, the object tracking engine 510 obtains the tracking data 506 after it has been subjected to the privacy architecture 508. In some implementations, the object tracking engine 510 determines / generates an object tracking vector 511 for the physical object based on the tracking data 506 and updates the object tracking vector 511 over time. As an example, the physical object corresponds to a proxy object detected in the physical environment 105 that does not have a communication channel to a computing system (e.g., the controller 110, the electronic device 120, etc.), such as a pencil, a pen, etc. As another example, the physical object corresponds to an electronic device (e.g., the control device 130) that has a wired or wireless communication channel to a computing system (e.g., the controller 110, the electronic device 120, etc.), such as a stylus, a finger-worn device, a handheld device, etc.

[0107] 5B shows an example data structure of object tracking vector 511 according to some implementations. As shown in FIG. 5B , object tracking vector 511 may correspond to an N-tuple characterization vector or characterization tensor that includes a timestamp 571 (e.g., the most recent time that object tracking vector 511 was updated), one or more translation values ​​572 of the physics object (e.g., x, y, and z values ​​relative to physical environment 105, the world at large, etc.), one or more rotation values ​​574 of the physics object (e.g., roll, pitch, and yaw values), one or more pressure values ​​576 associated with the physics object (e.g., a first input (pressure) value associated with contact between the end of control device 130 and a surface, a second input (pressure) value associated with the amount of pressure applied to the body of control device 130 while being gripped by user 149, etc.), optional touch input information 578 (e.g., information associated with a user touch input directed at touch-sensitive surface 175 of control device 130), and / or miscellaneous information 579. Those skilled in the art will understand that the data structure for object tracking vector 511 in FIG. 5B is merely an example that may contain different pieces of information in various other implementations and may be structured in countless ways in various other implementations.

[0108] According to some implementations, the eye tracking engine 512 acquires the local sensor data 503 and the remote sensor data 505 after being subjected to the privacy architecture 508. In some implementations, the eye tracking engine 512 determines / generates an eye tracking vector 513 associated with the gaze direction of the user 149 based on the input data and updates the eye tracking vector 513 over time.

[0109] 5B illustrates an exemplary data structure of an eye-tracking vector 513 according to some implementations. As shown in FIG. 5B, the eye-tracking vector 513 may correspond to an N-tuple characterization vector or characterization tensor that includes a timestamp 581 (e.g., the most recent time the eye-tracking vector 513 was updated), one or more angle values ​​582 (e.g., roll, pitch, and yaw values) relative to the current gaze direction of the user 149, one or more translation values ​​584 (e.g., x, y, and z values ​​relative to the physical environment 105, the world at large, etc.) relative to the current gaze direction of the user 149, and / or other information 586. Those skilled in the art will appreciate that the data structure of the eye-tracking vector 513 in FIG. 5B is merely an example that may include different pieces of information in various other implementations and may be structured in countless ways in various other implementations.

[0110] For example, the gaze direction indicates a point, physical object, or region of interest (ROI) within the physical environment 105 (e.g., associated with x, y, and z coordinates relative to the physical environment 105 or the world at large) that the user 149 is currently viewing. As another example, the gaze direction indicates a point, XR object, or region of interest (ROI) within the XR environment 128 (e.g., associated with x, y, and z coordinates relative to the XR environment 128) that the user 149 is currently viewing.

[0111] According to some implementations, the torso / head pose tracking engine 514 acquires the local sensor data 503 and the remote sensor data 505 after being subjected to the privacy architecture 508. In some implementations, the torso / head pose tracking engine 514 determines / generates a pose characterization vector 515 based on the input data and updates the pose characterization vector 515 over time.

[0112] 5B shows an example data structure of a posture characterization vector 515 according to some implementations. As shown in FIG. 5B, the posture characterization vector 515 may correspond to an N-tuple characterization vector or characterization tensor that includes a timestamp 591 (e.g., the most recent time the posture characterization vector 515 was updated), a head pose descriptor 592A (e.g., face up, face down, neutral, etc.), a head pose translation value 592B, a head pose rotation value 592C, a torso pose descriptor 594A (e.g., standing, sitting, prone, etc.), a torso segment / limb / extremity / joint translation value 594B, a torso segment / limb / extremity / joint rotation value 594C, and / or miscellaneous information 596. In some implementations, the posture characterization vector 515 also includes information associated with finger / hand / limb tracking. Those skilled in the art will appreciate that the data structure for posture characterization vector 515 in FIG. 5B is merely an example that may contain different pieces of information in various other implementations and may be structured in a myriad of ways in various other implementations.

[0113] According to some implementations, the data aggregator 521 obtains the object tracking vector 511, the gaze tracking vector 513, and the pose characterization vector 515 (sometimes collectively referred to herein as "input vectors 519"). In some implementations, the data aggregator 521 aggregates the object tracking vector 511, the gaze tracking vector 513, and the pose characterization vector 515 and determines / generates the characterization vector 531 based thereon for subsequent downstream use.

[0114] In some implementations, the content selection engine 522 determines a selected content portion 523 within the XR environment 128 based on the characterization vector 531 (or a portion thereof). For example, the content selection engine 522 determines the selected content portion 523 based on current context information, the user's 149 gaze direction, body posture information associated with the user 149, head posture information associated with the user 149, hand / limb tracking information associated with the user 149, position information associated with the physics object, rotation information associated with the physics object, etc. As one example, the content selection engine 522 performs a selection operation on a first portion of the content based on a direction in which a predetermined portion (e.g., an outward-facing end) of the physics object is pointing, following a determination that a grip posture associated with a manner in which the physics object is being held by the user does not correspond to the first grip. As another example, the content selection engine 522 performs a selection operation on a second portion of the content based on the user's gaze direction, following a determination that a grip posture associated with a manner in which the physics object is being held by the user does not correspond to the first grip.

[0115] In some implementations, the grip posture evaluator 524 determines a grip posture 525 associated with the current manner in which the physics object is being held by the user 149 based on the characterization vector 531 (or a portion thereof). For example, the grip posture evaluator 524 determines the grip posture 525 based on current context information, body posture information associated with the user 149, head posture information associated with the user 149, hand / limb tracking information associated with the user 149, position information associated with the physics object, rotation information associated with the physics object, etc. In some implementations, the grip posture 525 indicates the manner in which the user 149 is gripping the physics object. For example, the grip posture 525 corresponds to one of a remote-like grip, a wand-like grip, a writing grip, a reverse writing grip, a handle grip, a thumb-top grip, a level-like grip, a gamepad-like grip, a flute-like grip, a firestarter-like grip, etc.

[0116] In some implementations, the output modality selector 526 selects a current output modality 527 associated with a manner in which a physics object interacts with or manipulates the XR environment 128. For example, a first output modality corresponds to selecting / manipulating an object / content within the XR environment 128, and a second output modality corresponds to sketching, drawing, writing, etc. within the XR environment 128. As one example, the output modality selector 526 selects the associated first output modality as the physics object's current output modality 527 pursuant to a determination that movement of the physics object has caused the physics object to exceed a distance threshold for a first graphical element of the first plurality of graphical elements. As another example, the output modality selector 526 selects the second output modality as the physics object's current output modality 527 pursuant to a determination that movement of the physics object has caused the physics object to exceed a distance threshold for a second graphical element of the first plurality of graphical elements.

[0117] In some implementations, the parameter adjuster 528 adjusts a parameter value 529 (e.g., thickness, brightness, color, texture, etc.) associated with the marking input directed toward the XR environment 128 based on either a first input (pressure) value or a second input (pressure) value associated with the physical object. As one example, the parameter adjuster 528 adjusts the parameter value 529 associated with the detected marking input directed toward the XR environment 128 based on how hard the physical object is pressed against the physical surface (e.g., based on the first input (pressure) value). As another example, the parameter adjuster 528 adjusts the parameter value 529 associated with the detected marking input directed toward the XR environment 128 based on how hard the physical object is gripped by the user 149 (e.g., the second input (pressure) value) based on the determination that the marking input is not directed toward a physical surface. In this example, the marking input is detected while the physical object or a predetermined portion of the physical object, such as the tip of the physical object, is not in contact with any physical surface within the physical environment 105 .

[0118] FIG. 5C is a block diagram of a second portion 500B of an exemplary content distribution architecture according to some implementations. While relevant features are shown, those skilled in the art will understand from this disclosure that, for brevity's sake, various other features are not shown so as to not obscure more pertinent aspects of the exemplary implementations disclosed herein. To that end, by way of non-limiting example, the content distribution architecture may be rendered and presented by a computing system such as the controller 110 shown in FIGS. 1 and 2, the electronic device 120 shown in FIGS. 1 and 3, and / or a suitable combination thereof. FIG. 5C is similar to and adapted from FIG. 5A. Accordingly, similar reference numerals are used in FIGS. 5A and 5C. Therefore, for the sake of brevity, only the differences between FIGS. 5A and 5C will be described below.

[0119] According to some implementations, the interaction processor 540 obtains (e.g., receives, retrieves, or detects) one or more user inputs 541 provided by the user 149 related to selecting A / V content, one or more VAs, and / or XR content for presentation. For example, the one or more user inputs 541 correspond to gesture inputs that modify and / or manipulate XR content or VAs within the XR environment 128 detected via hand / limb tracking, gesture inputs that select XR content within the XR environment 128 or from a UI menu detected via hand / limb tracking, gaze inputs that select XR content from the XR environment 128 or from a UI menu detected via eye tracking, voice commands that select XR content from the XR environment 128 or from a UI menu detected via a microphone, etc. In some implementations, the content selector 542 selects XR content 547 from a content library 545 based on the one or more user inputs 541.

[0120] In various implementations, the content manager 530 manages and updates the layout, setup, structure, etc. of the XR environment 128, including one or more of the VA, the XR content, one or more UI elements associated with the XR content, etc., based on the selected content portion 523, the grip pose 525, the output modality 527, the parameter values ​​529, the characterization vector 531, etc. To that end, the content manager 530 includes a buffer 534, a content updater 536, and a feedback engine 538.

[0121] In some implementations, the buffer 534 includes XR content, rendered image frames, etc. for one or more past instances and / or frames. In some implementations, the content updater 536 modifies the XR environment 128 over time based on the selected content portion 523, the grip pose 525, the output modality 527, the parameter values ​​529, the characterization vectors 531, user input 541 associated with modifying and / or manipulating the XR content or VA, translational or rotational movements of objects in the physical environment 105, translational or rotational movements of the electronic device 120 (or user 149), etc. In some implementations, the feedback engine 538 generates sensory feedback associated with the XR environment 128 (e.g., visual feedback such as changes in text or lighting, audio feedback, haptic feedback, etc.).

[0122] 5C , a pose determiner 552 determines a current camera pose of the electronic device 120 and / or the user 149 relative to the XR environment 128 and / or the physical environment 105 based at least in part on the pose characterization vector 515. In some implementations, a renderer 554 renders the VA, the XR content 547, one or more UI elements associated with the XR content, etc., according to the current camera pose relative thereto.

[0123] According to some implementations, the optional image processing architecture 562 obtains an image stream from the image capture device 370, the image stream including one or more images of the physical environment 105 from the current camera pose of the electronic device 120 and / or the user 149. In some implementations, the image processing architecture 562 also performs one or more image processing operations on the image stream, such as warping, color correction, gamma correction, sharpening, noise reduction, white balance, etc. In some implementations, the optional compositor 564 composites rendered XR content with the processed image stream of the physical environment 105 from the image processing architecture 562 to generate rendered image frames of the XR environment 128. In various implementations, the presenter 570 presents the rendered image frames of the XR environment 128 to the user 149 via one or more displays 312. Those skilled in the art will understand that the optional image processing architecture 562 and the optional compositor 564 may not be applicable to fully virtual environments (or optical see-through scenarios).

[0124] 6A-6P illustrate a sequence of instances 610-6160 for a content distribution scenario according to some implementations. While certain features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the implementations disclosed herein. To that end, by way of non-limiting example, the sequence of instances 610-6160 may be rendered and presented by a computing system such as the controller 110 shown in FIGS. 1 and 2, the electronic device 120 shown in FIGS. 1 and 3, and / or a suitable combination thereof.

[0125] 6A-6P, the content delivery scenario includes a physical environment 105 and an XR environment 128 displayed on a display 122 of an electronic device 120 (e.g., associated with a user 149). The electronic device 120 presents the XR environment 128 to the user 149 while the user 149 is physically present in the physical environment 105, which includes a door 115 that is currently within the FOV 111 of an outward-facing image sensor of the electronic device 120. Thus, in some implementations, the user 149 holds the electronic device 120 in his or her left hand 150, similar to the operating environment 100 of FIG. 1.

[0126] In other words, in some implementations, electronic device 120 is configured to present XR content and enable optical see-through or video pass-through of at least a portion of physical environment 105 on display 122 (e.g., door 115 or a representation thereof). For example, electronic device 120 may correspond to a mobile phone, a tablet, a laptop, a near-eye system, a wearable computing device, etc.

[0127] 6A , during instance 610 (e.g., associated with time T1) of the content distribution scenario, electronic device 120 presents XR environment 128 that includes representation 116 of door 115 and virtual agent (VA) 606. As shown in FIG. 6A , control device 130 is not currently being held by user 149 and is not detecting any input directed at its touch-sensitive surface 175.

[0128] 6B and 6C illustrate a sequence in which a first plurality of graphical elements associated with a first plurality of output modalities are displayed within XR environment 128 in response to detecting a touch input directed at control device 130. As shown in FIG. 6B , during instance 620 of the content delivery scenario (e.g., associated with time T2), control device 130 detects a swipe input 622 directed at touch-sensitive surface 175. In some implementations, control device 130 provides an indication of swipe input 622 to controller 110 and / or electronic device 120. In some implementations, control device 130 communicates with electronic device 120 and / or controller 130.

[0129] As shown in FIG. 6C , during instance 630 (e.g., associated with time T3) of the content delivery scenario, electronic device 120 displays graphical elements 632A, 632B, 632C, and 632D (sometimes collectively referred to herein as first plurality of graphical elements 632) in response to obtaining an indication of swipe input 622 directed toward touch-sensitive surface 175 of control device 130 in FIG. 6B or detecting swipe input 622 directed toward touch-sensitive surface 175 of control device 130 in FIG. 6B .

[0130] 6C , the electronic device 120 displays a representation 153 of the right hand 152 of the user 149 gripping the representation 131 of the control device 130. For example, the right hand 152 of the user 149 is currently gripping the control device 130 in a pointing grip posture. In some implementations, the first plurality of graphical elements 632 is a function of the current grip posture. For example, the graphical element 632A corresponds to an output modality associated with generating a pencil-like mark in the XR environment 128, the graphical element 632B corresponds to an output modality associated with generating a pen-like mark in the XR environment 128, the graphical element 632C corresponds to an output modality associated with generating a marker-like mark in the XR environment 128, and the graphical element 632D corresponds to an output modality associated with generating an airbrush-like mark in the XR environment 128.

[0131] 6C , the spatial location of the representation 131 of the control device 130 is outside of the activation region 634 associated with the graphical element 632D. In some implementations, the activation region 634 corresponds to a predetermined distance threshold, such as an X cm radius, surrounding the graphical element 632D. In some implementations, the activation region 634 corresponds to a deterministic distance threshold surrounding the graphical element 632D.

[0132] 6D and 6E illustrate a sequence in which a first output modality (e.g., airbrush marking) is selected for the control device 130 pursuant to a determination that movement of the control device 130 has caused the control device 130 (or a representation thereof) to exceed an activation area 634 (e.g., a distance threshold) for the graphical element 632D. As shown in FIG. 6D , during instance 640 (e.g., associated with time T4) of the content delivery scenario, the electronic device 120 detects that movement of the control device 130 has caused the spatial location of the representation 131 of the control device 130 to exceed (or enter) the activation area 634 (e.g., a distance threshold) for the graphical element 632D. In response to the movement of the control device 130 causing the spatial location of the representation 131 of the control device 130 to exceed (or enter) the activation area 634 for the graphical element 632D, the electronic device 120 alters the appearance of the graphical element 632D to indicate its selection by displaying a border or frame 642 around the graphical element 632D. Those skilled in the art will appreciate that the appearance of graphical element 632D may be altered in other ways to indicate its selection, such as by altering its brightness, color, texture, shape, size, glow, shadow, and / or the like.

[0133] As shown in FIG. 6E, during instance 650 (e.g., associated with time T5) of the content distribution scenario, the electronic device 120 stops displaying graphical elements 632A, 632B, and 632C in response to detecting movement of the control device 130 such that the spatial position of the representation 131 of the control device 130 exceeds the activation area 634 relative to the graphical element 632D of FIG. 6D.

[0134] 6E , in response to detecting that movement of the control device 130 has caused the spatial location of the representation 131 of the control device 130 to exceed the activation area 634 relative to the graphical element 632D in FIG. 6D , the electronic device 120 displays a graphical element 632D superimposed on the tip of the representation 131 of the control device 130 within the XR environment 128. In some implementations, in response to selection of the graphical element 632D, the graphical element 632D remains anchored to the tip of the representation 131 of the control device 130, as shown in FIGS. 6E and 6F .

[0135] 6E and 6F illustrate a sequence in which detecting a marking input causes one or more marks to be displayed within the XR environment 128 in accordance with a currently selected first output modality (e.g., an airbrush mark). As shown in FIG. 6E, during instance 650 of the content delivery scenario (e.g., associated with time T5), the electronic device 120 detects a marking input 654 by the control device 130 via hand / limb tracking. As shown in FIG. 6F, during instance 660 of the content delivery scenario (e.g., associated with time T6), the electronic device 120 displays an airbrush-like mark 662 within the XR environment 128 in response to detecting the marking input 654 in FIG. 6E. For example, the shape, depth, length, angle, etc. of the airbrush-like mark 662 correspond to spatial parameters of the marking input 654 (e.g., a position value, a rotation value, a displacement, a spatial acceleration, a spatial velocity, an angular acceleration, an angular velocity, etc., associated with the marking input).

[0136] 6G-6I illustrate a sequence in which a second output modality (e.g., pen markings) is selected for control device 130 pursuant to a determination that movement of control device 130 causes control device 130 (or a representation thereof) to exceed activation area 634 (e.g., a distance threshold) for graphical element 632B. As shown in FIG. 6G, during instance 670 (e.g., associated with time T7) of the content delivery scenario, electronic device 120 displays graphical elements 632A, 632B, 632C, and 632D (sometimes collectively referred to herein as first plurality of graphical elements 632) in response to obtaining an indication of swipe input 622 directed toward touch-sensitive surface 175 of control device 130 of FIG. 6B or detecting swipe input 622 directed toward touch-sensitive surface 175 of control device 130 of FIG. 6B.

[0137] 6H , during instance 680 (e.g., associated with time T8) of the content delivery scenario, electronic device 120 detects that movement of control device 130 causes the spatial location of representation 131 of control device 130 to exceed (or enter) activation area 634 (e.g., a distance threshold) for graphical element 632B. In response to movement of control device 130 causing the spatial location of representation 131 of control device 130 to exceed activation area 634 for graphical element 632D, electronic device 120 alters the appearance of graphical element 632B to indicate its selection by displaying a border or frame 642 around graphical element 632B. Those skilled in the art will appreciate that the appearance of graphical element 632B may be altered in other ways to indicate its selection, such as by altering its brightness, color, texture, shape, size, glow, shadow, and / or the like.

[0138] 6I, during instance 690 (e.g., associated with time T9) of the content distribution scenario, electronic device 120 ceases displaying graphical elements 632A, 632C, and 632D in response to detecting that movement of control device 130 has caused the spatial location of representation 131 of control device 130 to exceed activation region 634 relative to graphical element 632B in FIG. 6H. In some implementations, in response to selection of graphical element 632B, graphical element 632B remains anchored to the tip of representation 131 of control device 130, as shown in FIGS. 6I-6N.

[0139] 6J and 6K illustrate a sequence in which detection of a first marking input causes one or more marks to be displayed within the XR environment 128 according to a currently selected second output modality (e.g., pen marking) and a current measurement of the input (pressure) value. As shown in FIG. 6J, at a time T 10 During instance 6100 (associated with), electronic device 120 detects marking input 6104 by control device 130 through hand / limb tracking. While marking input 6104 is being detected, electronic device 120 also detects an input (pressure) value or obtains an indication of an input (pressure) value associated with how tightly the control device 130 is being gripped by right hand 152 of user 149. As one example, the input (pressure) value is detected by one or more pressure sensors integrated into the body of control device 130. As another example, the input (pressure) value is detected by using computer vision techniques to analyze finger / skin deformations, etc., in one or more images captured by an outward-facing image sensor of electronic device 120. As shown in FIG. 6J , input (pressure) value indicator 6102 indicates a current measurement 6103 of an input (pressure) value associated with how tightly the control device 130 is being gripped by user 149. According to some implementations, the input (pressure) value indicator 6102 is a diagram to guide the reader that may or may not be displayed by the electronic device 120.

[0140] As shown in FIG. 6K, in a content distribution scenario (e.g., at time T 11 During instance 6110 (associated with), the electronic device 120 displays a pen-like mark 6112 within the XR environment 128 in response to detecting the marking input 6104 in FIG. 6J. For example, the shape, depth, length, angle, etc. of the pen-like mark 6112 correspond to the spatial parameters of the marking input 6104. Further, in FIG. 6K, the pen-like mark 6112 is associated with a first thickness value that corresponds to the current measured value of the input (pressure) value 6103 in FIG.

[0141] 6L and 6M illustrate a sequence in which detection of a second marking input causes one or more marks to be displayed within the XR environment 128 according to the currently selected second output modality (e.g., pen marking) and the current measured value of the input (pressure). As shown in FIG. 6L, at a time T 12 During instance 6120 (associated with the right hand 152 of the user 149), the electronic device 120 detects a marking input 6122 by the control device 130 through hand / limb tracking. While the marking input 6122 is being detected, the electronic device 120 also detects an input (pressure) value or obtains an indication of an input (pressure) value associated with how tightly the control device 130 is being gripped by the right hand 152 of the user 149. As shown in FIG. 6L , the input (pressure) value indicator 6102 indicates a current measured value 6123 of the input (pressure) value associated with how tightly the control device 130 is being gripped by the user 149. The current measured value 6123 of the input (pressure) value in FIG. 6L is greater than the measured value 6103 of the input (pressure) value in FIG. 6J.

[0142] As shown in FIG. 6M, in a content distribution scenario (e.g., at time T 13During instance 6130 (associated with instance 6130), the electronic device 120 displays a pen-like mark 6132 within the XR environment 128 in response to detecting the marking input 6122 in Figure 6L. For example, the shape, depth, length, angle, etc. of the pen-like mark 6132 correspond to the spatial parameters of the marking input 6122. Further, in Figure 6M, the pen-like mark 6132 is associated with a second thickness value that corresponds to the current measured value 6123 of the input (pressure) value in Figure 6L. The second thickness value associated with the pen-like mark 6132 in Figure 6M is greater than the first thickness value associated with the pen-like mark 6112 in Figure 6K.

[0143] 6N and 6O illustrate a sequence in which a second plurality of graphical elements associated with a second plurality of output modalities are displayed within the XR environment 128 in response to detecting touch input directed at the control device 130. As shown in FIG. 6N, at a time (e.g., time T 14 During instance 6140 (associated with), control device 130 detects swipe input 6142 directed at touch-sensitive surface 175. In some implementations, control device 130 provides an indication of swipe input 6142 to controller 110 and / or electronic device 120.

[0144] As shown in FIG. 6O, in a content distribution scenario (e.g., at time T 15 6N ), electronic device 120 displays graphical elements 6152A, 6152B, 6152C, and 6152D (sometimes collectively referred to herein as second plurality of graphical elements 6152) in response to obtaining an indication of swipe input 6142 directed toward touch-sensitive surface 175 of control device 130 in FIG. 6N or detecting swipe input 6142 directed toward touch-sensitive surface 175 of control device 130 in FIG. 6N .

[0145] 6O, the electronic device 120 displays a representation 153 of the right hand 152 of the user 149 gripping the representation 131 of the control device 130. For example, the right hand 152 of the user 149 is currently gripping the control device 130 in a writing grip posture with the first end 176 pointing downward and the second end 177 pointing upward. In some implementations, the second plurality of graphical elements 6152 is a function of the current grip posture (e.g., the writing grip posture). For example, graphical element 6152A corresponds to an output modality associated with generating pencil-like marks within the XR environment 128, graphical element 6152B corresponds to an output modality associated with generating pen-like marks within the XR environment 128, graphical element 6152C corresponds to an output modality associated with generating thin brush-like marks within the XR environment 128, and graphical element 6152D corresponds to an output modality associated with generating thick brush-like marks within the XR environment 128.

[0146] 6O and 6P illustrate a sequence in which a third plurality of graphical elements associated with a third plurality of output modalities are displayed within the XR environment 128 in response to detecting a change in the current grip posture of the control device 130. As shown in FIG. 6O, at a time (e.g., time T 15 During instance 6150 (associated with ), electronic device 120 detects the current grip posture of control device 130 using computer vision techniques, such that right hand 152 of user 149 is currently holding control device 130 in a writing grip posture with first end 176 pointing downward and second end 177 pointing upward. However, between Figures 6O and 6P, electronic device 120 detects a change in the current grip posture of control device 130 from the writing grip posture in Figure 6O to the reverse writing grip posture in Figure 6P.

[0147] As shown in FIG. 6P, in a content distribution scenario (e.g., at time T 16During instance 6160 (associated with ), the electronic device 120 uses computer vision techniques to detect a current grip posture of the control device 130, whereby the right hand 152 of the user 149 is currently grasping the control device 130 in a reverse writing grip posture with the first end 176 pointing upward and the second end 177 pointing downward. Thus, the control device 130 has flipped 180 degrees relative to the orientation of its ends between Figures 6O and 6P. In Figure 6P, the electronic device 120 displays graphical elements 6162A, 6162B, 6162C, and 6162D (sometimes collectively referred to herein as a third plurality of graphical elements 6162) in response to detecting a change in the current grip posture of the control device 130 from the writing grip posture of Figure 6O to the reverse writing grip posture of Figure 6P.

[0148] In some implementations, the third plurality of graphical elements 6162 is a function of the current grip posture (e.g., a reverse writing grip posture). For example, graphical element 6162A corresponds to an output modality associated with erasing or removing pixels in XR environment 128 based on a first radius value, graphical element 6162B corresponds to an output modality associated with erasing or removing pixels in XR environment 128 based on a second radius value that is greater than the first radius value, graphical element 6162C corresponds to an output modality associated with measurement marks in XR environment 128, and graphical element 6162D corresponds to an output modality associated with cutting marks in XR environment 128.

[0149] 7A-7N illustrate a sequence of instances 710-7140 for a content distribution scenario according to some implementations. While certain features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the implementations disclosed herein. To that end, by way of non-limiting example, the sequence of instances 710-7140 may be rendered and presented by a computing system such as the controller 110 shown in FIGS. 1 and 2, the electronic device 120 shown in FIGS. 1 and 3, and / or a suitable combination thereof.

[0150] 7A-7N, the content delivery scenario includes a physical environment 105 and an XR environment 128 displayed on a display 122 of an electronic device 120 (e.g., associated with a user 149). The electronic device 120 presents the XR environment 128 to the user 149 while the user 149 is physically present in the physical environment 105, including the position of a table 107 that is currently within the FOV 111 of an outward-facing image sensor of the electronic device 120. Thus, in some implementations, the user 149 holds the electronic device 120 in their left hand 150 or right hand 152.

[0151] In other words, in some implementations, electronic device 120 is configured to present XR content and enable optical see-through or video pass-through of at least a portion of physical environment 105 on display 122 (e.g., table 107). For example, electronic device 120 may correspond to a mobile phone, a tablet, a laptop, a near-eye system, a wearable computing device, etc.

[0152] As shown in FIG. 7A , during an instance 710 (e.g., associated with time T1) of the content distribution scenario, the electronic device 120 presents an XR environment 128 that includes a portion of the table 107, a virtual agent (VA) 606, an XR substrate 718 (e.g., a 2D or 3D canvas), and a menu 712. As shown in FIG. 7A , the electronic device 120 also displays a representation 151 of a left hand 150 of a user 149 holding a representation 131 of a control device 130 within the XR environment 128. For example, the left hand 150 of the user 149 is currently holding the control device 130 in a writing grip posture. As shown in FIG. 7A , the menu 712 includes multiple selectable options 714 associated with changing the appearance (e.g., a different color, texture, etc.) of marks created within the XR environment. For example, option 714A is currently selected from among the multiple selectable options 714. In this example, option 714A corresponds to a first appearance of a mark (e.g., a black mark) created within the XR environment 128. As shown in FIG. 7A, the menu 712 also includes a slider 716 for adjusting the thickness of marks created in the XR environment 128.

[0153] 7A and 7B illustrate a sequence in which detection of a first marking input causes one or more marks to be displayed within the XR environment 128 according to a first measurement of an input (pressure) value. As shown in FIG. 7A , during an instance 710 (e.g., associated with time T1) of the content distribution scenario, the electronic device 120 detects a marking input 715 by the control device 130 through hand / limb tracking. While the marking input 715 is being detected, the electronic device 120 also detects an input (pressure) value or obtains an indication of an input (pressure) value associated with how tightly the control device 130 is gripped by the left hand 150 of the user 149. As one example, the input (pressure) value is detected by one or more pressure sensors integrated into the body of the control device 130. As another example, the input (pressure) value is detected by using computer vision techniques to analyze finger / skin deformations, etc., in one or more images captured by an outward-facing image sensor of the electronic device 120. 7A, the input (pressure) value indicator 717 shows a current measurement 719 of the input (pressure) value associated with how tightly the control device 130 is being gripped by the user 149. According to some implementations, the input (pressure) value indicator 717 is a diagram for guiding the reader that may or may not be displayed by the electronic device 120.

[0154] As shown in Figure 7B, during an instance 720 (e.g., associated with time T2) of the content distribution scenario, the electronic device 120 displays a mark 722 on an XR substrate 718 within the XR environment 128 in response to detecting the marking input 715 in Figure 7A. For example, the shape, depth, length, angle, etc. of the marking input 715 correspond to the spatial parameters (e.g., the position value, rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc. associated with the marking input) of the marking input 715. Additionally, in Figure 7B, the mark 722 is associated with a first thickness value that corresponds to the current measured value 719 of the input (pressure) value in Figure 7A.

[0155] 7C and 7D illustrate a sequence in which detecting a second marking input causes one or more marks to be displayed within the XR environment 128 according to a second measurement of the input (pressure) value. As shown in FIG. 7C , during an instance 730 of the content distribution scenario (e.g., associated with time T3), the electronic device 120 detects a marking input 732 by the control device 130 through hand / limb tracking. While the marking input 732 is being detected, the electronic device 120 also detects an input (pressure) value or obtains an indication of an input (pressure) value associated with how tightly the control device 130 is being grasped by the left hand 150 of the user 149. As shown in FIG. 7C , the input (pressure) value indicator 717 indicates a current measurement 739 of the input (pressure) value associated with how tightly the control device 130 is being grasped by the user 149. For example, the current measurement 739 in FIG. 7C is greater than the measurement 719 in FIG. 7A .

[0156] As shown in Figure 7D, during instance 740 (e.g., associated with time T4) of the content delivery scenario, electronic device 120 displays mark 742 on XR substrate 718 within XR environment 128 in response to detecting marking input 732 in Figure 7C. For example, the shape, depth, length, angle, etc. of mark 742 correspond to the spatial parameters of marking input 732. Furthermore, in Figure 7D, mark 742 is associated with a second thickness value that corresponds to the current measured value 739 of the input (pressure) value in Figure 7C. For example, the second thickness value associated with mark 742 is greater than the first thickness value associated with mark 722.

[0157] 7E and 7F illustrate a sequence in which detection of a control input causes one or more marks to be translated within XR environment 128. As shown in FIG. 7E , during instance 750 (e.g., associated with time T5) of the content delivery scenario, electronic device 120 detects control input 752 by control device 130 corresponding to translating mark 742 within XR environment 128. While control input 752 is being detected, electronic device 120 also detects touch input 754 directed toward touch-sensitive surface 175 of control device 130 or obtains an indication of touch input 754 directed toward touch-sensitive surface 175 of control device 130. As one example, touch input 754 is detected by touch sensing 175 of control device 130. As another example, touch input 754 is detected by using computer vision techniques to analyze one or more images captured by an outward-facing image sensor of electronic device 120.

[0158] 7F , during instance 760 (e.g., associated with time T6) of the content delivery scenario, electronic device 120 translates mark 742 within XR environment 128 in response to detecting control input 752 in FIG. 7E while also detecting touch input 754 directed toward touch-sensitive surface 175 of control device 130 in FIG. 7E . In some implementations, the detection of control input 752 may be sufficient to cause translational movement of the mark within XR environment 128 without detecting touch input 754 directed toward touch-sensitive surface 175 of control device 130. In some implementations, the detection of control input 752 in conjunction with the detection of touch input 754 directed toward touch-sensitive surface 175 of control device 130 causes translational movement of the mark within XR environment 128. For example, the angle, directionality, displacement, etc. of the translational movement of mark 742 corresponds to the spatial parameters of control input 752 in FIG. 7E . In some implementations, the control input 752 may also cause a rotational movement of the mark 742 based on a rotation parameter of the control input 752 .

[0159] 7G and 7H illustrate a sequence in which detection of a first marking input causes one or more marks to be displayed within the XR environment 128 according to a first measurement of an input (pressure) value. As shown in FIG. 7G, during an instance 770 (e.g., associated with time T7) of the content distribution scenario, the electronic device 120 detects a marking input 772 by the control device 130 directed at an input area 774 on the table 107 via hand / limb tracking. For example, the input area 774 corresponds to a portion of a plane associated with the surface of the table 107. In some implementations, the input area 774 is visualized with an XR boundary or the like. In some implementations, the input area 774 is not visualized within the XR environment 128.

[0160] While the marking input 772 is being detected, the electronic device 120 also detects an input (pressure) value or obtains an indication of an input (pressure) value associated with how hard the control device 130 is being pressed against the table 107. As one example, the input (pressure) value is detected by one or more pressure sensors integrated into one of the tips of the control device 130. As another example, the input (pressure) value is detected by analyzing one or more images captured by an outward-facing image sensor of the electronic device 120 using computer vision techniques. As shown in FIG. 7G , the input (pressure) value indicator 777 indicates a current measurement 779 of the input (pressure) value associated with how hard the control device 130 is being pressed against the table 107. According to some implementations, the input (pressure) value indicator 777 is a diagram for guiding the reader that may or may not be displayed by the electronic device 120.

[0161] As shown in Figure 7H, during instance 780 of the content distribution scenario (e.g., associated with time T8), electronic device 120 displays mark 782B on XR substrate 718 within XR environment 128 and mark 782 on input area 774 in response to detecting marking input 715 in Figure 7G. For example, the shape, depth, length, angle, etc. of marks 782A and 782B correspond to spatial parameters (e.g., position value, rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc. associated with the marking input) of marking input 772 in Figure 7G. Additionally, in Figure 7H, marks 782A and 782B are associated with a first thickness value that corresponds to the current measured value 779 of the input (pressure) value in Figure 7G.

[0162] 7I and 7J illustrate a sequence in which detecting a second marking input causes one or more marks to be displayed within the XR environment 128 according to a second measurement of the input (pressure) value. As shown in FIG. 7I, during an instance 790 (e.g., associated with time T9) of the content distribution scenario, the electronic device 120 detects a marking input 792 by the control device 130 directed at an input area 774 on the table 107 via hand / limb tracking. While the marking input 792 is being detected, the electronic device 120 also detects an input (pressure) value or obtains an indication of an input (pressure) value associated with how hard the control device 130 is pressed against the table 107. As shown in FIG. 7I, the input (pressure) value indicator 777 indicates a current measurement 799 of the input (pressure) value associated with how hard the control device 130 is pressed against the table 107. For example, the current measurement 799 in FIG. 7I is greater than the measurement 779 in FIG. 7G.

[0163] As shown in FIG. 7J, in a content distribution scenario (e.g., at time T 10During instance 7100 (associated with ), electronic device 120 displays mark 7102A on XR substrate 718 and mark 7102B on input area 774 within XR environment 128 in response to detecting marking input 792 in FIG. 7I. For example, the shape, depth, length, angle, etc. of marks 7102A and 7102B correspond to the spatial parameters of marking input 792 in FIG. 7I (e.g., position value, rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc. associated with the marking input). Additionally, in FIG. 7J, marks 7102A and 7102B are associated with a second thickness value that corresponds to the current measured value 799 of the input (pressure) value in FIG. 7I. For example, the second thickness value associated with marks 7102A and 7102B is greater than the first thickness value associated with marks 782A and 782B.

[0164] 7K and 7L illustrate a sequence in which the detection of a first content placement input causes a first XR content to be displayed within the XR environment 128 according to the current measured input (pressure) value. As shown in FIG. 7K, at a time T 11 During instance 7110 (associated with), the electronic device 120 presents an XR environment 128 that includes a portion of the table 107, the VA 606, an XR board 7118 (e.g., a planar board), and a menu 7112. As shown in FIG. 7K, the menu 7112 includes multiple selectable options 7114 associated with changing the appearance (e.g., a different shape, color, texture, etc.) of the XR content placed within the XR environment. For example, option 7114A is currently selected from the multiple selectable options 7114. In this example, option 7114A corresponds to a first appearance of the XR content placed within the XR environment 128. As shown in FIG. 7A, the menu 7112 also includes a slider 7116 for adjusting the size of the XR content placed within the XR environment 128.

[0165] As shown in FIG. 7K, in a content distribution scenario (e.g., at time T 11During instance 7110 (associated with), electronic device 120 detects touch input 7111 directed toward touch-sensitive surface 175 of control device 130 or obtains an indication of touch input 7111 directed toward touch-sensitive surface 175 of control device 130 while representation 131 of control device 130 is at distance 7115 above XR substrate 7118. As one example, touch input 7111 is detected by touch-sensitive surface 175 of control device 130. As another example, touch input 7111 is detected by using computer vision techniques to analyze one or more images captured by an outward-facing image sensor of electronic device 120. For example, touch input 7111 corresponds to placing XR content (e.g., a cube) within XR environment 128. Those skilled in the art will understand that other XR content may be placed within XR environment 128 as well.

[0166] While the touch input 7111 is being detected, the electronic device 120 also detects an input (pressure) value or obtains an indication of an input (pressure) value associated with how tightly the control device 130 is being gripped by the left hand 150 of the user 149. As shown in FIG. 7K, the input (pressure) value indicator 717 indicates a current measurement 7119 of the input (pressure) value associated with how tightly the control device 130 is being gripped by the user 149. According to some implementations, the input (pressure) value indicator 717 is a diagram for guiding the reader that may or may not be displayed by the electronic device 120.

[0167] As shown in FIG. 7L, in a content distribution scenario (e.g., at time T 127K), electronic device 120 displays first XR content 7122 at a distance 7115 above XR substrate 7118 in XR environment 128 in response to detecting touch input 7111 in FIG. 7K. As shown in FIG. 7L, electronic device 120 also displays a shadow 7124 associated with first XR content 7122 on XR substrate 7118. For example, the position and rotation values ​​of first XR content 7122 and shadow 7124 correspond to the parameters (e.g., position value, rotation value, etc.) of representation 131 of control device 130 when touch input 7111 was detected in FIG. 7K. For example, first XR content 7122 is associated with a first size value that corresponds to the current measured value 7119 of input (pressure) value in FIG. 7K.

[0168] 7M and 7N illustrate a sequence in which the detection of a second content placement input causes second XR content to be displayed within the XR environment 128 according to the current measured input (pressure) value. As shown in FIG. 7M, at a time T 13 During instance 7130 (associated with), electronic device 120 detects touch input 7131 directed at touch-sensitive surface 175 of control device 130 or obtains an indication of touch input 7131 directed at touch-sensitive surface 175 of control device 130 while representation 131 of control device 130 is in contact with XR substrate 7118.

[0169] While the touch input 7131 is being detected, the electronic device 120 also detects an input (pressure) value or obtains an indication of an input (pressure) value associated with how tightly the control device 130 is being gripped by the left hand 150 of the user 149. As shown in FIG. 7M , the input (pressure) value indicator 717 indicates a current measurement 7139 of the input (pressure) value associated with how tightly the control device 130 is being gripped by the user 149. According to some implementations, the input (pressure) value indicator 717 is a diagram for guiding the reader that may or may not be displayed by the electronic device 120.

[0170] As shown in Figure 7N, during instance 7140 of the content delivery scenario (e.g., associated with time T14), electronic device 120 displays second XR content 7142 on XR board 7118 within XR environment 128 in response to detecting touch input 7131 in Figure 7M. As shown in Figure 7N, electronic device 120 does not display a shadow associated with second XR content 7142 on XR board 7118. For example, the position and rotation values ​​of second XR content 7142 correspond to the parameters (e.g., position value, rotation value, etc.) of representation 131 of control device 130 when touch input 7131 was detected in Figure 7M. For example, second XR content 7142 is associated with a second size value that corresponds to current measured input (pressure) value 7139 in Figure 7K. For example, the second size value associated with second XR content 7142 is greater than the first size value associated with first XR content 7122.

[0171] 8A-8M illustrate a sequence of instances 810-8130 for a content distribution scenario according to some implementations. While certain features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the implementations disclosed herein. To that end, by way of non-limiting example, the sequence of instances 810-8130 may be rendered and presented by a computing system such as the controller 110 shown in FIGS. 1 and 2, the electronic device 120 shown in FIGS. 1 and 3, and / or a suitable combination thereof.

[0172] 8A-8M, the content delivery scenario includes a physical environment 105 and an XR environment 128 displayed on a display 122 of an electronic device 120 (e.g., associated with a user 149). The electronic device 120 presents the XR environment 128 to the user 149 while the user 149 is physically present in the physical environment 105, including a door 115 that is currently within the FOV 111 of an outward-facing image sensor of the electronic device 120. Thus, in some implementations, the user 149 holds the electronic device 120 in their left hand 150 or right hand 152.

[0173] In other words, in some implementations, electronic device 120 is configured to present XR content and enable optical see-through or video pass-through of at least a portion of physical environment 105 on display 122 (e.g., door 115 or a representation thereof). For example, electronic device 120 may correspond to a mobile phone, a tablet, a laptop, a near-eye system, a wearable computing device, etc.

[0174] As shown in FIG. 8A , during an instance 810 (e.g., associated with time T1) of the content distribution scenario, the electronic device 120 presents an XR environment 128 that includes a representation 116 of a door 115 in the physical environment 105, a virtual agent (VA) 606, and XR content 802 (e.g., a cylinder). As shown in FIG. 8A , the electronic device 120 also displays a gaze direction 806 associated with the focal point of the user's 149's eye in the XR environment 128 based on eye tracking. In some implementations, the gaze direction 806 is not displayed or visualized. As shown in FIG. 8A , the electronic device 120 also displays a representation 153 of the user's 149's right hand 152 holding a representation 805 of a proxy object 804 (e.g., a stick, a ruler, or another physical object) in the XR environment 128. For example, the user's 149's right hand 152 is currently holding the proxy object 804 in a pointing grip pose.

[0175] 8A and 8B illustrate a sequence in which the size of an indicator element changes based on distance. As shown in FIG. 8A, the representation 805 of the proxy object 804 is at a first distance 814 from the XR content 802, and the electronic device 120 displays a first indicator element 812A on the XR content 802, the first indicator element 812A having a first size corresponding to a coincidence between the XR content 802 and a ray of light emanating from the tip / end of the representation 805 of the proxy object 804. In some implementations, the size of the first indicator element 812A is a function of the first distance 814. For example, the size of the indicator element increases as the distance decreases, and the size of the indicator element decreases as the distance increases.

[0176] As shown in FIG. 8B, during instance 820 (e.g., associated with time T2) of the content delivery scenario, electronic device 120 displays second indicator element 812B on XR content 802, having a second size corresponding to a point of coincidence between the XR content 802 and a ray of light emanating from the tip / end of representation 805 of proxy object 804. As shown in FIG. 8B, representation 805 of proxy object 804 is at a second distance 824 from XR content 802, which is less than first distance 814 in FIG. 8A. For example, second size of second indicator element 812B is greater than first size of first indicator element 812A.

[0177] 8B-8D show a sequence in which XR content is selected in response to detecting that a proxy object is pointed at the XR content and the XR content is translated in response to translating the proxy object. In some implementations, electronic device 120 selects XR content 802 and changes its appearance in response to detecting that a proxy object 804 is pointed at the XR content 802 for at least a predetermined period of time. In some implementations, electronic device 120 selects XR content 802 and changes its appearance in response to detecting that a proxy object 804 is pointed at the XR content 802 for at least a deterministic period of time.

[0178] As shown in Figure 8C, during instance 830 (e.g., associated with time T3) of the content delivery scenario, in response to detecting that proxy object 804 has been pointed at XR content 802 for at least the predetermined or deterministic period of time in Figures 8A and 8B, electronic device 120 changes the appearance of XR content 802 to a cross-hatched appearance 802A to visually indicate its selection. As shown in Figure 8C, electronic device 120 also detects translational movement 832 of proxy object 804 while XR content 802A is selected and representation 805 of proxy object 804 remains pointed at XR content 802A.

[0179] As shown in Figure 8D, during instance 840 (e.g., associated with time T4) of the content distribution scenario, electronic device 120 translates XR content 802 within XR environment 128 in response to detecting translational movement 832 of proxy object 804 in Figure 8C. For example, the direction and displacement of the translational movement of XR content 802 within XR environment 128 corresponds to the spatial parameters (e.g., change in position value, change in rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc.) of translational movement 832 in Figure 8C. Those skilled in the art will appreciate that XR content 802 may be similarly rotated.

[0180] 8E-8G show a sequence in which XR content is selected in response to detecting a gaze direction toward the XR content and the XR content is transformed based on a translation of the gaze direction. In some implementations, electronic device 120 selects XR content 802 and changes its appearance in response to detecting that gaze direction 806 is toward the XR content 802 for at least a predetermined period of time. In some implementations, electronic device 120 selects XR content 802 and changes its appearance in response to detecting that gaze direction 806 is toward the XR content 802 for at least a deterministic period of time.

[0181] 8E , during instance 850 (e.g., associated with time T5) of the content delivery scenario, the electronic device 120 displays, based on eye tracking, a gaze direction indicator element 852 on the XR content 802 associated with the focal point in the XR environment 128 of the eye of the user 149. For example, the gaze direction indicator element 852 corresponds to a point of coincidence between the XR content 802 and a ray of light emanating from the eye of the user 149. As shown in FIG. 8E , the electronic device 120 also displays a representation 153 of the right hand 152 of the user 149 holding a representation 131 of the control device 130 in the XR environment 128. For example, the right hand 152 of the user 149 is currently holding the control device 130 in a writing grip pose that is not directed toward any XR content in the XR environment 128.

[0182] As shown in Figure 8F, during instance 860 (e.g., associated with time T6) of the content delivery scenario, electronic device 120, in response to detecting gaze direction 806 being directed toward XR content 802 for at least the predetermined or deterministic period in Figure 8E, changes the appearance of XR content 802 to a cross-hatched appearance 802A to visually indicate its selection. As shown in Figure 8F, electronic device 120 also detects a translational movement 862 of gaze direction 806 while XR content 802A is selected.

[0183] As shown in Figure 8G, during instance 870 (e.g., associated with time T7) of the content distribution scenario, electronic device 120 translates XR content 802 within XR environment 128 in response to detecting translation 862 of gaze direction 806 in Figure 8F. For example, the direction and displacement of the translation of XR content 802 within XR environment 128 corresponds to spatial parameters (e.g., change in position value, change in rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc.) of translation 862 of gaze direction 806 in Figure 8F.

[0184] 8H-8J illustrate another sequence in which XR content is selected in response to detecting a gaze direction directed toward the XR content and the XR content is translated based on a translation of the gaze direction. As shown in FIG. 8H, during instance 880 (e.g., associated with time T8) of the content distribution scenario, electronic device 120 displays gaze direction indicator element 852 on XR content 802 associated with a focal point in XR environment 128 of user 149's eye based on gaze tracking. For example, gaze direction indicator element 852 corresponds to a coincidence point between XR content 802 and a ray of light emanating from user 149's eye. As shown in FIG. 8H, electronic device 120 detects that neither proxy object 804 nor control device 130 is being held by user 149.

[0185] As shown in Figure 8I, during instance 890 (e.g., associated with time T9) of the content delivery scenario, electronic device 120, in response to detecting gaze direction 806 being directed toward XR content 802 for at least the predetermined or deterministic period in Figure 8H, changes the appearance of XR content 802 to a cross-hatched appearance 802A to visually indicate its selection. As shown in Figure 8I, electronic device 120 also detects a translational movement 892 of gaze direction 806 while XR content 802A is selected.

[0186] As shown in FIG. 8J, in a content distribution scenario (e.g., at time T10 During instance 8100 (associated with ), electronic device 120 translates mark 802 in XR environment 128 in response to detecting translation 892 of gaze direction 806 in FIG. 8I. For example, the direction and displacement of the translation of XR content 802 in XR environment 128 corresponds to spatial parameters (e.g., change in position value, change in rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc.) of translation 892 of gaze direction 806 in FIG. 8I.

[0187] 8K-8M illustrate a sequence in which XR content is selected in response to detecting a gaze direction directed toward the XR content, and the XR content is translated based on hand / limb tracking input. As shown in FIG. 8K, the XR content is selected in response to detecting a gaze direction directed toward the XR content, and the XR content is translated based on hand / limb tracking input. 11 During instance 8110 (associated with), the electronic device 120 displays, based on eye tracking, a gaze direction indicator element 852 on the XR content 802 associated with the focal point in the XR environment 128 of the eye of the user 149. For example, the gaze direction indicator element 852 corresponds to a point of coincidence between the XR content 802 and a ray of light emanating from the eye of the user 149. As shown in FIG. 8K, the electronic device 120 detects that neither the proxy object 804 nor the control device 130 is being held by the user 149.

[0188] As shown in FIG. 8L, in a content distribution scenario (e.g., at time T 12 During instance 8120 (associated with selection), in response to detecting gaze direction 806 directed toward XR content 802 for at least the predetermined or deterministic period in FIG. 8K, electronic device 120 changes the appearance of XR content 802 to a cross-hatched appearance 802A to visually indicate its selection. As shown in FIG. 8L, electronic device 120 displays a representation 153 of user 149's right hand 152 near XR content 802A, which is detected and tracked using hand / limb tracking. As shown in FIG. 8L, electronic device 120 also detects translational movement 8122 of user 149's right hand 152.

[0189] As shown in FIG. 8M, in a content distribution scenario (e.g., at time T 13 During instance 8130 (associated with instance 8130), electronic device 120 translates XR content 802 within XR environment 128 in response to detecting translational movement 8122 in Figure 8L. For example, the direction and displacement of the translational movement of XR content 802 within XR environment 128 corresponds to the spatial parameters (e.g., change in position value, change in rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc.) of translational movement 8122 in Figure 8L. Those skilled in the art will appreciate that other XR content may be similarly rotated.

[0190] 9A-9C show a flowchart representation of a method 900 for selecting an output modality of a physical object when interacting with or manipulating an XR environment according to some implementations. In various implementations, the method 900 is performed in a computing system including non-transitory memory and one or more processors, the computing system being communicatively coupled to a display device and (optionally) one or more input devices (e.g., the electronic device 120 shown in FIGS. 1 and 3 , the controller 110 of FIGS. 1 and 2 , or a suitable combination thereof). In some implementations, the method 900 is performed by processing logic, the processing logic including hardware, firmware, software, or a combination thereof. In some implementations, the method 900 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In some implementations, the computing system corresponds to one of a tablet, a laptop, a mobile phone, a near-eye system, a wearable computing device, etc.

[0191] Typically, a user switches marking tools by selecting a new tool from a toolbar or menu. This interrupts the user's current workflow and may lead the user to search for a new tool for the appropriate control. In contrast, the method described herein allows a user to invoke a display of a toolset by swiping on a physical object (e.g., a proxy object such as a pencil, or an electronic device such as a stylus) and moving the physical object toward a graphical representation of one of the tools in the toolset. Thus, a user can switch tools without interrupting their workflow.

[0192] As represented by block 902, method 900 includes displaying, via a display device, a first plurality of graphical elements associated with a first plurality of output modalities within an extended reality (XR) environment. In some implementations, the output modalities cause changes within the UI or XR environment, such as adding, removing, or otherwise modifying pixels within the UI or XR environment. For example, the first plurality of graphical elements correspond to different tool types, such as pencils, markers, paintbrushes, erasers, etc., for making, modifying, etc., marks within the XR environment.

[0193] 6C , electronic device 120 displays graphical elements 632A, 632B, 632C, and 632D (sometimes collectively referred to herein as a first plurality of graphical elements 632). In this example, graphical element 632A corresponds to an output modality associated with generating pencil-like marks in XR environment 128, graphical element 632B corresponds to an output modality associated with generating pen-like marks in XR environment 128, graphical element 632C corresponds to an output modality associated with generating marker-like marks in XR environment 128, and graphical element 632D corresponds to an output modality associated with generating airbrush-like marks in XR environment 128. As another example, electronic device 120 displays graphical elements 6152A, 6152B, 6152C, and 6152D (sometimes collectively referred to herein as a second plurality of graphical elements 6152) in FIG. In this example, graphical element 6152A corresponds to an output modality associated with generating pencil-like marks within the XR environment 128, graphical element 6152B corresponds to an output modality associated with generating pen-like marks within the XR environment 128, graphical element 6152C corresponds to an output modality associated with generating thin brush-like marks within the XR environment 128, and graphical element 6152D corresponds to an output modality associated with generating thick brush-like marks within the XR environment 128.

[0194] In some implementations, the display device corresponds to a transparent lens assembly, and the representation of the XR environment is projected onto the transparent lens assembly. In some implementations, the display device corresponds to a near-eye system, and the presenting of the XR environment includes combining the representation of the XR environment with one or more images of the physical environment captured by an outward-facing image sensor.

[0195] In some implementations, method 900 includes acquiring (e.g., receiving, retrieving, or detecting) an indication of touch input directed at the physical object before displaying the first plurality of graphical elements, and displaying the first plurality of graphical elements in the XR environment includes displaying the first plurality of graphical elements in the XR environment in response to acquiring the indication of the touch input. In some implementations, the physical object corresponds to a stylus having a touch-sensitive area capable of detecting the touch input. For example, the stylus detects an upward or downward swipe gesture on its touch-sensitive surface, and the computing system acquires (e.g., receives or retrieves) the indication of the touch input from the stylus. For example, FIGS. 6B and 6C show a sequence in which electronic device 120 displays first plurality of graphical elements 632 associated with first plurality of output modalities in XR environment 128 of FIG. 6C in response to detecting touch input 622 directed at control device 130 of FIG. 6B.

[0196] In some implementations, the method 900 includes acquiring (e.g., receiving, retrieving, or determining) a grip posture associated with a current manner in which a physical object is being held by a user before displaying a first plurality of graphical elements, the first plurality of graphical elements being a function of the grip posture; and, in response to acquiring the grip posture, displaying, via the display device, a first plurality of graphical elements associated with a first plurality of output modalities within the XR environment in accordance with a determination that the grip posture corresponds to the first grip posture; and displaying, via the display device, a second plurality of graphical elements associated with a second plurality of output modalities within the XR environment in accordance with a determination that the grip posture corresponds to a second grip posture different from the first grip posture. For example, a pointing / wand grip corresponds to a first plurality of graphical elements associated with a first plurality of tools, and a writing grip corresponds to a second plurality of graphical elements associated with a second plurality of tools. In some implementations, the first and second plurality of output modalities include at least one overlapping output modality. In some implementations, the first and second plurality of output modalities include mutually exclusive output modalities.

[0197] 6C , electronic device 120 displays graphical element 632 within XR environment 128 pursuant to a determination that the current grip posture corresponds to a pointing grip posture. As another example, referring to FIG. 6O , electronic device 120 displays graphical element 6152 within XR environment 128 pursuant to a determination that the current grip posture corresponds to a writing grip posture in which first end 176 is pointed downward and second end 177 is pointed upward. As yet another example, referring to FIG. 6P , electronic device 120 displays graphical element 6162 within XR environment 128 pursuant to a determination that the current grip posture corresponds to a reverse writing grip posture in which first end 176 is pointed upward and second end 177 is pointed downward.

[0198] In some implementations, the method 900 includes detecting a change in grip posture from a first grip posture to a second grip posture after displaying a first plurality of graphical elements associated with a first plurality of output modalities within the XR environment, and replacing the display of the first plurality of graphical elements within the XR environment with a second plurality of graphical elements associated with a second plurality of output modalities within the XR environment in response to detecting the change in grip posture. In some implementations, the computing system also stops displaying the first plurality of graphical elements. For example, a pointing / wand grip corresponds to the first plurality of graphical elements associated with the first plurality of output modalities, and a writing grip corresponds to the second plurality of graphical elements associated with the second plurality of output modalities. In some implementations, the first and second plurality of graphical elements include at least some overlapping output modalities. In some implementations, the first and second plurality of graphical elements include mutually exclusive output modalities. For example, Figures 6O and 6P show a sequence in which the electronic device 120 replaces multiple graphical elements 6152 with multiple graphical elements 6162 in response to detecting a change in the current grip posture of the control device 130 (e.g., a change from the writing grip posture of Figure 6O to the reverse writing grip posture of Figure 6P).

[0199] In some implementations, method 900 includes, before displaying the first plurality of graphical elements, obtaining (e.g., receiving, retrieving, or determining) information indicating whether a first end or a second end of the physical object is outward-facing (e.g., outward-facing toward a surface, a user, a computing system, etc.); and, in response to obtaining the information indicating whether the first end or the second end of the physical object is outward-facing, displaying, via the display device, a first plurality of graphical elements associated with a first plurality of output modalities within the XR environment in accordance with a determination that the first end of the physical object is outward-facing; and displaying, via the display device, a second plurality of graphical elements associated with a second plurality of output modalities within the XR environment in accordance with a determination that the second end of the physical object is outward-facing. For example, the outward-facing first end corresponds to a first plurality of graphical elements associated with a first plurality of output modalities (e.g., a sketch tool and a writing tool), and the outward-facing second end corresponds to a second plurality of graphical elements associated with a second plurality of output modalities (e.g., an erase tool or an edit tool). In some implementations, the first and second plurality of output modalities include at least one overlapping output modality. In some implementations, the first and second plurality of output modalities include mutually exclusive output modalities.

[0200] 6O, electronic device 120 displays graphical element 6152 in XR environment 128 pursuant to a determination that the current grip posture corresponds to a writing grip posture in which first end 176 is pointed downward and second end 177 is pointed upward. As yet another example, referring to FIG. 6P, electronic device 120 displays graphical element 6162 in XR environment 128 pursuant to a determination that the current grip posture corresponds to a reverse writing grip posture in which first end 176 is pointed upward and second end 177 is pointed downward.

[0201] In some implementations, the method 900 includes, after displaying a first plurality of graphical elements associated with a first plurality of output modalities within the XR environment, detecting a change from an outward-facing first end of the physics object to an outward-facing second end of the physics object, and displaying, in response to detecting the change from the outward-facing first end of the physics object to the outward-facing second end of the physics object within the XR environment, a second plurality of graphical elements associated with a second plurality of output modalities via the display device. In some implementations, the computing system also stops displaying the first plurality of graphical elements. For example, the outward-facing first end corresponds to a first plurality of graphical elements associated with the first plurality of output modality tools (e.g., a sketch tool and a writing tool), and the outward-facing second end corresponds to a second plurality of graphical elements associated with the second plurality of output modalities (e.g., an erase tool or an editing tool). For example, Figures 6O and 6P show a sequence in which the electronic device 120 replaces the display of multiple graphical elements 6152 with multiple graphical elements 6162 in response to detecting a change in the current grip posture of the control device 130 (e.g., a change from the writing grip posture in Figure 6O to the reverse writing grip posture in Figure 6P).

[0202] As represented by block 904, method 900 includes detecting a first movement of a physical object while displaying the first plurality of graphical elements. In some implementations, the computing system obtains (e.g., receives, retrieves, or determines) translation and rotation values ​​of the physical object, and detecting the first movement corresponds to detecting a change to one of the translation or rotation values ​​of the physical object. For example, the computing system tracks the physical object via computer vision, a magnetic sensor, position information, or the like. As one example, the physical object corresponds to a proxy object, such as a pencil or pen, without a communication channel to the computing system. As another example, the physical object corresponds to an electronic device, such as a stylus or finger-worn device, with a wired or wireless communication channel to the computing system that includes an IMU, accelerometer, gyroscope, magnetometer, or the like for six degrees of freedom (6DOF) tracking.

[0203] In some implementations, the computing system maintains one or more N-tuple tracking vectors / tensors for the physics object (e.g., object tracking vector 511 in FIGS. 5A and 5B ) based on the tracking data 506. In some implementations, the one or more N-tuple tracking vectors / tensors for the physics object (e.g., object tracking vector 511 in FIGS. 5A and 5B ) include translation values ​​(e.g., x, y, and z) of the physics object relative to the world at large or the current operating environment, rotation values ​​(e.g., roll, pitch, and yaw) of the physics object, a grip pose indication for the physics object (e.g., pointing, writing, erasing, painting, dictating, etc.), a currently used tip / end indication (e.g., a physics object can have an asymmetric design with specific first and second tips or a symmetric design with non-specific first and second tips), a first input (pressure) value related to how hard the physics object is pressed against a physical surface, a second input (pressure) value related to how hard the physics object is being gripped by the user, touch input information, etc.

[0204] In some implementations, the tracking data 506 corresponds to one or more images of the physical environment including the physical object to enable 6DOF tracking via computer vision techniques. In some implementations, the tracking data 506 corresponds to data collected by various integrated sensors of the physical object, such as an IMU, an accelerometer, a gyroscope, a magnetometer, etc. For example, the tracking data 506 corresponds to raw sensor data or processed data, such as a translation value associated with the physical object (with respect to the physical environment or world at large), a rotation value associated with the physical object (with respect to gravity), a velocity value associated with the physical object, an angular velocity value associated with the physical object, an acceleration value associated with the physical object, an angular acceleration value associated with the physical object, a first input (pressure) value associated with how tightly the physical object is touching a physical surface, a second input (pressure) value associated with how tightly the physical object is being gripped or touched by a user on the physical surface, etc.

[0205] In some implementations, the computing system also acquires finger manipulation data detected by the physical object via the communication interface. For example, the finger manipulation data includes touch input or gestures directed at a touch-sensitive area of ​​the physical object, etc. For example, the finger manipulation data includes contact intensity data with respect to a body of the physical object. In some implementations, the physical object includes a touch-sensitive surface / area configured to detect touch input directed at the physical object, such as a vertically extending touch-sensitive surface. In some implementations, determining the translation and rotation values ​​for the physical object includes determining the translation and rotation values ​​for the physical object based on at least one of IMU data from the physical object, one or more images of physical environment 105 including the physical object, magnetic tracking data, etc.

[0206] In some implementations, the computing system is further communicatively coupled to the physical object, and acquiring tracking data 506 associated with the physical object includes acquiring tracking data 506 from the physical object, the tracking data corresponding to output data from one or more integrated sensors of the physical object. 6C-6P show user 149 holding control device 130, which is used to communicate with electronic device 120 and interact with XR environment 128. For example, the one or more integrated sensors include at least one of an IMU, an accelerometer, a gyroscope, a GPS, a magnetometer, one or more contact intensity sensors, a touch-sensitive surface, and / or the like. In some implementations, tracking data 506 further indicates whether a tip of the physical object touches a physical surface and a pressure value associated therewith.

[0207] In some implementations, the method 900 includes acquiring one or more images of a physical environment, recognizing a physical object using the one or more images of the physical environment, and assigning a physical object (e.g., a proxy object) to function as a focus selection when interacting with the XR environment 128. FIGS. 8A-8D show a user 149 unable to communicate with the electronic device 120 and holding a proxy object 804 (e.g., a ruler, a stick, etc.) used to interact with the XR environment 128. In some implementations, the computing system designates the physical object as the focus selection when the physical object is held by the user. In some implementations, the computing system designates the physical object as the focus selection when the physical object is held by the user and the physical object meets predefined constraints (e.g., a maximum or minimum size, a specific shape, a digital rights management (DRM) non-qualifier, etc.). Thus, in some implementations, the user 149 can interact with the XR environment using a household object. In some implementations, the pose and grip indicators may be fixed to the proxy object (or a representation thereof) as the proxy object moves and / or the FOV moves.

[0208] As represented by block 906, in response to detecting a first movement of the physics object and following a determination that the first movement of the physics object has caused the physics object (e.g., a predetermined portion of the physics object, such as a tip of the physics object) to exceed a distance threshold for a first graphical element of the first plurality of graphical elements, method 900 includes selecting a first output modality associated with the first graphical element as a current output modality for the physics object. In some implementations, the distance threshold is non-deterministic (i.e., a predetermined X mm radius) or deterministic based on one or more factors, such as user preference, tool usage history, depth of the graphical element relative to the scene, occlusion, current content, current context, etc.

[0209] According to some implementations, the computing system or a component thereof (e.g., output modality selector 526 of FIG. 5A ) selects the associated first output modality as the current output modality 527 for the physical object pursuant to a determination that a first movement of the physical object causes the physical object to exceed a distance threshold for a first graphical element of the first plurality of graphical elements. As an example, FIGS. 6D and 6E illustrate a sequence in which electronic device 120 selects a first output modality (e.g., airbrush marking) for control device 130 pursuant to a determination that a movement of control device 130 causes control device 130 (or a representation thereof) to exceed an activation area 634 (e.g., a distance threshold) for graphical element 632D.

[0210] 6D and 6E , the electronic device 120 maintains display of a first graphical element (e.g., graphical element 632D) of the first plurality of graphical elements (e.g., graphical element 632) superimposed on an upper portion of the representation 131 of the control device 130 in the XR environment 128, and removes display of the remainder of the first plurality of graphical elements (e.g., graphical elements 632A, 632B, and 632C) from the XR environment 128. In some implementations, following a determination that the first movement of the physical object causes the physical object to exceed a distance threshold for a first graphical element of the first plurality of graphical elements, the method 900 includes maintaining display of the first graphical element adjacent to the physical object and ceasing display of the remainder of the first plurality of graphical elements that do not include the first graphical element. As an example, referring to FIG. 6D and FIG. 6E , the electronic device 120 maintains display of a first graphical element (e.g., graphical element 632D) of the first plurality of graphical elements (e.g., graphical element 632) superimposed on an upper portion of the representation 131 of the control device 130 in the XR environment 128, and removes display of the remainder of the first plurality of graphical elements (e.g., graphical elements 632A, 632B, and 632C) from the XR environment 128.

[0211] In some implementations, the method 900 includes detecting a second movement of the physical object after selecting a first output modality associated with the first graphical element as the current output modality of the physical object, and, in response to detecting the second movement of the physical object, moving the first graphical element based on the second movement of the physical object to maintain display of the first graphical element adjacent to the physical object. In some implementations, the first graphical element is anchored to an outward-facing edge / tip of the physical object. In some implementations, the first graphical element is presented offset from or to a side of the outward-facing edge / tip of the physical object. In some implementations, the first graphical element “snaps” to the representation of the physical object. As an example, referring to FIGS. 6E and 6F , the electronic device 120 maintains display of a graphical element 632D superimposed on a tip of the representation 131 of the control device 130 in the XR environment 128 after detecting movement of the control device to perform the marking input 654.

[0212] In some implementations, method 900 includes acquiring (e.g., receiving, retrieving, or detecting) an indication of touch input directed at the physical object after ceasing display of the remainder of the first plurality of graphical elements, and re-displaying the first plurality of graphical elements within the XR environment via the display device in response to acquiring the indication of the touch input. In some implementations, the physical object corresponds to a stylus having a touch-sensitive area capable of detecting the touch input. For example, the stylus detects an upward or downward swipe gesture on its touch-sensitive surface, and the computing system acquires (e.g., receives or retrieves) the indication of the touch input from the stylus. As an example, referring to FIGS. 6N and 6O, electronic device 120 displays a second plurality of graphical elements 6152 associated with a second plurality of output modalities within XR environment 128 in response to detecting touch input 6142 directed at control device 130.

[0213] As represented by block 908, in response to detecting a first movement of the physics object and following a determination that the first movement of the physics object has caused the physics object to exceed a distance threshold for a second graphical element of the first plurality of graphical elements, method 900 includes selecting a second output modality associated with the second graphical element as a current output modality for the physics object.

[0214] According to some implementations, the computing system or a component thereof (e.g., output modality selector 526 of FIG. 5A ) selects the associated second output modality as the physical object's current output modality 527 pursuant to a determination that a first movement of the physical object causes the physical object to exceed a distance threshold for a second graphical element of the first plurality of graphical elements. As an example, FIGS. 6G-6I illustrate a sequence in which electronic device 120 selects a second output modality (e.g., pen marking) for control device 130 pursuant to a determination that a movement of control device 130 causes control device 130 (or a representation thereof) to exceed activation area 634 (e.g., a distance threshold) for graphical element 632B.

[0215] In some implementations, pursuant to determining that the first movement of the physical object causes the physical object to exceed a distance threshold relative to a second graphical element of the first plurality of graphical elements, method 900 includes maintaining display of the second graphical element adjacent to the physical object and ceasing display of the remainder of the first plurality of graphical elements that do not include the second graphical element. As an example, referring to FIGS. 6H and 6I , electronic device 120 maintains display of a second graphical element (e.g., graphical element 632B) of the first plurality of graphical elements (e.g., graphical element 632) overlaid on an end portion of representation 131 of control device 130 in XR environment 128, and removes display of the remainder of the first plurality of graphical elements (e.g., graphical elements 632A, 632C, and 632D) from XR environment 128.

[0216] In some implementations, the first and second output modalities cause different visual changes within the XR environment, as represented by block 910. For example, the first output modality is associated with selecting / manipulating objects / content within the XR environment, and the second output modality is associated with sketching, drawing, writing, etc. within the XR environment. As an example, referring to FIG. 6F , electronic device 120 displays airbrush-like mark 662 within XR environment 128 in response to detecting marking input 654 in FIG. 6E while the current output modality corresponds to graphical element 632D. As another example, referring to FIG. 6J , electronic device 120 displays pen-like mark 6112 within XR environment 128 in response to detecting marking input 6104 in FIG. 6J while the current output modality corresponds to graphical element 632B.

[0217] In some implementations, following a determination that the first movement of the physical object does not cause the physical object to exceed a distance threshold for the first graphical element or the second graphical element, method 900 includes maintaining the initial output modality as a current output modality of the physical object and maintaining the display of the first plurality of graphical elements. As an example, referring to FIG. 6C , electronic device 120 maintains the initial output modality as the current output modality of control device 130 while representation 131 of control device 130 is outside activation region 634. As another example, referring to FIG. 6G , electronic device 120 maintains the initial output modality as the current output modality of control device 130 while representation 131 of control device 130 is outside activation region 634.

[0218] In some implementations, as represented by block 912, method 900 includes detecting a subsequent marking input with the physical object after selecting a first output modality associated with the first graphical element as the current output modality of the physical object, and displaying one or more marks within the XR environment via the display device based on the subsequent marking input (e.g., a shape, a displacement, etc. of the subsequent marking input) and the first output modality in response to detecting the subsequent marking input. According to some implementations, the computing system detects the subsequent marking input with the physical object by tracking the physical object in 3D using IMU data, computer vision, magnetic tracking, etc. In some implementations, the one or more marks correspond to XR content displayed within XR environment 128, such as a sketch, handwritten text, doodles, etc. As an example, referring to FIG. 6F, electronic device 120 displays airbrush-like mark 662 within XR environment 128 in response to detecting marking input 654 in FIG. 6E while the current output modality corresponds to graphical element 632D. For example, the shape, depth, length, angle, etc. of the airbrush-like mark 662 correspond to the spatial parameters of the marking input 654 (e.g., position values, rotation values, displacements, spatial accelerations, spatial velocities, angular accelerations, angular velocities, etc. associated with the marking input).

[0219] In some implementations, as represented by block 914, in response to detecting the subsequent marking input, method 900 includes: in accordance with a determination that the input associated with how hard the physical object is pressed against the physical surface corresponds to a first input value, displaying, via the display device, one or more marks having a first appearance based on the subsequent marking input (e.g., a shape, a displacement, etc. of the subsequent marking input) and a first output modality within the XR environment, wherein the first appearance is associated with parameters of the one or more marks corresponding to the first input value; and in accordance with a determination that the input associated with how hard the physical object is pressed against the physical surface corresponds to a second input value, displaying, via the display device, one or more marks having a second appearance based on the subsequent marking input (e.g., a shape, a displacement, etc. of the subsequent marking input) and the first output modality within the XR environment, wherein the second appearance is associated with parameters of the one or more marks corresponding to the second input value.

[0220] In some implementations, the one or more marks correspond to XR content displayed within the XR environment 128, such as sketches, handwritten text, doodles, etc. In some implementations, the computing system obtains (e.g., receives, retrieves, or determines) the first and second input (pressure) values ​​based on locally or remotely collected data. As an example, the physical object corresponds to an electronic device having a pressure sensor at one or both of its ends / tip to detect the input (pressure) value when pressed against a physical surface. In some implementations, as represented by block 916, the parameter corresponds to one of a radius, width, thickness, intensity, translucency, opacity, color, or texture of the one or more marks in the XR environment.

[0221] 7H , in response to detecting marking input 715 in FIG. 7G , electronic device 120 displays mark 782A on XR substrate 718 within XR environment 128 and displays mark 782B on input area 774. For example, the shape, depth, length, angle, etc. of marks 782A and 782B correspond to the spatial parameters (e.g., position value, rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc. associated with the marking input) of marking input 772 in FIG. 7G . Additionally, in FIG. 7H , marks 782A and 782B are associated with a first thickness value that corresponds to the current measured value 779 of the input (pressure) value in FIG. 7G .

[0222] 7J , in response to detecting marking input 792 in FIG. 7I , electronic device 120 displays mark 7102A on XR substrate 718 within XR environment 128 and displays mark 7102B in input area 774. For example, the shape, depth, length, angle, etc. of marks 7102A and 7102B correspond to the spatial parameters of marking input 792 in FIG. 7I (e.g., the position value, rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc. associated with the marking input). Further, in FIG. 7J , marks 7102A and 7102B are associated with a second thickness value that corresponds to the current measured value 799 of the input (pressure) value in FIG. 7I . For example, the second thickness value associated with marks 7102A and 7102B is greater than the first thickness value associated with marks 782A and 782B.

[0223] In some implementations, as represented by block 918, in response to detecting the subsequent marking input, method 900 includes: in accordance with a determination that the input associated with how tightly the physical object is gripped by the user corresponds to a first input value, displaying, via the display device, one or more marks having a first appearance based on the subsequent marking input (e.g., a shape, a displacement, etc. of the subsequent marking input) and a first output modality within the XR environment, wherein the first appearance is associated with parameters of the one or more marks corresponding to the first input value; and in accordance with a determination that the input associated with how tightly the physical object is gripped by the user corresponds to a second input value, displaying, via the display device, one or more marks having a second appearance based on the subsequent marking input (e.g., a shape, a displacement, etc. of the subsequent marking input) and the first output modality within the XR environment, wherein the second appearance is associated with parameters of the one or more marks corresponding to the second input value.

[0224] In some implementations, the one or more marks correspond to XR content displayed within the XR environment 128, such as a sketch, handwritten text, or doodles. In some implementations, the computing system obtains (e.g., receives, retrieves, or determines) the first and second input (pressure) values ​​based on locally or remotely collected data. As an example, the physical object corresponds to an electronic device having a built-in pressure sensor for detecting the input (pressure) values ​​when grasped by a user. In some implementations, as represented by block 920, the parameter corresponds to one of a radius, width, thickness, intensity, translucency, opacity, color, or texture of the mark in the XR environment.

[0225] 6J , while the current output modality corresponds to the graphical element 632B, the electronic device 120 displays a pen-like mark 6112 within the XR environment 128 in response to detecting the marking input 6104 in FIG. 6J . For example, the shape, depth, length, angle, etc. of the pen-like mark 6112 correspond to the spatial parameters of the marking input 6104. Further, in FIG. 6K , the pen-like mark 6112 is associated with a first thickness value that corresponds to the current measured value of the input (pressure) value 6103 in FIG. 6J .

[0226] In another example, referring to Figure 6M, the electronic device 120 displays a pen-like mark 6132 within the XR environment 128 in response to detecting the marking input 6122 in Figure 6L. For example, the shape, depth, length, angle, etc. of the pen-like mark 6132 correspond to spatial parameters of the marking input 6122. Further, in Figure 6M, the pen-like mark 6132 is associated with a second thickness value that corresponds to the current measured value of the input (pressure) value 6123 in Figure 6L. The second thickness value associated with the pen-like mark 6132 in Figure 6M is greater than the first thickness value associated with the pen-like mark 6112 in Figure 6K.

[0227] 10A and 10B show a flowchart representation of a method 1000 for modifying parameters of a mark based on a first input (pressure) value during direct marking on a physical surface or based on a second input (pressure) value during indirect marking, according to some implementations. In various implementations, the method 1000 is performed in a computing system including non-transitory memory and one or more processors, the computing system being communicatively coupled to a display device and (optionally) one or more input devices (e.g., the electronic device 120 shown in FIGS. 1 and 3 , the controller 110 of FIGS. 1 and 2 , or a suitable combination thereof). In some implementations, the method 1000 is performed by processing logic, the processing logic including hardware, firmware, software, or a combination thereof. In some implementations, the method 1000 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). In some implementations, the computing system corresponds to one of a tablet, a laptop, a mobile phone, a near-eye system, a wearable computing device, etc.

[0228] Typically, a user can adjust marking parameters, such as line thickness, by moving a slider or the like in a toolbar or control panel. This can interrupt the user's current workflow and lead the user to search through various menus for the appropriate control. In contrast, the method described herein adjusts the marking parameters based on a first input (pressure) value between a physical object (e.g., a proxy object or stylus) and the physical surface when the marking input is directed at the physical surface, or based on a second input (pressure) value associated with the user's grip on the physical object. Thus, the user can adjust the marking parameters more quickly and efficiently.

[0229] As represented by block 1002, method 1000 includes displaying a user interface via a display device. In some implementations, the user interface includes a two-dimensional marking area in which marks are displayed (1004). (e.g., a flat canvas) In some implementations, the user interface includes a three-dimensional marking area in which marks are displayed (1006) (e.g., the marks are associated with a 3D painting or drawing). As an example, referring to FIG. 7A , electronic device 120 presents XR environment 128 including XR substrate 718 (e.g., a 2D or 3D canvas).

[0230] In some implementations, the display device corresponds to a transparent lens assembly, and the presentation of the user interface is projected onto the transparent lens assembly. In some implementations, the display device corresponds to a near-eye system, and the presentation of the user interface includes combining the presentation of the user interface with one or more images of the physical environment captured by an outward-facing image sensor.

[0231] In some implementations, the method 1000 includes displaying, via a display device, a user interface element (e.g., a toolbar, a menu, etc.) having a plurality of different selectable tools associated with markings in the user interface. In some implementations, the user interface element is anchored to a point in space. For example, the user interface element may be moved to a new fixed point in space. In some implementations, when a user rotates his / her head, the user interface element remains anchored to a point in space and may leave the field of view until the user completes an inverse head rotation action (e.g., world / object lock). In some implementations, the user interface element is anchored to a point in a user's field of view of the computing system (e.g., head / body locked). For example, the user interface element may be moved to a new fixed point in the FOV. In some implementations, when a user rotates his / her head, the user interface element will remain anchored to a point in the FOV such that the toolbar remains within the FOV.

[0232] 7A , the electronic device 120 presents an XR environment 128 that includes a menu 712. As shown in FIG. 7A , the menu 712 includes a plurality of selectable options 714 associated with changing the appearance of marks created within the XR environment (e.g., a different color, texture, etc.). For example, option 714A is currently selected from among the plurality of selectable options 714. In this example, option 714A corresponds to a first appearance of marks created within the XR environment 128 (e.g., a black mark). As shown in FIG. 7A , the menu 712 also includes a slider 716 for adjusting the thickness of marks created within the XR environment 128.

[0233] As represented by block 1008, while displaying the user interface, method 1000 includes detecting marking input by a physical object. For example, the marking input corresponds to the creation of 2D or 3D XR content, such as a sketch, handwritten text, or doodle. In some implementations, the computing system obtains (e.g., receives, retrieves, or determines) translation and rotation values ​​of the physical object, and detecting the first movement corresponds to detecting a change to one of the translation or rotation values ​​of the physical object. For example, the computing system tracks the physical object via computer vision, a magnetic sensor, or the like. As one example, the physical object corresponds to a proxy object, such as a pencil or pen, without a communication channel to the computing system. As another example, the physical object corresponds to an electronic device, such as a stylus or a finger-worn device, with a wired or wireless communication channel to the computing system, including an IMU, accelerometer, gyroscope, or the like for 6DOF tracking.

[0234] In some implementations, the computing system may generate one or more N-tuple tracking vectors / tensors for a physics object (e.g., object tracking vector 511 in FIGS. 5A and 5B ) that include translation values ​​(e.g., x, y, and z) relative to the world at large or the current operating environment, rotation values ​​(e.g., roll, pitch, and yaw), grip pose instructions (e.g., pointing, writing, erasing, painting, dictating, etc.), currently used tip / end instructions (e.g., a physics object may have an asymmetric design with specific first and second tips, or a symmetric design with non-specific first and second tips), a first input (pressure) value related to how hard the physics object is being pressed against a physical surface, a second input (pressure) value related to how hard the physics object is being gripped by the user, etc.

[0235] In some implementations, the physics object includes a touch-sensitive surface / region configured to detect touch input directed at the physics object, such as a vertically extending touch-sensitive surface. In some implementations, obtaining the translation and rotation values ​​of the physics object includes determining the translation and rotation values ​​of the physics object based on at least one of inertial measurement unit (IMU) data from the physics object, one or more images of the physical environment including the physical object, magnetic tracking data, etc.

[0236] As represented by block 1010, in response to detecting the marking input and following a determination that the marking input is directed toward a physical surface (e.g., a tabletop, another flat surface, etc.), method 1000 includes displaying, via a display device, a mark within a user interface based on the marking input (e.g., a shape, a size, an orientation, etc.) of the mark, wherein a parameter of the mark displayed based on the marking input is determined based on how hard a physical object is pressed against the physical surface. In some implementations, the computing system or a component thereof (e.g., parameter adjuster 528 of FIG. 5A ) following a determination that the marking input is directed toward the physical surface (e.g., a tabletop, another flat surface, etc.), adjusts an output parameter (e.g., a thickness, brightness, color, texture, etc. of the mark) associated with the detected marking input directed toward XR environment 128 based on how hard the physical object is pressed against the physical surface (e.g., a first input (pressure) value). In some implementations, the parameter corresponds to one of the radius, width, thickness, intensity, translucency, opacity, color, or texture of the mark in the user interface (1014).

[0237] According to some implementations, the parameters of the mark are determined based on how hard a predefined portion of the physical object, such as a tip of the physical object in contact with a physical surface in a three-dimensional environment, is pressed against the physical surface. As an example, the physical object corresponds to an electronic device having a pressure sensor at one or both of its ends / tips to detect a first input (pressure) value when pressed against the physical surface. In some implementations, the computing system maps the marking input on the physical surface to a 3D marking area or a 2D canvas in the XR environment. For example, the marking area and the physical surface correspond to a vertical plane offset by Y cm.

[0238] 7G and 7H illustrate a sequence in which detection of marking input 772 causes marks 782A and 782B to be displayed within XR environment 128 according to measured input (pressure) values ​​779. For example, the shape, depth, length, angle, etc. of marks 782A and 782B correspond to spatial parameters (e.g., position values, rotation values, displacements, spatial accelerations, spatial velocities, angular accelerations, angular velocities, etc. associated with the marking input) of marking input 772 in FIG. 7G. Additionally, in FIG. 7H, marks 782A and 782B are associated with a first thickness value that corresponds to measured input (pressure) values ​​779 in FIG.

[0239] 7I and 7J illustrate a sequence in which detection of a marking input 792 causes marks 7102A and 7102B to be displayed within XR environment 128 according to a measured input (pressure) value 799. For example, the shape, depth, length, angle, etc. of marks 7102A and 7102B correspond to the spatial parameters (e.g., position value, rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc. associated with the marking input) of marking input 792 in FIG. 7I. Furthermore, in FIG. 7J, marks 7102A and 7102B are associated with a second thickness value that corresponds to the current measured input (pressure) value 799 in FIG. 7I. For example, the second thickness value associated with marks 7102A and 7102B is greater than the first thickness value associated with marks 782A and 782B.

[0240] As represented by block 1012, in response to detecting the marking input and following a determination that the marking input is not directed toward a physical surface, method 1000 includes displaying, via a display device, a mark within a user interface based on the marking input (e.g., a shape, size, orientation, etc. of the marking input), wherein a parameter of the mark displayed based on the marking input is determined based on how tightly the physical object is grasped by the user. In some implementations, the computing system or a component thereof (e.g., parameter adjuster 528 of FIG. 5A ) following a determination that the marking input is not directed toward a physical surface, adjusts an output parameter (e.g., thickness, brightness, color, texture, etc. of the mark) associated with the detected marking input directed toward XR environment 128 based on how tightly the physical object is grasped by user 149 (e.g., the second input (pressure) value). In some implementations, the parameter corresponds to one of a radius, width, thickness, intensity, translucency, opacity, color, or texture of the mark within the user interface (1014).

[0241] According to some implementations, the computing system detects a marking input while the physical object or a predetermined portion of the physical object, such as a tip of the physical object, is not in contact with any physical surface in the three-dimensional environment. For example, the physical object corresponds to an electronic device having a built-in pressure sensor for detecting a second input (pressure) value when grasped by a user.

[0242] 7A and 7B illustrate a sequence in which detection of a marking input 715 causes a mark 722 to be displayed within the XR environment 128 according to a current measured value 719 of the input (pressure) value. For example, the shape, depth, length, angle, etc. of the marking input 715 correspond to spatial parameters (e.g., position value, rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc. associated with the marking input). Additionally, in FIG. 7B, the mark 722 is associated with a first thickness value that corresponds to the current measured value 719 of the input (pressure) value in FIG. 7A.

[0243] 7C and 7D illustrate a sequence in which detection of a marking input 732 causes a mark 742 to be displayed within the XR environment 128 according to a measured input (pressure) value 739. For example, the shape, depth, length, angle, etc. of the mark 742 correspond to the spatial parameters of the marking input 732. Additionally, in FIG. 7D, the mark 742 is associated with a second thickness value that corresponds to the current measured input (pressure) value 739 in FIG. 7C. For example, the second thickness value associated with the mark 742 is greater than the first thickness value associated with the mark 722.

[0244] In some implementations, as represented by block 1016, the method 1000 includes, after displaying the mark within the user interface, detecting a subsequent input by a physical object associated with moving (e.g., translating and / or rotating) the mark within the user interface, and, in response to detecting the subsequent input, moving the mark within the user interface based on the subsequent input. As an example, FIGS. 7E and 7F show a sequence in which the electronic device translates a mark 742 within the XR environment 128 in response to detecting a manipulation input 752. For example, the angle, directionality, displacement, etc. of the translational movement of the mark 742 corresponds to the spatial parameters of the manipulation input 752 of FIG. 7E . In some implementations, the manipulation input 752 may also cause a rotational movement of the mark 742 based on the rotational parameters of the manipulation input 752.

[0245] In some implementations, detecting the subsequent input, as represented by block 1018, corresponds to obtaining an indication that an affordance on the physical object has been actuated and detecting at least one of a rotational movement or a translational movement of the physical object. For example, actuation of the affordance corresponds to detecting a touch input directed at the touch-sensitive surface of control device 130. As an example, Figures 7E and 7F show a sequence in which electronic device 120 translates mark 742 within XR environment 128 in response to detecting manipulation input 752 while also detecting touch input 754 directed at touch-sensitive surface 175 of control device 130 in Figure 7E.

[0246] In some implementations, detecting the subsequent input, as represented by block 1020, corresponds to obtaining an indication that an input value associated with how tightly the physics object is gripped by the user exceeds a threshold input value and detecting at least one of a rotational movement or a translational movement of the physics object. For example, the input (pressure) value corresponds to a selected portion of the subsequent input. In some implementations, the pressure threshold is non-deterministic (i.e., a predetermined pressure value) or deterministic based on one or more factors, such as user preference, usage history, current content, current context, etc.

[0247] In some implementations, in response to detecting the subsequent input, method 1000 includes changing the appearance of at least some content within the user interface while moving the mark within the user interface, as represented by block 1022. For example, the computing system increases the opacity, translucency, blur radius, etc. of at least some content, such as the 2D canvas or the 3D marking area.

[0248] In some implementations, as represented by block 1024, in response to detecting the marking input and following a determination that the marking input is directed toward the physical surface, method 1000 includes displaying, via the display device, a simulated shadow in the XR environment corresponding to a distance between the physical surface and the physical object. In some implementations, the size, angle, etc. of the shadow changes as the physical object moves closer to or farther from the physical surface. As an example, as the physical object moves farther from the physical surface, the size of the simulated shadow increases and the associated opacity value decreases. Continuing with this example, as the physical object moves closer to the physical surface, the size of the simulated shadow decreases and the associated opacity value increases. In some implementations, a shadow may also be shown when the marking input is not directed toward the physical surface.

[0249] 7K and 7L illustrate a sequence in which electronic device 120 displays first XR content 7122 within XR environment 128 according to a current measured value 7119 of the input (pressure) value in response to detecting a first content-positioning input associated with touch input 7111. As shown in FIG. 7L, electronic device 120 also displays a shadow 7124 associated with the first XR content 7122 on XR substrate 7118. For example, the position and rotation values ​​of first XR content 7122 and shadow 7124 correspond to the parameters (e.g., position value, rotation value, etc.) of representation 131 of control device 130 when touch input 7111 was detected in FIG. 7K. For example, first XR content 7122 is associated with a first size value that corresponds to the current measured value 7119 of the input (pressure) value in FIG. 7K.

[0250] 7M and 7N illustrate a sequence in which electronic device 120 displays second XR content 7142 within XR environment 128 according to measured input (pressure) value 7139 in response to detecting a second content-positioning input associated with touch input 7131. As shown in FIG. 7N, electronic device 120 does not display a shadow associated with second XR content 7142 on XR substrate 7118. For example, the position and rotation values ​​of second XR content 7142 correspond to the parameters (e.g., position value, rotation value, etc.) of representation 131 of control device 130 when touch input 7131 is detected in FIG. 7M. For example, second XR content 7142 is associated with a second size value that corresponds to the current measured input (pressure) value 7139 in FIG. 7K. For example, the second size value associated with second XR content 7142 is greater than the first size value associated with first XR content 7122.

[0251] 11 is a flowchart representation of a method 1100 for changing a selection modality based on whether a user is currently grasping a physical object, according to some implementations. In various implementations, the method 1100 is performed in a computing system including non-transitory memory and one or more processors, the computing system being communicatively coupled to a display device and (optionally) one or more input devices (e.g., the electronic device 120 shown in FIGS. 1 and 3 , the controller 110 of FIGS. 1 and 2 , or a suitable combination thereof). In some implementations, the method 1100 is performed by processing logic, the processing logic including hardware, firmware, software, or a combination thereof. In some implementations, the method 1100 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). In some implementations, the computing system corresponds to one of a tablet, a laptop, a mobile phone, a near-eye system, a wearable computing device, etc.

[0252] Typically, when navigating content within a user interface, users are limited to one or more input modalities, such as touch input or voice commands. Furthermore, one or more input modalities may be applicable regardless of the current situation, such as while operating a vehicle, while moving, or while hands are full, which may raise usability and safety concerns. In contrast, the methods described herein enable content selection based on gaze direction when a user is not holding a physical object (e.g., a proxy object or a stylus) with a pointing grip, and also enable content selection based on the orientation of a physical object when a user is holding a physical object with a pointing grip. Thus, the input modality for selecting content dynamically changes based on the current context.

[0253] As represented by block 1102, method 1100 includes displaying content via a display device. As one example, the content corresponds to volumetric or 3D content within an XR environment. As another example, the content corresponds to flat or 2D content within a user interface (UI). For example, with reference to FIGS. 8A-8D , electronic device 120 displays VA 606 and XR content 802 within XR environment 128.

[0254] In some implementations, the display device corresponds to a transparent lens assembly, and the presentation of the content is projected onto the transparent lens assembly. In some implementations, the display device corresponds to a near-eye system, and the presentation of the content includes combining the presentation of the content with one or more images of a physical environment captured by an outward-facing image sensor.

[0255] As represented by block 1104, while displaying the content and while the physical object is being held by the user, method 1100 includes detecting a selection input. As one example, the physical object corresponds to a proxy object detected in the physical environment that has no communication channel to the computing system, such as a pencil, pen, etc. With reference to FIGS. 8A-8D , electronic device 120 displays a representation 153 of a right hand 152 of user 149 holding a representation 805 of a proxy object 804 (e.g., a stick, ruler, or another physical object) in XR environment 128. For example, right hand 152 of user 149 is currently holding proxy object 804 in a pointing grip pose. As another example, the physical object corresponds to an electronic device that has a wired or wireless communication channel to the computing system, such as a stylus, a finger-worn device, or a handheld device. 8E-8G, electronic device 120 displays representation 153 of right hand 152 of user 149 holding representation 131 of control device 130 within XR environment 128. For example, right hand 152 of user 149 is currently holding control device 130 in a writing grip pose that is not directed toward any XR content within XR environment 128.

[0256] As represented by block 1106, in response to detecting the selection input, the method 1100 includes performing an operation corresponding to the selection input. In some implementations, the computing system or a component thereof (e.g., the content selection engine 522 of FIG. 5A ) determines the selected content portion 523 based on the characterization vector 531 (or a portion thereof). For example, the content selection engine 522 determines the selected content portion 523 based on current context information, a gaze direction of the user 149, body pose information associated with the user 149, head pose information associated with the user 149, hand / limb tracking information associated with the user 149, position information associated with a physics object, rotation information associated with a physics object, etc.

[0257] As one example, following a determination that a grip posture associated with the manner in which the physics object is being held by the user corresponds to a first grip (e.g., first grip = pointing / wand-like grip), the content selection engine 522 performs a selection operation on a first portion of the content based on a direction in which a ray of light projecting from a predetermined portion (e.g., an outward-facing end) of the physics object is pointing. As another example, following a determination that a grip posture associated with the manner in which the physics object is being held by the user does not correspond to the first grip, the content selection engine 522 performs a selection operation on a second portion of the content based on a gaze direction of the user.

[0258] In some implementations, as represented by block 1108, method 1100 includes altering the appearance of the first or second portion of content. As one example, altering the appearance of the first or second portion of content corresponds to altering the color, texture, brightness, etc. of the first or second portion of content to indicate its selection. As another example, altering the appearance of the first or second portion of content corresponds to displaying a bounding box, highlight, spotlight, etc. associated with the first or second portion of content to indicate its selection. For example, with reference to FIG. 8C , in response to detecting that proxy object 804 has been pointed at XR content 802 in FIGS. 8A and 8B for at least a predetermined or deterministic period of time, electronic device 120 alters the appearance of XR content 802 to a cross-hatched appearance 802A to visually indicate its selection. For example, referring to FIG. 6D , in response to detecting that movement of the control device 130 has caused the spatial location of the representation 131 of the control device 130 to exceed the activation area 634 for the graphical element 632D, the electronic device 120 changes the appearance of the graphical element 632D by displaying a border or frame 642 around the graphical element 632D to indicate its selection.

[0259] As represented by block 1110, pursuant to determining that a grip posture associated with the manner in which the physics object is being held by the user corresponds to a first grip (e.g., first grip = pointing / wand-like grip), method 1100 includes performing a selection operation on a first portion of the content, where the first portion of the content is selected based on a direction in which a predetermined portion (e.g., an outward-facing end) of the physics object is pointing (e.g., a ray projecting from it) (e.g., regardless of a user's gaze direction). In some implementations, a computing system or a component thereof (e.g., content selection engine 522 of FIG. 5A ) performs a selection operation on the first portion of the content based on a direction in which a predetermined portion (e.g., an outward-facing end) of the physics object is pointing (e.g., a ray projecting from it) pursuant to determining that a grip posture associated with the manner in which the physics object is being held by the user corresponds to a first grip (e.g., first grip = pointing / wand-like grip). As an example, FIGS. 8B and 8C illustrate a sequence in which the electronic device 120 selects the XR content 802 in response to detecting that the proxy object 804 (or a representation thereof 805) has been pointed at the XR content 802 for at least a predetermined or deterministic period of time, and following a determination that a grip posture associated with the manner in which the physics object 804 is being held by the user 149 corresponds to a first grip (e.g., a pointing grip).

[0260] In some implementations, a computing system obtains (e.g., receives, retrieves, or determines) translation and rotation values ​​for a physics object and obtains (e.g., receives, retrieves, or determines) a grip pose associated with the current manner in which the physics object is being held by a user. For example, the computing system tracks the physics object via computer vision, a magnetic sensor, or the like. As one example, the physics object corresponds to a proxy object, such as a pencil, pen, or the like, that has no communication channel to the computing system. As another example, the physics object corresponds to an electronic device, such as a stylus, finger-worn device, or the like, that has a wired or wireless communication channel to the computing system, including an IMU, accelerometer, gyroscope, or the like for 6DOF tracking. In some implementations, the computing system may track one or more N-tuple tracking vectors / tensors of the physics object, including translation values ​​(e.g., x, y, and z) relative to the world at large or the current operating environment, rotation values ​​(e.g., roll, pitch, and yaw), grip pose indications (e.g., pointing, writing, erasing, painting, dictation, etc., poses), currently used tip / end indications (e.g., the physics object may have an asymmetric design with specific first and second tips, or a symmetric design with non-specific first and second tips), a first input (pressure) value related to how hard the physics object is pressed against a physical surface, a second input (pressure) value related to how hard the physics object is gripped by the user, etc. In some implementations, the physics object includes a touch-sensitive surface / region configured to detect touch input directed at the physics object, such as a vertically extending touch-sensitive surface. In some implementations, obtaining the translation and rotation values ​​of the physics object includes determining the translation and rotation values ​​of the physics object based on at least one of IMU data from the physics object, one or more images of the physical environment including the physics object, magnetic tracking data, etc.

[0261] As represented by block 1112, following a determination that the grip pose associated with the manner in which the physics object is being held by the user does not correspond to the first grip, method 1100 includes performing a selection operation on a second portion of the content that differs from the first portion of the content, where the second portion of the content is selected based on the user's gaze direction (e.g., regardless of the direction of a ray casting from a given portion of the physics object). In some implementations, the computing system or a component thereof (e.g., the gaze tracking engine 512 of FIG. 5A ) determines and updates an gaze tracking vector 513 that includes x and y coordinates, a focal length or focus, etc. associated with the gaze direction relative to the world at large or the current operating environment. In some implementations, the computing system determines the gaze tracking vector 513 based on one or more images of the user's eye(s) from an inward-facing image sensor. In some implementations, the computing system determines a region of interest (ROI) (e.g., an N×M mm ROI) within the XR environment 128 based on the gaze direction.

[0262] In some implementations, the computing system or a component thereof (e.g., content selection engine 522 of FIG. 5A ) performs a selection operation on a second portion of the content based on the user's gaze direction pursuant to a determination that a grip pose associated with the manner in which the physical object is being held by the user does not correspond to the first grip. As an example, FIGS. 8E and 8F show a sequence in which electronic device 120 selects XR content 802 in response to detecting that gaze direction 806 of user 149 is directed toward XR content 802 for at least a predetermined or deterministic period of time, and pursuant to a determination that a grip pose associated with the manner in which the physical object 130 is being held by user 149 does not correspond to the first grip.

[0263] In some implementations, following a determination that the grip pose corresponds to the first grip, as represented by block 1114, the method 1100 includes displaying, via the display device, a first graphical element indicating a direction in which a predetermined portion of the physical object is pointing relative to the content. For example, the first graphical element is displayed at a coincidence point where a ray of light projected from the predetermined portion of the physical object coincides with content, a 2D canvas, a 3D marking area, a backplane, etc., within the XR environment 128. As an example, referring to FIG. 8A , the electronic device 120 displays a first indicator element 812A on the XR content 802 having a first size corresponding to a coincidence point between the XR content 802 and a ray of light emanating from a tip / edge of the representation 805 of the proxy object 804.

[0264] In some implementations, a size parameter (e.g., radius) of the first graphical element is a function of the distance between the first portion of the content and the physical object (1116). In some implementations, the size of the first indicator element increases as the distance between the first portion of the content and the physical object decreases, and the size of the first indicator element decreases as the distance between the first portion of the content and the physical object increases. As an example, referring to FIG. 8A , the electronic device 120 displays a first indicator element 812A on the XR content 802, the first indicator element 812A having a first size corresponding to a coincidence point between the XR content 802 and a ray of light emanating from the tip / edge of the representation 805 of the proxy object 804. In this example, the size of the first indicator element 812A is a function of the first distance 814. 8B , the electronic device 120 displays a second indicator element 812B on the XR content 802, the second indicator element 812B having a second size corresponding to a coincidence point between the XR content 802 and a ray emanating from the leading / trailing end of the representation 805 of the proxy object 804. As shown in FIG. 8B , the representation 805 of the proxy object 804 is at a second distance 824 from the XR content 802, which is less than the first distance 814 in FIG. 8A . For example, the second size of the second indicator element 812B is greater than the first size of the first indicator element 812A.

[0265] In some implementations, as represented by block 1118, following a determination that the grip pose does not correspond to the first grip, the method 1100 includes displaying, via the display device, a second graphical element indicating the user's gaze direction relative to the content. For example, the second graphical element is different from the first graphical element. For example, the second graphical element is displayed at a coincidence point where a ray projected from one or more of the user's eyes strikes the content, the 2D canvas, the 3D marking area, the backplane, etc. within the XR environment. For example, referring to FIG. 8E , the electronic device 120 displays, on the XR content 802, a gaze direction indicator element 852 associated with the focal point within the XR environment 128 of the user's 149 eyes based on eye tracking.

[0266] In some implementations, a size parameter (e.g., radius) of the second graphical element is a function of the distance between the second portion of the content and one or more eyes of the user (1120). In some implementations, the size of the second indicator element increases as the distance between the first portion of the content and the physical object decreases, and the size of the second indicator element decreases as the distance between the first portion of the content and the physical object increases.

[0267] In some implementations, as represented by block 1122, the method 1100 includes detecting a subsequent input by a physics object associated with the movement (e.g., translation and / or rotation) of the content while displaying the content, and, in response to detecting the subsequent input, moving the content based on the subsequent input. As an example, referring to FIG. 8D , the electronic device 120 translates the XR content 802 within the XR environment 128 in response to detecting translational movement 832 of the proxy object 804 in FIG. 8C . For example, the direction and displacement of the translational movement of the XR content 802 within the XR environment 128 correspond to spatial parameters (e.g., change in position value, change in rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc.) of the translational movement 832 of the proxy object 804 in FIG. 8C . Those skilled in the art will appreciate that the XR content 802 may be similarly rotated.

[0268] 8G , electronic device 120 translates XR content 802 within XR environment 128 in response to detecting translation 862 of gaze direction 806 in FIG. 8F . For example, the direction and displacement of the translation of XR content 802 within XR environment 128 corresponds to spatial parameters (e.g., change in position value, change in rotation value, displacement, spatial acceleration, spatial velocity, angular acceleration, angular velocity, etc.) of translation 862 of gaze direction 806 in FIG. 8F .

[0269] In some implementations, detecting the subsequent input corresponds to obtaining an indication that an affordance on the physics object has been actuated and detecting at least one of a rotational movement or a translational movement of the physics object (1124). For example, detecting the actuation of the affordance corresponds to the selection portion of the subsequent input.

[0270] In some implementations, detecting the subsequent input corresponds to obtaining an indication that an input value associated with how tightly the physics object is gripped by the user exceeds a threshold input value and detecting at least one of a rotational movement or a translational movement of the physics object (1126). For example, the input (pressure) value corresponds to a selected portion of the subsequent input. In some implementations, the pressure threshold is non-deterministic (i.e., a predetermined pressure value) or deterministic based on one or more factors, such as user preference, usage history, current content, current context, etc.

[0271] In some implementations, the magnitude of the subsequent input is modified by an amplification factor to determine the magnitude of the content movement (1128). In some implementations, the amplification factor is non-deterministic (e.g., a predetermined value) or deterministic based on one or more factors such as user preferences, usage history, selected content, current context, etc.

[0272] While various aspects of implementations within the scope of the appended claims have been described above, it should be apparent that various features of the above-described implementations may be embodied in a wide variety of forms, and that any specific structure and / or function described above is merely illustrative. Based on this disclosure, one skilled in the art should understand that any aspect described herein may be practiced independently of any other aspect, or that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and / or a method may be practiced using any number of the aspects described herein. In addition, such an apparatus may be implemented and / or such a method may be practiced using structure and / or functionality in addition to one or more of the aspects described herein.

[0273] Although terms such as "first" and "second" may be used herein to describe various elements, it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first media item can be referred to as a second media item, and similarly, a second media item can be referred to as a first media item, without changing the meaning of the description, as long as all occurrences of "first media item" are consistently renamed and all occurrences of "second media item" are consistently renamed. Although a first media item and a second media item are both media items, they are not the same media item.

[0274] The terminology used herein is for the purpose of describing particular implementations and is not intended to limit the scope of the claims. When used in the description of the illustrated implementations and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. Also, as used herein, the term "and / or" should be understood to refer to and include any and all possible combinations of one or more of the associated listed items. It should be further understood that the terms "comprises" and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0275] As used herein, the term "if" can be interpreted to mean "when" or "upon" or "upon determining" or "in accordance with determining" or "in accordance with detecting" that the aforementioned condition is true, depending on the context. Similarly, the phrases "if it is determined that [the aforementioned condition is true]," "if [the aforementioned condition is true]," or "when [the aforementioned condition is true]" can be interpreted to mean "upon determining," "in accordance with determining," "upon detecting," or "in accordance with detecting" that the aforementioned condition is true.

Claims

1. 1. A method comprising:

1. A computing system comprising: a non-transitory memory; and one or more processors, communicatively coupled to a display device and one or more input devices, displaying, via the display device, a first plurality of graphical elements associated with a first plurality of output modalities associated with how the physical object interacts within an augmented reality (XR) environment, the first plurality of output modalities being associated with how the physical object interacts within the XR ... Detecting a first movement of the physical object while displaying the first plurality of graphical elements within the augmented reality (XR) environment including the representation of the physical object; in response to detecting the first movement of the physics object; pursuant to determining that the first movement of the physical object causes the physical object to move beyond a three-dimensional activation area of ​​a first graphical element of the first plurality of graphical elements, selecting a first output modality associated with the first graphical element as a current output modality for the physical object, maintaining display of the first graphical element overlaid on the tip of the representation of the physical object in the augmented reality (XR) environment, and ceasing display of the remainder of the first plurality of graphical elements not including the first graphical element; pursuant to determining that the first movement of the physical object causes the physical object to exceed a three-dimensional activation area of ​​a second graphical element of the first plurality of graphical elements, selecting a second output modality associated with the second graphical element as the current output modality of the physical object, maintaining display of the second graphical element overlaid on the tip of the representation of the physical object in the augmented reality (XR) environment, and ceasing display of the remainder of the first plurality of graphical elements not including the second graphical element; A method comprising:

2. The method of claim 1 , wherein the first and second output modalities cause different visual changes within the XR environment.

3. detecting a second movement of the physical object after selecting the first output modality associated with the first graphical element as the current output modality of the physical object; in response to detecting a second movement of the physical object, moving the first graphical element based on the second movement of the physical object to maintain a display of the first graphical element overlaid on the tip of the representation of the physical object within the augmented reality (XR) environment; The method of claim 1 further comprising:

4. obtaining an indication of touch input directed at the physical object after ceasing the display of the remainder of the first plurality of graphical elements; re-displaying the first plurality of graphical elements within the XR environment via the display device in response to obtaining the indication of the touch input; The method of claim 1 further comprising:

5. and further comprising, prior to displaying the first plurality of graphical elements, obtaining an indication of touch input directed at the physical object; displaying the first plurality of graphical elements within the XR environment in response to obtaining the indication of the touch input; The method of claim 1 , comprising:

6. responsive to determining that the first movement of the physics object does not cause the physics object to exceed the three-dimensional activation area of ​​the first graphical element or the second graphical element; maintaining an initial output modality as the current output modality of the physical object; maintaining a display of the first plurality of graphical elements; The method of claim 1 further comprising:

7. detecting a subsequent marking input by the physical object after selecting the first output modality associated with the first graphical element as the current output modality of the physical object; In response to detecting the subsequent marking input, displaying, via the display device, one or more marks within the XR environment based on the subsequent marking input and the first output modality; The method of claim 1 further comprising:

8. in response to detecting the subsequent marking input; pursuant to determining that an input associated with how hard the physical object is pressed against a physical surface corresponds to a first input value, displaying, via the display device, within the XR environment based on the subsequent marking input and the first output modality, one or more marks having a first appearance, the first appearance being associated with a parameter of the one or more marks corresponding to the first input value; pursuant to determining that the input associated with how hard the physical object is pressed against a physical surface corresponds to a second input value, displaying, via the display device, within the XR environment based on the subsequent marking input and the first output modality, one or more marks having a second appearance, the second appearance being associated with the parameter of the one or more marks corresponding to the second input value; The method of claim 7 further comprising:

9. The method of claim 8 , wherein the parameter corresponds to one of a radius, width, thickness, intensity, translucency, opacity, color, or texture of the one or more marks in the XR environment.

10. in response to detecting the subsequent marking input; pursuant to determining that an input associated with how tightly the physical object is gripped by the user corresponds to a first input value, displaying, via the display device, within the XR environment based on the subsequent marking input and the first output modality, one or more marks having a first appearance, the first appearance being associated with a parameter of the one or more marks corresponding to the first input value; pursuant to determining that the input associated with how tightly the physical object is gripped by the user corresponds to a second input value, displaying, via the display device, within the XR environment based on the subsequent marking input and the first output modality, one or more marks having a second appearance, the second appearance being associated with the parameter of the one or more marks corresponding to the second input value; The method of claim 7 further comprising:

11. The method of claim 10 , wherein the parameter corresponds to one of a radius, width, thickness, intensity, translucency, opacity, color, or texture of the one or more marks in the XR environment.

12. obtaining a grip posture associated with a current manner in which the physical object is being held by a user prior to displaying the first plurality of graphical elements, the first plurality of graphical elements being a function of the grip posture; In response to acquiring the grip posture, displaying, via the display device, the first plurality of graphical elements associated with the first plurality of output modalities within the XR environment in accordance with determining that the grip posture corresponds to a first grip posture; displaying, via the display device, a second plurality of graphical elements associated with a second plurality of output modalities within the XR environment in accordance with determining that the grip posture corresponds to a second grip posture different from the first grip posture; The method of claim 1 further comprising:

13. detecting a change in the grip posture from the first grip posture to the second grip posture after displaying the first plurality of graphical elements associated with the first plurality of output modalities within the XR environment; responsive to detecting the change in grip posture, replacing a display of the first plurality of graphical elements in the XR environment with the second plurality of graphical elements associated with the second plurality of output modalities in the XR environment; The method of claim 12 further comprising:

14. obtaining information indicating whether a first end or a second end of the physics object faces outward before displaying the first plurality of graphical elements; in response to obtaining the information indicating whether the first end or the second end of the physical object is outwardly facing; displaying, via the display device, the first plurality of graphical elements associated with the first plurality of output modalities within the XR environment in accordance with determining that the first end of the physical object is facing outward; displaying, via the display device, a second plurality of graphical elements associated with a second plurality of output modalities within the XR environment in accordance with determining that the second end of the physical object is facing outward; The method of claim 1 further comprising:

15. detecting a change from an outward-facing first end of the physical object to an outward-facing second end of the physical object after displaying the first plurality of graphical elements associated with the first plurality of output modalities within the XR environment; displaying, via the display device, within the XR environment, the second plurality of graphical elements associated with the second plurality of output modalities in response to detecting the change from the outward-facing first end of the physical object to the outward-facing second end of the physical object; The method of claim 14 further comprising:

16. 1. An electronic device comprising: one or more processors; a non-transient memory; one or more programs stored in the non-transitory memory, the programs comprising instructions configured to be executed by the one or more processors to perform the method of any one of claims 1 to 15; , an electronic device.

17. 16. A non-transitory computer-readable medium storing one or more programs that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method of any one of claims 1 to 15.

Images