System and method for annotating in three-dimensional environment

Abstract

The present disclosure provides a system and method for annotating in a three-dimensional environment. Some examples of the present disclosure relate to systems and methods for displaying and editing virtual scenes. In some examples, an electronic device may display an immersive virtual scene. In some examples, the immersive virtual scene is shared in a communication session with other electronic devices. In some examples, the electronic device may detect a user input requesting insertion and display of an annotation into the virtual scene. In some examples, the electronic device may determine a user context to determine placement and content associated with an inserted annotation, and display a representation of the annotation. In some examples, the electronic device may capture a virtual screenshot of the virtual scene. In some examples, the electronic device may display a user interface that provides an overview of the annotation associated with the virtual scene.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 586,783, filed on September 29, 2023, and U.S. Patent Application No. 18 / 891,984, filed on September 20, 2024, the contents of both applications are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present application generally relates to systems and methods for displaying a virtual environment and adding virtual content to the virtual environment. Background Art

[0004] The proliferation of software and hardware capable of creating digital scenes has enabled users of electronic devices to create special effects backgrounds, computer graphics, and immersive virtual environments for theaters, software applications, interactive exhibits, and the like. Current systems use computing displays to facilitate the creation and editing of digital scenes, but typically require input directed to computing peripherals such as joysticks, computing mice, trackpads, keyboards, and the like. However, notes and annotations for such digital scenes are spatially separated from related content within the digital scenes. In addition, inputting such notes and annotations using conventional computing peripherals may be inefficient. Therefore, it is understandable that a system may be needed to facilitate annotation of digital assets in a simulated three-dimensional environment and improve user input efficiency. Summary of the invention

[0005] Some examples of the present disclosure relate to systems and methods for inserting annotations and facilitating collaboration when viewing a three-dimensional scene that is at least partially virtual. In some examples, an electronic device may present and / or display the three-dimensional environment to a user of the three-dimensional environment. In some examples, the three-dimensional environment may include representations of physical objects and / or individuals. In some examples, the three-dimensional environment may include one or more virtual objects and / or virtual assets.

[0006] In some examples, the electronic device may display one or more representations of a physical individual. In some examples, the one or more representations include a representation of a user of another electronic device participating in a communication session with the electronic device. In some examples, the electronic device displays a virtual scene that can be shared with the other electronic device. In some examples, the virtual scene is displayed as if the user of the electronic device is present within a physical equivalent of the virtual scene. In some examples, the representation of the physical individual moves relative to the virtual scene.

[0007] In some examples, the electronic device displays an indication of the user's attention within the three-dimensional environment. In some examples, the electronic device displays an indication of the attention corresponding to the attention of other users who check the virtual scene. In some examples, the electronic device detects an input requesting the annotation to be inserted and displayed in the virtual scene, and in response to the request, displays a representation of the annotation. In some examples, the user input requesting the annotation to be inserted includes an air gesture, a voice command, the movement of the user's body, and / or the gaze of the user pointing to the virtual content included in the virtual scene. In some examples, the electronic device determines the context of the user's interaction with the virtual scene and / or the three-dimensional environment in response to detecting the user input requesting the annotation to be inserted. In some examples, the electronic device obtains information such as text, voice recording, the movement of the user's body, and / or the movement of the user's attention relative to the virtual scene in response to the user input, and associates the provided information with the inserted annotation.

[0008] In some examples, the electronic device determines the context of the user's environment to determine the placement and / or orientation of the representation of the annotation. In some examples, the electronic device uses the context to determine the information stored and corresponding to the representation of the annotation. In some examples, the context of the user includes the modality of the user input. In some examples, the context of the user includes parsed speech and / or natural language processing to determine the semantics of the user's speech. In some examples, the visual appearance of the representation of the annotation has visual characteristics or attributes indicating the author of the corresponding annotation. In some examples, in response to detecting a user input pointing to a representation of an annotation, the electronic device presents information, such as a recording of audio, a display of text, playback of a spatial recording of an individual's representation, playback of a recording of the individual's attention, and / or some combination thereof.

[0009] In some examples, the electronic device captures a virtual screenshot of the virtual scene. In some examples, the virtual screenshot is a virtual object that a user can annotate, similar to or identical to an annotation pointing to the virtual scene. In some examples, the annotation inserted into the virtual screenshot can be inserted into the virtual scene by the electronic device. In some examples, the electronic device can change a visual attribute or characteristic of the virtual screenshot in response to an angular movement relative to the virtual screenshot. In some examples, the electronic device can export the screenshot to another device.

[0010] In some examples, the electronic device displays a user interface that provides an overview of the annotations included in the virtual scene. In some examples, the user interface can classify and / or group lists and / or icons corresponding to the annotations according to the type of the corresponding annotation, the author who provided the annotation, and / or the chronological order of the input of the annotation. In some examples, in response to detecting a user input pointing to a representation of an annotation within the user interface, the electronic device can display, visually emphasize, and / or present the annotation information associated with the target of the user input.

[0011] A comprehensive description of these examples is provided in the drawings and detailed description, and it should be understood that this summary does not limit the scope of the present disclosure in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a better understanding of the various examples described herein, reference should be made to the following detailed description and the following drawings. Throughout the drawings, like reference numerals generally refer to corresponding parts.

[0013] Figure 1 An electronic device presenting an extended reality environment according to some examples of the present disclosure is illustrated.

[0014] Figure 2 A block diagram illustrating an exemplary architecture of a device according to some examples of the present disclosure.

[0015] FIG. 3A to FIG. 3L Example interactions including annotation, editing, and inspection of a virtual scene according to some examples of the present disclosure are illustrated.

[0016] Figure 4 A flow chart illustrating an example process for interaction including annotation, editing, and inspection of a virtual scene according to some examples of the present disclosure is illustrated.

[0017] FIG. 5A to FIG. 5H Example interactions with a user interface for interacting with a virtual scene according to some examples of the present disclosure are illustrated. DETAILED DESCRIPTION

[0018] Some examples of the present disclosure relate to systems and methods for inserting annotations and facilitating collaboration when viewing a three-dimensional scene that is at least partially virtual. In some examples, an electronic device may present and / or display the three-dimensional environment to a user of the three-dimensional environment. In some examples, the three-dimensional environment may include representations of physical objects and / or individuals. In some examples, the three-dimensional environment may include one or more virtual objects and / or virtual assets.

[0019] In some examples, the electronic device may display one or more representations of a physical individual. In some examples, the one or more representations include a representation of a user of another electronic device participating in a communication session with the electronic device. In some examples, the electronic device displays a virtual scene that can be shared with the other electronic device. In some examples, the virtual scene is displayed as if the user of the electronic device is present within a physical equivalent of the virtual scene. In some examples, the representation of the physical individual moves relative to the virtual scene.

[0020] In some examples, the electronic device displays an indication of the user's attention within the three-dimensional environment. In some examples, the electronic device displays an indication of the attention corresponding to the attention of other users who check the virtual scene. In some examples, the electronic device detects an input requesting the annotation to be inserted and displayed in the virtual scene, and in response to the request, displays a representation of the annotation. In some examples, the user input requesting the annotation to be inserted includes an air gesture, a voice command, the movement of the user's body, and / or the gaze of the user pointing to the virtual content included in the virtual scene. In some examples, the electronic device determines the context of the user's interaction with the virtual scene and / or the three-dimensional environment in response to detecting the user input requesting the annotation to be inserted. In some examples, the electronic device obtains information such as text, voice recording, the movement of the user's body, and / or the movement of the user's attention relative to the virtual scene in response to the user input, and associates the provided information with the inserted annotation.

[0021] In some examples, the electronic device determines the context of the user's environment to determine the placement and / or orientation of the representation of the annotation. In some examples, the electronic device uses the context to determine the information stored and corresponding to the representation of the annotation. In some examples, the context of the user includes the modality of the user input. In some examples, the context of the user includes parsed speech and / or natural language processing to determine the semantics of the user's speech. In some examples, the visual appearance of the representation of the annotation has visual characteristics or attributes indicating the author of the corresponding annotation. In some examples, in response to detecting a user input pointing to a representation of an annotation, the electronic device presents information, such as a recording of audio, a display of text, playback of a spatial recording of an individual's representation, playback of a recording of the individual's attention, and / or some combination thereof.

[0022] In some examples, the electronic device captures a virtual screenshot of the virtual scene. In some examples, the virtual screenshot is a virtual object that a user can annotate, similar to or identical to an annotation pointing to the virtual scene. In some examples, the annotation inserted into the virtual screenshot can be inserted into the virtual scene by the electronic device. In some examples, the electronic device can change a visual attribute or characteristic of the virtual screenshot in response to an angular movement relative to the virtual screenshot. In some examples, the electronic device can export the screenshot to another device.

[0023] In some examples, the electronic device displays a user interface that provides an overview of the annotations included in the virtual scene. In some examples, the user interface may classify and / or group lists and / or icons corresponding to the annotations according to the type of the corresponding annotation, the author providing the annotation, and / or the chronological order of the input of the annotation. In some examples, in response to detecting a user input pointing to a representation of an annotation within the user interface, the electronic device may display, visually emphasize, and / or present the annotation information associated with the target of the user input. In some examples, the user interface may include selectable options that may be selected to insert into a virtual scene and / or interact with the virtual scene.

[0024] Figure 1 An electronic device 101 is illustrated that presents an extended reality (XR) environment (e.g., a computer-generated environment) according to some examples of the present disclosure. In some examples, the electronic device 101 is a handheld device or a mobile device, such as a tablet, a laptop, a smart phone, or a head-mounted display. Figure 2 An example of the device 101 is described by using an architectural block diagram of FIG. Figure 1 As shown, electronic device 101 and table 106 are located in physical environment 100. The physical environment may include physical features, such as physical surfaces (e.g., floors, walls) or physical objects (e.g., tables, lights, etc.). In some examples, electronic device 101 may be configured to capture an image of physical environment 100 including table 106 (illustrated in the field of view of electronic device 101). In some examples, in response to a trigger, electronic device 101 may be configured to capture an image of physical environment 100 including table 106 (illustrated in the field of view of electronic device 101). Figure 1 1. A virtual object 104 (e.g., two-dimensional virtual content) is displayed in the physical environment 100 (e.g., a rectangular representation illustrated), which virtual object does not exist in the physical environment 100 but is displayed in the computer-generated environment positioned on top of (e.g., anchored to) the computer-generated representation 106' of the real-world table 106. For example, in response to detecting a flat surface of the table 106 in the physical environment 100, the virtual object 104 may be displayed on the surface of the computer-generated representation 106' of the table in the computer-generated environment displayed via the electronic device 101.

[0025] It should be understood that the virtual object 104 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or other three-dimensional virtual objects) in a three-dimensional computer-generated environment. For example, a virtual object may represent an application or user interface displayed in a computer-generated environment. In some examples, a virtual object may represent content corresponding to an application and / or displayed via a user interface in a computer-generated environment. In some examples, the virtual object 104 is optionally configured to be interactive and responsive to user input, so that a user can virtually touch, tap, move, rotate, or otherwise interact with the virtual object 104. In some examples, the virtual object 104 may be displayed in a specific orientation in a three-dimensional computer-generated environment. For example, the virtual object 104 may be displayed in a tilt-locked orientation, a head-locked orientation, a body-locked orientation, or a world-locked orientation in a three-dimensional environment. In some such examples, as described in more detail below, when the virtual object 104 is displayed in a three-dimensional environment, the electronic device selectively moves the virtual object 104 according to the specific orientation in which the virtual object is displayed in response to user input (e.g., direct input or indirect input). For example, the electronic device selectively moves the virtual object 104 in response to movement of the user's viewpoint depending on whether the virtual object 104 is body locked, head locked, tilt locked, or world locked. Additionally, it should be understood that the 3D environment (or 3D virtual object) described herein may be a representation of a 3D environment (or three-dimensional virtual object) projected or presented at an electronic device.

[0026] In the following discussion, an electronic device that communicates with a display generation component and one or more input devices is described. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. In addition, as described herein, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed between two or more devices. Therefore, as used in this disclosure, information displayed on or by an electronic device is optionally used to describe information output by the electronic device to be displayed on an independent display device (touch-sensitive or non-touch-sensitive). Similarly, as referred to herein, input received on an electronic device (e.g., touch input received on a touch-sensitive surface of an electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on an independent input device, which is communicated to and / or indicated to the electronic device.

[0027] The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television channel browsing application, and / or a digital video player application.

[0028] Figure 2 A block diagram of an exemplary architecture of a device 201 according to some examples of the present disclosure is illustrated. In some examples, the device 201 includes one or more electronic devices. For example, the electronic device 201 can be a portable device such as a mobile phone, a smart phone, a tablet computer, a laptop computer, an auxiliary device that communicates with another device, a head-mounted display, etc.

[0029] like Figure 2 As illustrated, the electronic device 201 optionally includes various sensors (e.g., one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206, one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, etc.), one or more display generation components 214, one or more speakers 216, one or more processors 218, one or more memories 220, and / or communication circuits 222. One or more communication buses 208 are optionally used for communication between the above components of the electronic device 201.

[0030] The communication circuit 222 optionally includes circuits for communicating with electronic devices, networks such as the Internet, intranets, wired networks and / or wireless networks, cellular networks, and wireless local area networks (LANs). The communication circuit 222 optionally includes circuits for using near field communication (NFC) and / or short range communication such as ) circuit for communication.

[0031] Processor 218 includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 220 is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions, which are configured to be executed by processor 218 to perform the techniques, processes, and / or methods described below. In some examples, memory 220 may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in conjunction with an instruction execution system, device, and apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memory (such as flash memory, solid-state drives, etc.).

[0032] In some examples, display generation component 214 includes a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generation component 214 includes multiple displays. In some examples, display generation component 214 may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, etc. In some examples, electronic device 201 includes a touch-sensitive surface 209 for receiving user inputs (such as tap inputs and swipe inputs or other gestures). In some examples, display generation component 214 and touch-sensitive surface 209 form a touch-sensitive display (e.g., a touch screen integrated with electronic device 201 or a touch screen that communicates with electronic device 201 outside electronic device 201).

[0033] The electronic device 201 optionally includes an image sensor 206. The image sensor 206 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real-world environment. The image sensor 206 also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensor 206 also optionally includes one or more cameras configured to capture the movement of physical objects in the real-world environment. The image sensor 206 also optionally includes one or more depth sensors configured to detect the distance of the physical object from the electronic device 201. In some examples, information from the one or more depth sensors may allow the device to identify an object in the real-world environment and distinguish it from other objects in the real-world environment. In some examples, the one or more depth sensors may allow the device to determine the texture and / or topography of an object in the real-world environment.

[0034] In some examples, the electronic device 201 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around the electronic device 201. In some examples, the image sensor 206 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information of physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some examples, the electronic device 201 uses the image sensor 206 to detect the positioning and orientation of the electronic device 201 and / or the display generation component 214 in the real-world environment. For example, the electronic device 201 uses the image sensor 206 to track the positioning and orientation of the display generation component 214 relative to one or more fixed objects in the real-world environment.

[0035] In some examples, the electronic device 201 includes a microphone 213 or other audio sensor. The electronic device 201 optionally uses the microphone 213 to detect sounds from the user and / or the user's real-world environment. In some examples, the microphone 213 includes a microphone array (multiple microphones) that optionally operate in conjunction, such as to identify ambient noise or locate a sound source in the space of the real-world environment.

[0036] The electronic device 201 includes a location sensor 204 for detecting the location of the electronic device 201 and / or the display generation component 214. For example, the location sensor 204 may include a GPS receiver that receives data from one or more satellites and allows the electronic device 201 to determine the absolute location of the device in the physical world.

[0037] The electronic device 201 includes an orientation sensor 210 for detecting the orientation and / or movement of the electronic device 201 and / or the display generation component 214. For example, the electronic device 201 uses the orientation sensor 210 to track changes in the position and / or orientation of the electronic device 201 and / or the display generation component 214, such as relative to physical objects in a real-world environment. The orientation sensor 210 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0038] In some examples, the electronic device 201 includes a hand tracking sensor 202 and / or an eye tracking sensor 212. The hand tracking sensor 202 is configured to track the position / location of one or more parts of a user's hand, and / or the movement of one or more parts of a user's hand relative to an extended reality environment, relative to a display generation component 214, and / or relative to another defined coordinate system. The eye tracking sensor 212 is configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or extended reality environment and / or relative to the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented together with the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented separately from the display generation component 214.

[0039] In some examples, the hand tracking sensor 202 may use an image sensor 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that captures three-dimensional information from the real world including one or more hands (e.g., one or more hands of a human user). In some examples, the hands may be resolved with sufficient resolution to distinguish fingers and their corresponding positioning. In some examples, one or more image sensors 206 are positioned relative to the user to define a field of view and an interaction space of the image sensor 206, in which the finger / hand position, orientation, and / or movement captured by the image sensor is used as input (e.g., to distinguish from the user's resting hand or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0040] In some examples, eye tracking sensor 212 includes at least one eye tracking camera (e.g., an infrared (IR) camera) and / or an illumination source (e.g., an IR light source such as an LED) that emits light toward an eye of the user. The eye tracking camera may be pointed toward the user's eye to receive reflected IR light from the light source directly or indirectly from the eye. In some examples, both eyes are tracked separately by respective eye tracking cameras and illumination sources, and focus / gaze may be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by respective eye tracking cameras / illumination sources.

[0041] The electronic device 201 is not limited to Figure 2 Rather than being a single component and configuration, the device 201 may include fewer components, other components, or additional components in a variety of configurations. In some examples, the device 201 may be implemented between two electronic devices (e.g., as a system). One or more people using the electronic device 201 are optionally referred to herein as one or more users of the device.

[0042] When creating a cinematic experience, a virtual background can be used as a digitized background, such as a background for chroma key compositing and / or a display on a light emitting diode (LED) wall array. A virtual scene can also be included in immersive virtual content, such as an immersive virtual scene for virtual reality (VR), extended reality (XR), and / or mixed reality (MR) applications, in which virtual assets consume at least a portion of the viewer's view of the physical environment. Editing a virtual scene using traditional methods, especially when the editing process is a collaborative process, can be cumbersome and unintuitive. The present disclosure contemplates methods and systems for improving the efficiency of user interaction with virtual assets included in a scene, such as placing annotations indicating virtual assets that can be added, deleted, edited, or otherwise of interest to a virtual scene editor. One or more of the examples of the present disclosure involve inserting spatial annotations (e.g., annotations with simulated positioning relative to a virtual scene, similar to the placement of physical objects within a physical environment) to draw user attention to the annotations, and facilitate communication between individuals who are simultaneously or asynchronously reviewing a virtual scene and / or annotations associated with the virtual scene. In one or more examples, the device facilitating annotations can communicate with one or more other devices, and the user of the respective device can insert the respective annotations into the scene and / or examine annotations placed by other users of other devices. In one or more examples, the device can determine the context of the user of the device to intelligently place the annotations within the virtual scene. In one or more examples, the device can intelligently select between one or more categories of user annotations to be inserted into the virtual scene.

[0043] Attention is now directed to methods and systems for facilitating annotation of a virtual scene displayed in a three-dimensional environment presented at an electronic device (e.g., corresponding to electronic device 201). As previously described, it will be appreciated that extended reality (XR) editing of a virtual scene improves the efficiency of user interaction when editing, annotating, and reviewing annotations associated with the virtual scene. In addition, it will be appreciated that engaging multiple devices in a communication session to collaboratively edit and / or annotate a virtual scene improves the clarity and efficiency of communication between device users. The present disclosure contemplates examples of methods and systems for editing and / or annotating a virtual scene, thereby improving human-computer interaction when synthesizing a virtual scene.

[0044] For example, an electronic device may display a virtual scene that completely replaces the view of the physical environment, as if the user is physically in a physical equivalent of the virtual scene. In some examples, displaying a virtual scene that replaces the view of the user's physical environment may correspond to displaying the virtual scene at an immersion level greater than a threshold immersion level, which immersion level is further described herein. In some examples, the virtual scene may only partially replace the view of the physical environment, so that a portion of the user's physical environment is still visible to the user, and / or the virtual scene may be displayed at an immersion level (e.g., opacity) less than the threshold immersion level. In some examples, in response to detecting a change in the user's viewpoint in the physical environment (e.g., a change in the user's position and / or orientation), the electronic device may change the perspective of the virtual scene, as if the user is changing the position within the virtual scene. In some examples, user input such as gaze, gesture, or hand movement may be detected together with the context of the user input, and corresponding virtual annotations such as text notes, voice recordings, simulated markers, and / or related media may be added to the virtual scene.

[0045] It will be appreciated that the placement of annotations in the virtual scene and / or the appearance of these annotations may be determined based on the characteristics of the user input. For example, in response to detecting an air gesture performed by a hand, the electronic device may display a positioning indicator that points to and / or otherwise emphasizes a portion of the virtual scene. In another example, in response to detecting user input, the electronic device may determine the context between the user and / or its input and the three-dimensional environment. For example, in response to detecting a user's voice, the electronic device may use one or more natural language processing methods (e.g., algorithms, machine learning techniques, and / or predefined behaviors for annotation placement in the absence of possible targets) to determine possible targets (e.g., specific virtual objects, portions of virtual scenes, and / or some combination thereof) given the annotations included in the voice, and display the annotations at locations corresponding to the possible targets.

[0046] FIG. 3A to FIG. 3LThe example interaction including annotation, editing and inspection of a virtual scene according to an example of the present disclosure is illustrated. FIG. 3A to FIG. 3L The particular order of input, determination, information presentation, and other operations described is exemplary only, and it is contemplated that the order of execution of such operations may differ from the examples explicitly described without departing from the scope of the present disclosure.

[0047] The discussion of the examples that follow will initially focus on FIG. 3A to FIG. 3F The illustrated examples are inputs, "user contexts" and / or operations performed from the perspective of a user of the electronic device. Further examples of the present disclosure focus on also being based on FIG. 3A to FIG. 3F The illustrated examples include similar inputs, user contexts, and / or operations performed at other electronic devices participating in a communication session with the electronic device, as well as operations performed by the user's electronic device in response to obtaining information from other electronic devices. FIG. 3F to FIG. 3H Description of virtual "screen shots" of the examples illustrated, and Figure 3H to Figure 3L A description of a user interface for viewing annotations associated with a virtual scene of the illustrated example. It should be understood that operations performed at the user's electronic device may be performed at other electronic devices, and vice versa.

[0048] Back to Figure 3A , illustrates an example device for visually indicating a target of an annotation according to some examples of the present disclosure. Figure 3A In addition to the user's view of the physical environment, the three-dimensional environment 300 also includes a plurality of virtual objects and / or textures included in the virtual scene displayed via the display 120 of the electronic device 101. FIG. 3A to FIG. 3L , a three-dimensional environment 300 is illustrated from the perspective of the electronic device 101 and additionally from a top-down perspective in a graphical character below the perspective of the electronic device 101. In some examples, the virtual scene is an immersive three-dimensional environment. For example, a user of the electronic device 101 can be in his or her entire physical environment (including Figure 3AThe user is physically moved in a virtual environment 300 (an area of ​​the physical environment outside the end of the housing of the electronic device 101), and the device 101 optionally updates a simulated perspective of a virtual sky, a virtual floor, and an object in response to detecting a change in the user's viewpoint (e.g., the user's position and / or orientation relative to his physical environment), the simulated perspective being similar to the physical perspective of the physical sky, the physical floor, and / or one or more physical objects when the user moves relative to his physical environment. In some examples, the virtual scene included in the three-dimensional environment 300 optionally includes a simulated texture covering a physical representation of the floor of the user's physical environment and / or a virtual floor having a simulated spatial contour (e.g., topography) that is different from the user's physical environment. In addition, the virtual scene may include a simulated atmosphere, such as a virtual sky (e.g., simulating the lower atmosphere at dawn, daytime, dusk, nighttime, etc.). It should be understood that the virtual scene may be any suitable computer-generated environment without departing from the scope of the present disclosure.

[0049] In some examples, the virtual scene is displayed relative to the user's physical environment. The physical environment visible outside the housing of the electronic device 101 may include the physical room occupied by the user 318. In some examples, the virtual scene may be displayed within the display 120, at least partially replacing the view of the representation of the user's physical environment, thereby "consuming" the view of the physical environment. For example, the electronic device 101 may include one or more outward-facing cameras that obtain images of the user's physical environment, and these images may be displayed via the display 120 as if the user was able to directly view the physical environment without the assistance of the electronic device 101. At least a portion or all of such a view of the physical environment may be displayed at corresponding locations of display 120 with an opacity level less than a threshold level (e.g., 0%, 1%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 45%, or 50% opacity), and the virtual scene may be displayed at these corresponding locations with an opacity level greater than the opacity threshold level (e.g., 0%, 1%, 5%, 15%, 25%, 40%, 50%, 60%, 65%, 75%, 90%, or 100% opacity). In some examples, the physical environment may include one or more physical objects in the user's environment, physical individuals in the user's environment, physical walls, physical floors, etc. In some examples, a representation of the user's physical environment may be displayed. For example, Figure 3AThe virtual screen included in the three-dimensional environment 300 in the embodiment may be displayed at least partially semi-transparently, overlaying a representation of the user's physical environment displayed via the display 120 (e.g., images collected by sensors 114a to 114c of the user's physical environment). In some examples, the representation of the user's physical environment includes one or more images of the user's physical environment. In some examples, the electronic device 101 displays a real-time or near-real-time image stream (e.g., a video) of one or more portions of the physical environment corresponding to the "representation" of the user's physical environment.

[0050] As previously described, in some examples of the present disclosure, a virtual scene may include one or more virtual objects. The virtual objects may include digital assets that model physical objects, virtual placeholder objects (e.g., polygons, prisms, and / or simulated two-dimensional or three-dimensional shapes), virtual objects that include user interfaces for applications (e.g., stored in memory by the electronic device 101), and / or other virtual objects that may be displayed within a VR, XR, and / or MR environment. For example, the three-dimensional environment 300 includes a barrel 316, which is optionally a virtual asset displayed at a simulated position within the virtual scene that is similar to the physical positioning and orientation of the physical barrel relative to the viewpoint of the user 318. Similarly, the crate 312 is included in the three-dimensional environment at a simulated position and / or orientation that is different from the positioning of the barrel 316. In Figure 3A 3, buildings 310a and 310b are also included in the virtual scene, which are also virtual assets (e.g., virtual buildings) having positions and orientations relative to the virtual scene and / or the viewpoint of user 318. It should be understood that more, fewer, and / or alternative virtual objects may be displayed without departing from the scope of the present disclosure.

[0051] In some examples, electronic device 101 displays one or more representations of individuals other than user 318 via display 120. For example, representation 302a and representation 302b are included in Figure 3A 300, representing other individuals that are virtually or physically included in the user's 300 three-dimensional environment. For example, representation 302a and / or representation 302b are optionally expressive avatars, such as anthropomorphic avatars, having one or more body parts that can move relative to each other. These body parts optionally include a head, hands, arms, shoulders, neck, legs, fingers, toes, facial features, etc. In some examples, representation 302a and / or representation 302b can be individuals that share the user's physical environment, such as individuals in the user's physical room. As an example of an expressive avatar or a representation of a physical individual that shares the user's 318 physical environment, the illustrated representation 302a includes multiple body parts. Figure 3AThe illustrated representation 302b includes a partially expressive avatar or representation of a user of a computer system that does not physically share the physical environment of user 318. It should be understood that such representations are merely exemplary, and that additional or alternative representations of users of corresponding computer systems may be included in three-dimensional environment 300, and that representation 302a may have one or more characteristics that are similar or the same as those described with reference to representation 302b, or vice versa.

[0052] In some examples, representation 302a and / or representation 302b may be individuals that are not in the user's physical environment but are represented using spatial information. In some examples, electronic device 101 uses spatial information to map portions of the user 318's physical environment to portions of the virtual scene, and / or to map portions of the physical environment of representation 302a and / or representation 302b to portions of the virtual scene. For example, a communication session between electronic device 101, a first computer system used by representation 302a, and a second computer system used by representation 302b may be continuous to facilitate mapping between the physical environments of the respective users of the respective computer systems. In some examples, the communication session includes communication of information corresponding to real-time or near-real-time communication of sounds detected by the computer system (e.g., voice, sounds emitted by the user, and / or ambient sounds). In some examples, the communication session includes communication of information corresponding to real-time or near-real-time movement and / or a request to move a representation (e.g., avatar) corresponding to a user participating in the communication session.

[0053] For example, the first computer system may detect movement of the user corresponding to representation 302a in a physical environment of the user (e.g., different from the physical environment of user 318), and may communicate information indicating the movement through the communication session. Prior to the movement, the first computer system may display the virtual scene relative to the viewpoint of the first computer system (e.g., a position and / or orientation relative to the virtual scene similar to the physical position and / or orientation of the user relative to the physical equivalent of the virtual scene). In response to the movement (e.g., obtaining information indicating the movement from another computer system), the first computer system may update the viewpoint of the user of the first computer system to an updated viewpoint based on the physical movement (e.g., in a certain direction and / or moving a certain movement magnitude), as if the user of the first computer system is physically moving through the physical equivalent of the virtual scene. It will be appreciated that in addition to or as an alternative to detecting the physical movement of the user, the request for such movement may be directed to an input device (e.g., a virtual joystick, a trackpad, a physical joystick, a virtual button, a physical button, and / or another suitable control). The electronic device 101 may receive such information and, in response, may cause the representation 302a to move relative to the virtual scene by a magnitude and / or direction that mimics the physical movement of a user of the first computer system relative to a physical equivalent of the virtual scene. It should be understood that other computer systems (such as a computer system corresponding to the representation 302b and / or the electronic device 101) may also detect similar inputs described with reference to the first computer system and cause their corresponding representations to move within the virtual scene.

[0054] In some examples, a device communicating with the electronic device 101 may cause a user's representation, a virtual scene, and / or a virtual annotation to be displayed. For example, the electronic device 101 may display a virtual annotation received from a computer system (such as a desktop computer or a laptop computer) other than the electronic device 101. In some examples, the computer system may display a view of a virtual scene, such as a representation of a virtual scene on a flat display. In some examples, the computer system may detect an input requesting input pointing to an annotation of the virtual scene, and may communicate an indication of the input to the electronic device 101. In response to receiving the input, the electronic device 101 may display a virtual annotation at a position corresponding to (e.g., the same as) the position indicated in the input within the three-dimensional environment 300 and / or within the virtual scene. Therefore, the electronic device 101 and the computer system may synchronize the annotation within the virtual scene. Both devices may place annotations in the virtual scene, and / or may display an indication of the received annotation based on an indication received from another device. It should be understood that some or all of the operations described herein with reference to the electronic device 101 detecting input for a virtual annotation, displaying a representation of the virtual annotation, and / or recording the content of the virtual annotation can be performed at the computer system, and the electronic device 101 can synchronize the placement of the virtual annotation provided by the computer system with the annotation inserted locally into the electronic device 101 (e.g., in real time or near real time).

[0055] It should also be appreciated that the movement and / or placement of representations of users participating in the communication session may be defined relative to a shared coordinate system, rather than strictly relative to the virtual dimensions of the virtual scene. For example, the electronic device 101 may present a view of the physical environment of the user 318 that does not include the virtual scene, and may display the representations at locations within the view of the physical environment and / or display the movement of the representations within the view of the physical environment. FIG. 3A to FIG. 3L The described examples may occur during a communication session (described herein), and information conveying the location, orientation, audio, and / or other aspects of a physical user and / or information provided by the physical user may be exchanged via the communication session to devices participating in the communication session. It should be understood that, depending on the context, operations described with reference to virtual content displayed relative to a virtual scene may be displayed relative to a representation of the user's physical environment (such as a visual indication of user 318's attention described below).

[0056] In some examples, electronic device 101 displays one or more visual indications indicating user attention within three-dimensional environment 300. For example, electronic device 101 detects a virtual location of a target of user attention 306a (e.g., gaze) and displays visual indication 304c at that virtual location, thereby presenting a visual indication of the portion of three-dimensional environment 300 to which the user's attention is directed. In some examples, one or more parts of the user's body other than the eyes are used to indicate the target of the user's attention. For example, although Figure 3A 304c is not shown, but the electronic device 101 can detect a spatial relationship between a point of contact between two fingers included in the hand 308 (e.g., forming an air pinch gesture) and the electronic device 101. The spatial relationship can be based on a ray projected (not shown) from a portion of the electronic device 101 (such as the center of the electronic device 101), passing through parts of the user's body (e.g., through an air pinch gesture) and extending toward a location within the virtual scene. In some examples, the virtual scene has simulated depth, and the visual indication 304c is displayed at a location based on the user's attention and / or the spatial relationship between the device 101 and the air gesture. For example, due to the user's gaze and / or a ray projected from the device 101 through the air pinch gesture that intersects with a location on the virtual floor, Figure 3A The electronic device 101 in the virtual scene displays a visual indication 304c at the location on the surface of the virtual floor included in the virtual scene.

[0057] In some examples, electronic device 101 may display an indication of the attention of other users. Figure 3A , the user's attention (e.g., gaze) corresponding to representation 302a is directed toward bucket 316. The user's computer system may detect that bucket 316 is the target of the user's attention and may communicate information indicating the target to electronic device 101. In response to obtaining this information, electronic device 101 may display a visual indication of attention, such as Figure 3A Similarly, the computer system corresponding to representation 302b can detect the attention of the user corresponding to representation 302b and can display a visual indication of the user's attention, such as Figure 3A It should be understood that in some examples, in response to detecting that the corresponding user's attention has changed relative to the three-dimensional environment 300 and / or the virtual scene, the corresponding electronic device may convey information to move the attention of the updated attention target. In response to obtaining such updated information, the electronic device 101 may move the visual indication of attention (e.g., move the visual indication 304a and / or the visual indication 304b) to an updated position and / or orientation relative to the content included in the virtual scene according to the updated information.

[0058] In some examples, the electronic device 101 displays one or more visual indications of attention, which illustrate the aggregation of one or more targets of attention over time. For example, the electronic device 101 may display a heat map indicating the location of attention over a period of time (e.g., 1, 3, 5, 10, 15, 30, or 60 seconds, and / or over a period of time when the user is speaking and / or pointing a finger at the three-dimensional environment 300). The electronic device 101 may display a heat map that may encode the duration of the attention using a color gradient and / or saturation level. For example, a dark red portion of the heat map covering the three-dimensional environment 300 may indicate that attention is directed to the corresponding first portion of the three-dimensional environment 300. The dark blue portion of the heat map may indicate that attention is directed to the corresponding second portion of the three-dimensional environment 300 for a relatively shorter time than the first portion of the three-dimensional environment 300. In some examples, light blue, light red, and intermediate shades of additional or alternative colors may indicate that attention is directed to the corresponding one or more portions of the three-dimensional environment 300 over a corresponding time period greater than the dark blue portion and / or less than the dark red portion. It can be appreciated that the electronic device 101 can implement the heat map using a range of saturation, opacity, fill patterns, other colors, and / or some combination thereof to indicate the duration of attention paid to various portions of the three-dimensional environment 300 .

[0059] In some examples, electronic device 101 selectively displays a visual indication of attention (e.g., Figure 3A 304c in the electronic device 101). For example, when an interactive mode (e.g., an editing mode) with respect to a virtual scene is enabled, the electronic device 101 may display a visual indication of attention. In some examples, when the interactive mode is disabled, the electronic device forgoes displaying a visual indication of attention. Similarly, when the interactive mode is enabled, the electronic device 101 may display other visual indications of the attention of other users, and when the interactive mode is disabled, the electronic device 101 may forgo displaying a visual indication of the attention of other users. In some examples, the electronic device 101 displays a visual indication of attention based on user preferences. For example, a user setting specified by the electronic device 101 may allow or prohibit sharing a visual indication of the attention of the user 318 with other users participating in a communication session with the electronic device 101. In some examples, a visual indication of attention may be displayed in response to detecting a visual indication displayed within three-dimensional environment 300 (e.g., a predefined mid-air gesture performed by a user's body, a posture of one or more parts of a user's body, a verbal request to display a visual indication, and / or selection of a virtual and / or physical control (e.g., a button, slider, and / or menu option)) and / or an explicit request to share the visual indication with other devices participating in a communication session with electronic device 101.

[0060] As an additional example, in Figure 3B , the attention of representation 302a is directed to a portion of crate 312 that is not visible from the viewpoint of user 318. Based on determining that the user's attention is directed to an invisible portion of the virtual scene (e.g., as if a physical user were gazing at a portion of a physical crate that is not visible to user 318 from their perspective), the electronic device 101 forgoes displaying the representation of the attention of representation 302a. Additionally or alternatively, the electronic device 101 may display a visual indication of attention via a modified appearance (e.g., a different spatial outline, such as a simulated glow surrounding a portion of crate 312, an arrow with simulated depth that curves behind crate 312, and / or having visual characteristics (e.g., opacity, blur, saturation, and / or simulated lighting effects)) to convey that the target of the user's attention is not currently visible to user 318. It should be understood that the visual indication of attention may be displayed and sometimes omitted from the figure for convenience. For example, the visual indication of attention may be displayed Figure 3B A visual indication in FIG. 302b pointing to a representation of building 310b.

[0061] In some examples, the electronic device 101 detects one or more user inputs associated with a request to display a virtual annotation. In some examples, the one or more user inputs optionally include: Figure 3A In some examples, the one or more user inputs include movement of one or more parts of the user's body. For example, from Figure 3B to Figure 3C , the one or more user inputs include movement of the hand 308 relative to the user's viewpoint and / or relative to the electronic device 101. Figure 2 As described, the electronic device 101 includes one or more sensors and / or communicates with one or more sensors to detect the spatial relationship between the user's viewpoint, one or more parts of the user's body and the virtual scene. For example, the electronic device 101 optionally projects one or more rays from one or more sensors, thereby intersecting with one or more parts of the user's body, and further intersecting with one or more parts of the virtual scene. In some examples, the electronic device 101 detects that the user input is pointing to one or more parts of the virtual scene based on determining that one or more rays correspond to one or more parts of the virtual scene (e.g., virtually intersecting with one or more parts of the virtual scene). In some examples, one or more parts of the user's body include one or more fingers, a part of a finger, a palm, a hand, a wrist, a forearm and / or an arm of the user. In some examples, the electronic device detects and / or determines the location and / or orientation of multiple body parts in the aforementioned one or more body parts, and determines the target of the user input based on the location and / or orientation combination of multiple body parts in the one or more body parts.

[0062] In some examples, one or more user inputs include the user's attention. For example, Figure 3A The electronic device 101 in 3D environment 300 detects the position and / or orientation of one or more eyes of a user (e.g., attention 306a) using one or more imaging sensors such as one or more cameras included in sensors 114a-114c. In some examples, the electronic device 101 detects the position and / or orientation of one or more rays projected from the one or more eyes of the user to the corresponding one or more locations within the three-dimensional environment 300. In some examples, the electronic device 101 detects user input to one or more corresponding locations corresponding to (e.g., intersecting or approaching) the one or more rays projected from the one or more eyes of the user.

[0063] In some examples, one or more user inputs include a user's voice. Figure 3D , audio 368 is detected by the electronic device 101. The audio optionally includes one or more words spoken by the user, which are detected by one or more microphones included in the electronic device 101 and / or communicating with the electronic device. In some examples, the electronic device 101 parses the information and / or communicates the information to one or more external computing devices to determine the content of the user's voice, including recognizing the user's words and / or obtaining a semantic understanding of the voice. For example, the electronic device includes one or more processors configured to perform natural language processing to detect one or more words and determine the possible meaning of the sequence of the one or more words. In some examples, the electronic device 101 additionally or alternatively determines the meaning of the sequence of one or more words based on the user's determined context. Figure 3B Further representatively, and exemplarily, an example is shown in which the user corresponding to representation 302b is speaking, as indicated by the curve indicating audio from representation 302b.

[0064] In some examples, the electronic device 101 determines the context of the user's interaction with the three-dimensional environment 300. For example, when the input is detected, the electronic device 101 determines the context of the user based in part or in whole on the location and / or orientation of the visual indication 304c. For example, because the user's attention is directed toward when the mid-pinch is detected Figure 3A The electronic device 101 detects the mid-air pinch and moves the finger to the virtual floor. Figure 3BThe annotation 320b is displayed at the illustrated location. In some examples, additional or alternative factors are used to determine the user context. For example, in the absence of a mid-air pinch gesture including contact between the user's fingers, the electronic device 101 may display the annotation at a location corresponding to the location where the target of attention is directed in response to detecting the user input requesting the placement of the annotation. For example, the electronic device 101 optionally parses Figure 3D to determine that user 318 is likely referring to crate 312 because the user refers to moving "this crate" and crate 312 is the only virtual object that resembles a physical crate. Therefore, electronic device 101 optionally determines Figure 3D The user context in corresponds to a "crate" or a crate-like virtual object, and crate 312 is optionally determined to correspond to the crate of interest. Figure 3C In response to obtaining Figure 3B Due to the user's attention and / or the voice suggesting the user's context, the electronic device 101 displays the annotation 320e at a location facing the building 310b. It should be understood that in some examples of the present disclosure, "annotation" and "annotation representation" can be used interchangeably.

[0065] In some examples, in response to inserting the representation of the annotation into the virtual scene, simultaneously with the insertion of the representation of the annotation into the virtual scene, and / or after the representation of the annotation is inserted into the virtual scene, the electronic device 101 detects and / or prompts the user 318 with information corresponding to the annotation. For example, the electronic device 101 displays a user interface that prompts the user to provide voice, air gestures, text input (e.g., via a virtual or physical keyboard), movement, attention, and / or other suitable information modalities. Such a user interface may include one or more virtual buttons to start text input, sound recording, movement recording, and / or recording of the user's attention, and / or stop such text input and / or recording. After the text input and / or recording provided by the user 318 is completed, the electronic device 101 may stop the display of the user interface and / or associate the provided information with the corresponding representation of the annotation. In some examples, the electronic device 101 starts recording and / or starts text input in response to inserting the representation of the annotation into the virtual scene without displaying a dedicated user interface.

[0066] In some examples, after inserting a virtual annotation into a virtual scene, the virtual annotation will be associated with the virtual scene and can be viewed again independently of the ongoing communication session. For example, one or more users can exit the virtual scene, and when the virtual scene is displayed again, the electronic device presenting the virtual scene can display the annotation inserted during the previous communication session with or without communicating with the additional electronic device.

[0067] In some examples, the electronic device 101 may map the user's voice to the virtual object based on determining that the user's voice describes an object similar to the virtual object. For example, a voice that mentions a box, a rectangular prism, a cuboid, a container, a basket (e.g., if the virtual object includes an opening on at least one side of the crate 312), etc. may be determined to correspond to the crate 312. Additionally, a voice that mentions a name assigned to the crate 312 (e.g., "Crate 1") may be detected and determined to correspond to the crate 312. In such examples, the electronic device 101 may interpret the pronoun "this" object as a virtual object to which the user 318 directs his or her attention within a threshold amount of time (e.g., 0 seconds, 0.01 seconds, 0.05 seconds, 0.1 seconds, 0.5 seconds, 1 second, 1.5 seconds, 2 seconds, 3 seconds, 5 seconds, 10 seconds, or 30 seconds), a physical object to which the user physically gestures (e.g., pointing, moving his or her fingers and / or hands toward the virtual object, moving his or her lips toward the virtual object, tilting his or her head toward the virtual object, moving his or her arms toward the virtual object, and / or pointing his or her legs and / or feet toward the virtual object), and / or a virtual object within a threshold distance of the user (e.g., 0m, 0.01m, 0.05m, 0.1m, 0.5m, 1m, 1.5m, or 3m). Similarly, speech referring to a barrel, a cylinder, a vat, a bucket, a basin, and / or a keg may be determined to correspond to a virtual object. Figure 3D 316 in the bucket. It should be understood that additional or alternative factors may be envisioned without departing from the scope of the present disclosure. For example, speech indicating "that," "these," "those," "objects there," etc. may be detected by the electronic device 101 and mapped to one or more virtual objects based on a determination of the user context.

[0068] In some examples, electronic device 101 may determine user context based on movement, indication of attention, and / or other factors. Figure 3CIn response to detecting movement, attention, and / or speech of a user corresponding to representation 302a, a computer system corresponding to representation 302a may communicate information indicating the detected movement, attention, and / or speech. In response to detecting the information, an electronic device corresponding to representation 302a may initiate spatial recording, storage, and / or capture information to later present an animation and / or audio recording including an indication of the user's movement, speech, and / or attention. In such examples, recording may be initiated without detecting an explicit input (e.g., actuation of a virtual or physical button, a voice command, an air gesture, and / or another suitable input as further described herein), or may be initiated in response to the explicit input. For example, the electronic device may detect that the user begins talking about crate 312, is looking at crate 312, and / or is moving around crate 312 and / or within a threshold distance of crate, and determines that the user's context is related to crate 312. The recording may continue until the electronic device detects a cessation of speech, a pause in speech, movement beyond a threshold distance from crate 312, a change of attention away from crate 312, and / or until an input (e.g., a quick input) is detected requesting that the recording be stopped. After the recording ends and / or in response to the cessation of the recording, information indicative of the recording may be communicated to other computer systems in real time. In response to obtaining such information associated with the recording, the electronic device 101 may display an annotation (such as 320d) within the virtual scene, which may be later interacted with as further described herein to optionally simultaneously initiate presentation of a spatial representation of the user's speech, an indication of attention, and / or presentation of audio included in the recording.

[0069] In some examples, user context can be used to determine a possible target among multiple potential targets. For example, a user voice indicating "this" building can be detected, and the electronic device 101 can be used to determine whether the voice corresponds to building 310a or building 310b based on the determined user context (e.g., as described with reference to crate 312). For example, when such a voice is detected, the user's attention is directed to or has recently been directed to building 310a. In response to detecting such a voice, the electronic device 101 may display a virtual annotation pointing to building 310a instead of pointing to building 310b. Similarly, a voice referring to "bar" can be detected by the electronic device 101, and due to the name assigned to building 310b and / or the text included in building 310b (e.g., "bar"), the voice is determined to correspond to building 310b.

[0070] In some examples, the user context may be determined to correspond to multiple possible targets, and the electronic device 101 may display annotations pointing to the multiple targets. Figure 3DWhen the viewpoint relative to the virtual scene and / or three-dimensional environment 300 is illustrated, the electronic device 101 may detect a voice pointing to "these" or "those" buildings. In response to detecting the voice, the electronic device 101 may determine that the building 101a and the building 101b are within the viewport of the electronic device 101, and may display annotations pointing to the building 101a and the building 101b and / or visually emphasizing (e.g., displaying the buildings 101a and 101b with simulated halo outlines) the virtual buildings accordingly. The "viewport" of the electronic device 101 is further described herein and may refer to a virtual viewport having a viewport boundary that defines the extent to which the three-dimensional environment (optionally including portions of the virtual scene) is visible to the user.

[0071] In some examples, one or more factors indicative of user context (e.g., voice, air gestures, air postures, gaze, previous interactions with the three-dimensional environment 300, etc.) may be used in combination to determine a possible set of one or more virtual objects that are possible annotation targets. For example, the electronic device 101 may use gaze in conjunction with voice, may ignore one or more factors associated with user context (e.g., air gestures that override gaze and / or voice), and / or may probabilistically determine user context based on placement (e.g., display) of previous annotations in light of previous determinations of user context.

[0072] In some examples, the user context may be determined to be general and not explicitly refer to virtual objects within a three-dimensional environment. Figure 3E Audio 370 generated by user 318 is illustrated, which contemplates a “scene texture update reminder from Robert.” In such an example, the electronic device 101 may determine that the reminder is generally directed to the virtual scene and does not explicitly mention a specific virtual object. The electronic device 101 may determine that the audio 370 generally refers to multiple portions of the virtual scene (e.g., the scene texture may correspond to a virtual texture covering a floor of the virtual scene, one or more textures applied to one or more virtual objects, and / or a virtual texture covering a virtual sky included in the three-dimensional environment 300). Therefore, the electronic device 101 may display an annotation 320h within the three-dimensional environment 300 in response to determining that the user context is generally directed to the virtual scene. Generally placed annotations (such as annotation 320h) may be displayed at a predetermined virtual distance (e.g., 0m, 0.01m, 0.05m, 0.1m, 0.5m, 1m, 1.5m, or 3m) from the viewpoint of the user 318, such as, for example. Figure 3F exemplified.

[0073] In some examples, the user context is determined to be general and / or directional, such as "to my left," "to my right," "in front of me," and / or in a simulated cardinal direction specified by the user's speech. In such examples, the electronic device 101 may display virtual annotations based on a determination of the meaning of the user's speech. For example, the electronic device 101 may parse the user's speech, determine the relative portion of the three-dimensional environment 300 that the speech may refer to relative to the user's 318 viewpoint when the speech is received, and in response to detecting the speech, display and / or place annotations at a predetermined distance from the user's 318 viewpoint toward the relative portion of the three-dimensional environment. For example, a discussion linked to a virtual scene to the left of the user is optionally mapped to portions of the three-dimensional environment 300 to the left of the center of the user's viewpoint (e.g., including Figure 3F 316 and crates 312 in the user's viewpoint). In response to detecting such discussion, the electronic device 101 may place an annotation (e.g., annotation 320h) at a predetermined distance to the left of the center of the user's viewpoint. Additionally or alternatively, discussions linked to the virtual scene to the user's right may optionally be mapped to portions of the three-dimensional environment 300 to the right of the center of the user's viewpoint. In response to detecting speech indicating a portion of the virtual scene "to the user's right," the electronic device 101 may place the annotation at a predetermined distance, optionally toward the right of the center of the user's viewpoint. It should be understood that additional or alternative directions relative to the user's viewpoint and / or simulated cardinal directions may be included as factors in determining user context and / or placement of annotations (e.g., behind the user's viewpoint, "north" of the user's viewpoint, and / or above or below visible via the viewport when the user 318 has a particular viewpoint).

[0074] In some examples, the electronic device 101 displays and / or places virtual annotations within the virtual scene based on the determined user context. Figure 3B In the virtual scene, the electronic device 101 is Figure 3A The annotation 320b is displayed at the location indicated by the visual indication 304c in Figure 3A Similarly, the electronic device 101 responds to obtaining an indication of a gaze and / or air gesture target (given by Figure 3A The information indicated by the visual indication 304a in FIG. Figure 3B Note 320a in the Figure 3A Similarly, the electronic device 101 responds to obtaining an indication of a gaze and / or air gesture target (given by Figure 3A The information indicated by the visual indication 304b in FIG. Figure 3B Note 320c in Figure 3A302b in the representation 302b is provided by a computer system associated with the representation 302b in the virtual scene. In some examples, based on determining that information is obtained including a request to insert an annotation at a location that is not currently visible to the user 318 in the virtual scene, the electronic device 101 presents spatial feedback indicating the location of the requested annotation. For example, the spatial feedback includes audio generated using one or more characteristics to mimic the perception of a physical audio source placed at a physical equivalent of the location of the requested annotation in the physical environment of the user playing the audio feedback (e.g., a ringtone, voice, and / or another notification noise). In some examples, the spatial feedback additionally or alternatively includes a visual indication, such as a simulated glowing effect that illuminates one or more portions of the user's viewport that are close to and / or generally toward the location of the requested annotation.

[0075] In some examples, virtual annotations are displayed based on the target of the user's gaze. Figure 3B In , it is indicated that 302b is gazing at the second floor of building 310b and initiating speech. Figure 3C In FIG. 1 , the electronic device 101 displays an elevated annotation 302e near the second floor of the building 310b based on determining that the user's context is associated with the second floor of the building 310b. In some examples, the positioning of the annotation is explicitly indicated by the user's movement. For example, in Figure 3B , the electronic device 101 detects attention and / or an air gesture (e.g., pinch) directed to the right of the crate 312, and Figure 3B to Figure 3C The moving path of the hand 308 of the user 318 holding the air gesture is detected. Figure 3C , the electronic device 101 displays an annotation 320f, i.e., a simulated marker described herein, starting from the right side of the crate 312, along the movement path of the hand 308, and terminating at a location corresponding to the projection of a ray between the electronic device 101 and the hand 308 into the virtual scene when the air pinch is released.

[0076] exist Figure 3C , after detecting information indicating that the spatial recording of the movement of representation 302a has stopped, annotation 320d is displayed. Figure 3C The positioning of annotation 320d in corresponds to the positioning of representation 302a when recording was initiated, but can be other locations, such as the positioning of representation 302a when recording was terminated and / or an explicitly defined positioning (e.g., via voice commands, air gestures (e.g., air pinch and movement of air pinch) to move annotation 320d, and / or in light of determined user context, such as determining that the user's movement is close to a virtual object that representation 302a is talking about, pointing to, and / or gazing at).

[0077] In some examples, the virtual annotation is displayed at a location based on the user context (such as speech). Figure 3D , an annotation 320g is displayed in response to audio 368 provided by user 318 and detected by electronic device 101. Annotation 320g may be displayed such that the annotation draws the user's attention to a virtual object and / or a portion of the virtual scene associated with audio 368, such as including an arrow pointing toward and approaching crate 312. Similarly, in Figure 3E In response to the electronic device 101 detecting the audio 370, an annotation 320h is displayed at a predetermined distance directed toward the floor of the virtual scene as further described herein.

[0078] In some examples, the electronic device 101 determines the location and / or orientation of the virtual annotation based on a determination of the user context and / or based on the virtual content included in the virtual scene. For example, annotations 320a, 320b, 320c, and 320d are displayed as pointing downward toward a virtual object and / or a portion of the virtual scene, which may optionally be the default orientation of such annotations. Additionally or alternatively, the annotation may point downward toward the virtual object to further indicate that a particular aspect of the virtual scene is noteworthy, and / or toward a particular location on a surface (e.g., a virtual floor) included in the virtual scene. As an additional example, Figure 3C The annotation 320f in extends from the front face of the crate 312 in accordance with movement of the hand 308 that defines a context associated with the annotation 320f involving a well-defined path. In some examples, the visual characteristics of the annotation 320f are based on a currently selected analog marking tool. For example, the width, color, opacity, and / or additional characteristics of the annotation 320f optionally simulate the appearance of a pen, pencil, brush, wedge-tipped marking tool, and / or spray paint when the corresponding analog marking tool is currently selected. For example, when a wedge-tipped marking tool is selected when the user 318 generates the annotation 320f, the visual appearance of the annotation 320f may be relatively thicker and / or darker, compared to the relatively thin and / or diffuse appearance of the annotation 320f when the analog pencil is selected. Figure 3D , annotation 320g is displayed with an orientation pointing toward crate 312. In some examples, the annotation orientation may vary based on the surrounding virtual content. For example, based on a determination that annotation 320g will occlude another virtual object and / or may present a simulated occlusion of another annotation, electronic device 101 may display annotation 320g rotated away from a default orientation and / or translated away from a default positioning.

[0079] Additionally or alternatively, the annotation may lack precise indications, such as arrows pointing to specific locations within the virtual scene. In such examples, the annotation may be displayed by covering, surrounding, and / or extending from a virtual object of interest, and the orientation of the virtual object may be modified to accurately draw attention to a portion of the virtual object and / or a portion of the virtual scene of interest. For example, in response to detecting a voice pointing to a wall included in the building 310a, the electronic device 101 may display a simulated glow and / or lighting effect that illuminates the wall, extends along one or more dimensions of the wall. Additionally or alternatively, a halo and / or circular annotation that surrounds the base of the virtual object of interest and / or hovers above the virtual object of interest may be displayed. It should be understood that the spatial outline, visual appearance, positioning, and / or orientation of the representation of the annotation may be the same or different from that described herein without departing from the scope of the present disclosure.

[0080] In some examples, the representation of the annotation is displayed by one or more visual characteristics that indicate the author of the annotation. Figure 3F , multiple annotations are displayed within a virtual scene, including fill patterns corresponding to a particular user (such as user 318), a user corresponding to representation 302a, and / or a user corresponding to representation 302b. Additionally or alternatively, the annotations may be displayed by color, text, outline, shape, size, displayed by simulating a glow effect, and / or displayed simultaneously with additional visual indications (e.g., with initials, names, icons, electronic communication addresses, and / or photos) that are separate from the annotations corresponding to the authors of the annotations. For example, annotations 320c and 320e may correspond to representation 302b and, therefore, may be displayed by the same fill pattern. For similar reasons, annotations 320a and 320d corresponding to representation 302a may also be displayed by the same fill pattern. Additionally, annotations 320b and 320g are displayed by simulating a glow effect with the annotations corresponding to the author of the annotations. Figure 3F In some examples, the annotations may be displayed with visual characteristics that indicate the commonality of the annotations and / or further distinguish such annotations from other annotations. For example, in Figure 3E, annotation 320h is displayed with a unique fill pattern because, although annotation 320h is associated with user 318, the audio 370 of user 318 is directed toward the entire virtual scene, rather than toward a specific portion of the virtual scene and / or virtual objects. In some examples, the visual appearance of the representation of the annotation indicates the type of annotation. For example, the shape of the annotation, the icon included in the annotation, the text included in the annotation and / or displayed near the annotation, optionally depends on the corresponding category of the annotation. In addition to or as an alternative to a pointing indicator that draws attention to a portion of the virtual scene, speech, for example, is optionally represented by an icon that includes one or more sound waves. As another example, a spatial recording of a user's movements is optionally represented by a screenshot and / or icon of the user's representation. It should be understood that reference Figure 3F The description of the visual appearance and / or visual characteristics of the described annotations is exemplary only and may be applied to reference FIG. 3A to FIG. 3L Description of the note.

[0081] In some examples, electronic device 101 presents information associated with the annotation in response to detecting user input. Figure 3C , multiple indications of attention directed to various representations of annotations within a virtual scene are illustrated. It should be understood that the user input requesting presentation of information associated with the annotation may have one or more characteristics similar to or the same as one or more characteristics of other user inputs described herein. For example, attention 306c is directed to annotation 320b, and attention 306d is directed to annotation 320c. In response to detecting such user input, the electronic device 101 Figure 3D Text 322 and text 324 are displayed in a virtual scene in . The text may include text entered using a virtual or physical keyboard, dictation provided by a user requesting input of such text, and / or simulated handwriting provided via a stylus on a touch-sensitive or non-touch-sensitive surface detected by the electronic device 101 and / or the stylus.

[0082] In some examples, the electronic device 101 initiates playback of the recording in response to a user input directed to an annotation. Figure 3C Attention 306e in the electronic device 101 plays Figure 3D 302b, the audio 326 corresponds to a recording of the user's speech corresponding to representation 302b. In some examples, the audio is generated with a simulated spatial quality as if the audio came from the positioning of annotation 320e. In some examples, the recording is a spatial recording that includes a display of the movement of the user's representation. For example, in Figure 3CIn FIG. 3 , the electronic device 101 detects attention directed to the annotation 320 d. In response to detecting the attention, the electronic device 101 may initiate display of a virtual representation corresponding to the representation 302 a moving throughout the virtual scene, thereby animating the movement of the virtual representation. Figure 3D In FIG. 3 , representation 328 is moved from the location of annotation 320d, mimicking the movement of representation 302a when the recording was captured. Figure 3E , representation 328 continues to move throughout the virtual scene, and the electronic device 101 displays animations corresponding to the user's movements and gestures.

[0083] exist Figure 3D In FIG. 1 , a visual indication 330 is displayed and included in the spatial recording, thereby illustrating the target of the user's attention when recording. Figure 3D In the example, audio corresponding to the recorded audio provided by representation 302a is generated and modified (e.g., using one or more filters, digital attenuators, etc.) to present the spatial quality of the audio as if the user corresponding to representation 302a were speaking from their recorded position relative to the virtual scene. It should be understood that similar to the changes described with reference to the virtual scene, the viewpoint of user 318 can be changed relative to three-dimensional environment 300, and the animation can be changed in response to the change in the viewpoint (e.g., relative to the orientation of user 318 and locked relative to the virtual scene) to simulate the appearance of representation 302a moving and speaking relative to the virtual scene.

[0084] In some examples, the appearance of representation 328 is different from the appearance of representation 302a. For example, representation 328 can be displayed with a different set of one or more visual characteristics and / or can be displayed with an avatar having a different spatial profile. As an example, representation 328 can be a wireframe model animated to convey the movement of linear limbs and polygonal appendages, rather than a realistic avatar illustrating details of the user of representation 302a (e.g., the user's facial features, skin color, musculature, and other characteristics). In some examples, representation 328 has one or more characteristics that are similar or identical to representation 302a.

[0085] Additionally or alternatively, the spatial record may include an animation showing the user's attention to the representation. Attention 306b may include, for example, a user's gaze directed toward annotation 320d, an air gesture (e.g., an air pinch) directed toward annotation 320d, a voice command directed toward annotation 320d, and / or another factor that determines the user's goal and / or context corresponding to representation 328.

[0086] In some examples, electronic device 101 may capture a virtual screenshot of the virtual scene. Figure 3FIn the example, the electronic device 101 detects a user input requesting the capture of a virtual screenshot. The user input is optionally directed to a mechanical button, a circuit (e.g., a touch-sensitive surface and / or an electromechanical crown button that can be rotated and / or pressed), a voice command, an air gesture performed by the user's hand 308 (e.g., air pinching, air pointing of one or more fingers, one or more fingers clenched into a fist and / or one or more fingers spread apart from each other), a selection of a virtual button (e.g., via an air gesture), and / or another suitable input. In response to detecting the user input, the electronic device 101 may initiate the display of a virtual screenshot that captures at least a portion or all of the content visible via the viewport of the electronic device 101. In some examples, the virtual screenshot strictly / only includes a virtual scene. In some examples, in addition to or as an alternative to the virtual scene only, the virtual screenshot includes a representation of the user's physical environment. In some examples, in response to the user input, an animation is displayed. For example, in response to the user input, the entire virtual scene visible via the viewport is optionally captured and the animation is displayed by gradually changing the zoom ratio (e.g., zooming out and / or maintaining the aspect ratio of the screenshot) until it is consistent with the image. Figure 3F The examples are shown similarly or identically.

[0087] exist Figure 3G In response to detecting Figure 3H For example, in Figure 3G In the embodiment, the electronic device displays a screenshot 334 within the three-dimensional environment 300. The screenshot 334 is optionally a virtual object, such as a two-dimensional virtual object and / or an approximately two-dimensional virtual object. For example, the approximately two-dimensional object optionally includes at least two parallel two-dimensional surfaces and a simulated depth extending between the parallel two-dimensional surfaces, so that the virtual object is partially visible when the user's viewpoint is perpendicular to the normal of the two-dimensional surface. In some examples, the screenshot 334 includes or is detected Figure 3H 318. The screenshot 334 includes images of portions of the virtual scene that are visible when the user inputs in . For example, the screenshot 334 includes representations of the barrel 316, crate 312, and buildings 310a and 310b, the virtual sky, and the virtual floor included in the virtual scene, and does not include additional virtual objects such as annotations and / or simulated markers visible within the virtual scene outside of the screenshot 334. In some examples, the screenshot 334 may include representations of annotations and / or simulated markers visible within the virtual scene. In some examples, the screenshot 334 is displayed at a predetermined location relative to the viewport of the electronic device 101 (such as relatively centered within the user's viewpoint) and / or at a predetermined distance from the viewpoint of the user 318.

[0088] In some examples, the electronic device 101 facilitates annotation of the screenshot 334. For example, Figure 3G to Figure 3H , the electronic device 101 detects the movement of the air gesture, and in response, displays an annotation 336 covering the screenshot 334. In some examples, a visual indication of the user's attention is displayed simultaneously with the screenshot 334, covering the screenshot 334. In some examples, the annotation 336 has one or more characteristics that are similar or identical to those described with reference to the annotation displayed within the virtual scene but not included in the screenshot 334. Additionally or alternatively, the electronic device 101 may detect a user context associated with and / or pointing to the screenshot 334 similar or identical to that further described herein, and may display an annotation covering the screenshot. In some examples, the annotations included in the screenshot 334 are not included in the virtual scene, except for those included in the screenshot 334. In some examples, the annotations included in the screenshot 334 are automatically filled in the virtual scene. Thus, the electronic device 101 is able to translate the virtual annotations and / or virtual markers applied to the two-dimensional screenshot 334, and is able to insert similar annotations relative to the virtual scene by spatially determining the correspondence between the screenshot 334 and the virtual scene. In some examples, in response to detecting a selection input (e.g., an input similar or identical to those described herein, such as including an air gesture), the electronic device 101 visually distinguishes the content included in the screenshot 334, such as by visually emphasizing (e.g., a modified color, translucency level, simulated glow effect, saturation, arrows, and / or other positioning indicators) to display the annotation and / or another portion of the screenshot.

[0089] In some examples, electronic device 101 may export a virtual screenshot. For example, virtual screenshot 334 may be communicated to another electronic device participating in communication with electronic device 101 and / or transmitted to another device similar to or different from electronic device 101. For example, a flat screen screenshot may be transmitted and displayed via a flat display (such as a touch screen included in a mobile phone, laptop computer, and / or head-mounted device capable of displaying immersive virtual content within an at least partially virtual three-dimensional environment).

[0090] In some examples, the visual prominence of screenshot 334 may change in response to detecting a change in the viewpoint of user 318. For example, from Figure 3H to Figure 3I , the electronic device 101 detects that the user has radially rotated relative to the screenshot 334 to Fig. 3I In response, the electronic device 101 displays Fig. 3I 338 in FIG. 1 , optionally maintaining its positioning relative to the virtual scene. Fig. 3IThe electronic device 101 in the embodiment may modify one or more visual characteristics of the screenshot 338 to convey that the user is at a relative offset angle relative to the plane of the screenshot 338. For example, in response to detecting a change in viewpoint, the screenshot may be dimmed, blurred, transparent, display or not display outlines, and / or desaturated. In some examples, the visual characteristics of the screenshot 338 are configured to gradually block content included in the screenshot 338 and / or make the content less visible, and / or make the virtual scene more visible, in response to detecting a change in viewpoint that increases the visual quality of the content, respectively, in the screenshot 338. Figure 3H and Fig. 3I The angle formed between a vector extending through the center of the viewpoint of user 318 and a vector extending perpendicular to the surface of screenshots 334 and / or 338 .

[0091] Figure 3J Additional or alternative visual processing applied to screenshot 338 is shown, wherein the virtual scene is at least partially visible through screenshot 338. In some examples, based on determining that the angle between the screenshot and the user's viewpoint exceeds a threshold angle (e.g., 5 degrees, 10 degrees, 20 degrees, 30 degrees, 45 degrees, 60 degrees, 75 degrees, or 80 degrees), the electronic device 101 stops displaying the screenshot 338, or displays a placeholder virtual object (such as a boundary) that does not include the screenshot. In some examples, based on similar changes to the screenshot itself, the content included in the screenshot 338 is gradually made more visible or less visible relative to the virtual scene.

[0092] In some examples, the electronic device 101 displays a user interface intended to view an overview of and interact with annotations associated with the virtual scene. Fig. 3I In the embodiment, the electronic device 101 detects input directed to a virtual button, including an air pinch gesture performed by the hand 308. In response to detecting the input, the electronic device 101 displays Figure 3J The user interface in 348. Figure 3J In , the user interface 348 includes a list of annotations present in the virtual scene, the list of annotations being organized by category or type of annotation. Figure 3JIn the user interface 348, the user interface 348 includes a sorted list of voice recordings (e.g., "Voice Memos"), placed annotations and / or simulated markers (e.g., "Marks"), spatial recordings of a user's representation (e.g., "Motion Captures"), and / or virtual screenshots that have been inserted into a virtual scene (e.g., "Screenshots"). In response to detecting user input (e.g., a voice command, an air gesture, attention, contact with a trackpad, and / or selection of a virtual or physical button), the electronic device 101 may present information associated with the annotation. For example, as further described herein, in response to detecting a selection input of a representation of an annotation included in the user interface 348 (e.g., "Voice Memo A," "Mark A," "Motion Capture A," "Screenshot A," etc.), the electronic device 101 may display text, generate audio, initiate playback of a spatial recording, and / or perform some combination thereof.

[0093] In some examples, in response to selection of a particular representation and / or multiple representations, the electronic device 101 ceases to display the user interface 348 and / or visually emphasizes the corresponding representation of the annotation within the virtual scene, such as Figure 3J Additionally or alternatively, in response to such a selection, the electronic device 101 may temporarily stop displaying other representations that were not selected to visually focus attention on the annotation of interest.

[0094] In some examples, representations of annotations included in user interface 348 may have one or more visual indications (e.g., icons, visual characteristics, fonts, etc.) included in user interface 348 to convey the author of the corresponding annotation, similar to as described with reference to other virtual annotations herein.

[0095] exist Figure 3J In the example, the electronic device 101 detects an input directed to a virtual button 306e (e.g., "Author"), which corresponds to a virtual button of a changeable user interface designed to view and interact with an overview of annotations associated with a virtual scene. In response to the input, the electronic device 101 displays Figure 3K 342. User interface 362 includes a similar or identical list of annotations associated with the virtual scene as illustrated in user interface 342, organized by the user who provided the annotations rather than by the category of the annotations. For example, a title 358 (e.g., "User B") may correspond to representation 302b, and one or more annotations listed under title 358 may include annotations inserted into the virtual scene by representation 302b. Similarly, a title 360 ​​(e.g., "User C") may correspond to user 318, and one or more annotations listed under title 360 ​​may include annotations inserted into the virtual scene by user 318.

[0096] exist Figure 3K In FIG. 1 , the electronic device 101 detects an input directed to a virtual button 306f (e.g., “time”), which corresponds to a virtual button that can display a user interface for viewing and interacting with an overview of annotations associated with a virtual scene organized in chronological order. In response to the input, Figure 3L In the example, electronic device 101 displays user interface 364. As indicated by title 366, the list of annotations in user interface 364 can be organized in chronological order (e.g., in forward or reverse chronological order). Figure 3B The "marker A" corresponding to the annotation 320a in is first inserted into the virtual scene, so the "marker A" is listed first in the user interface 364. In some examples, the user interface presenting an overview of the annotations of the virtual scene can be displayed in a manner different from explicit illustration.

[0097] In some examples, a top view of a three-dimensional environment and / or virtual scene is displayed by the electronic device 101. For example, a graphic character illustrating the three-dimensional environment 300 may be displayed by the electronic device 101 in response to detecting a suitable user input requesting display of a user interface and / or virtual object including the graphic character. In some examples, a virtual object including a graphic character may be displayed in a similar or identical manner as described with reference to the virtual screenshots herein. In some examples, the electronic device 101 may detect input pointing to the virtual screenshot, such as a selection of a representation of an annotation, and may present information similar or identical to that described with reference to annotations 320a, 320b, 320c, 320d, 320e, 320f, 320g, and / or 320h. For example, the electronic device 101 may generate audio from a voice recording, may display text, may display a visual indication of attention, and / or may display the movement of the user's representation within the entire virtual scene from a top-down perspective corresponding to the user's recorded movement. In some examples, the graphic character is displayed simultaneously with other virtual content, including a virtual scene, one or more virtual screenshots, virtual objects, representations of annotations, and the like.

[0098] Figure 4 A flowchart illustrating an example process for interaction including annotation, editing, and inspection of a virtual scene according to an example of the present disclosure is illustrated. Figure 4In some examples, the method 400 may be performed at a computer system in communication with one or more input devices and a display generation component. In some examples, when the computer system is participating in a communication session including one or more participants, and when a virtual scene is displayed from a viewpoint of a user of the computer system within a three-dimensional environment (402a), the computer system obtains first information (402b) via one or more input devices, the first information including a request to display a first annotation at a location within the three-dimensional environment, wherein the correspondence between the first annotation and the corresponding location within the three-dimensional environment is determined based on the context of the user's interaction with the three-dimensional environment. In some examples, in response to obtaining the first information (402c), and based on determining that the first annotation corresponds to the first location within the virtual scene, the computer system displays (402d) a first representation of the first annotation at the first location. In some examples, in response to obtaining the first information (402c), and based on determining that the first annotation corresponds to a second location in the virtual scene, the computer system displays (402e) the first annotation at the second location.

[0099] Some examples of the present disclosure relate to virtual three-dimensional models of virtual scenes. For example, FIG. 5A to FIG. 5H The example interaction with the user interface according to some examples of the present disclosure is illustrated, and the user interface is used to interact with the virtual scene. By using the model and / or user interface as illustrated in the above figures to facilitate the inspection of the virtual scene and the interaction with the virtual scene, the electronic device 101 improves the efficiency of the user's interaction with the virtual scene.

[0100] In some examples, electronic device 101 displays a representation of a virtual scene within a three-dimensional environment and multiple selectable options for interacting with the representation of the virtual scene. For example, model 505 is a virtual object that includes a virtual asset corresponding to the virtual scene. In some examples, model 505 is part of multiple selectable options for inserting annotations into the virtual scene and / or into model 505, and / or is displayed simultaneously with the multiple selectable options. For example, menu 502 and menu 520 include selectable options that can be selected to start a mode for inserting annotations and / or reviewing annotations, respectively.

[0101] Some examples described herein involve selectable options that can be selected to cause some behavior or operation to be performed. It should be understood that in such examples, the electronic device 101 can detect input pointing to a specific selectable option, and in response to detecting the selectable option, the mode can be activated, the mode can be disabled, and / or an operation can be initiated (e.g., displaying a menu and / or selectable options, activating or displaying virtual content, switching selection of a virtual tool, initiating playback of virtual content, and / or performing some combination thereof). It should be understood that the aforementioned inputs can have one or more characteristics similar to or the same as other inputs described herein, such as air gestures and / or input pointing to a peripheral computing device (such as a stylus and / or an electronic pointing device for guiding a cursor according to the positioning of the electronic device).

[0102] Figure 5A The menu 502 in includes a plurality of selectable options for inserting simulation markup into the model 505. For example, selectable option 510 may be selected to place a text-based annotation. Figure 5A The height and / or dimensions of the selectable option 510 in FIG. 5 indicate that the selectable option 510 is currently selected. FIG. 5B to FIG. 5F As illustrated, the electronic device may detect input that initiates annotation placement when selectable option 510 is selected, and anchor the positioning of the annotation at a selected location within model 505, such as at a location to which a user's attention was directed when the input was detected.

[0103] Menu 502 may include additional or alternative selectable options. For example, tool 508 corresponds to a simulated erase tool. When selected, electronic device 101 may detect input for an annotation within model 505 and may delete the annotation or a portion thereof. Selectable options 512 and 514 may correspond to simulated writing instruments, such as simulated markers, pens, pencils, highlighters, and / or some combination thereof. Selectable option 516 may correspond to a simulated ruler and, when selected, may initiate display of a ruler that may be positioned within three-dimensional environment 500 and / or model 505. When displayed, the simulated markers may be aligned along the dimensions of the ruler.

[0104] The selectable options 518 may correspond to a plurality of selectable options, each of which corresponds to a different visual characteristic, such as the color, opacity, fill pattern, saturation, and / or some combination thereof, of the annotation. When a particular selectable option in the selectable options 518 is selected, the annotation inserted into the virtual scene represented by the model 505 is displayed by the visual characteristics of the selected selectable option in the selectable options 518. For example, when a corresponding color selectable option in the selectable options 518 is selected, the simulated handwriting may be displayed by blue, red, or green. Additionally or alternatively, the color of the virtual pin corresponding to the selectable option 512 may be displayed by the color of the selected selectable option in the selectable options 518.

[0105] Menu 521 may include one or more selectable options that, when selected, may change the currently selected interaction and / or review mode of the virtual scene. For example, selectable option 522 may correspond to a review playback operation. In response to detecting the selection of selectable option 522, electronic device 101 may initiate the operation described in reference to FIG. Figure 3D 524 may correspond to display of a timeline representing the progress of the insertion of annotations. For example, a user of electronic device 101 may detect input selecting selectable option 524 and, in response, may initiate display of a timeline of the insertion of annotations into the virtual scene over a period of time (e.g., as shown in FIG. 525 ). Figure 5H exemplified).

[0106] Menu 521 may also include selectable options 526, 528, and 530, which may be selected to enable interaction modes associated with the virtual scene represented by model 505. For example, in response to detecting input selecting selectable option 526, electronic device 101 may initiate display of menu 520 and / or may enable annotation insertion mode. In response to detecting input selecting selectable option 528, electronic device 101 may initiate display of a list of annotations (e.g., such as Fig. 5F 505), the list of annotations is associated with the virtual scene represented by the model 505. In response to detecting the selection of the selectable option 530, the electronic device 101 may initiate the display of the menu (e.g., Figure 5G exemplified), the menu includes selectable options for selecting a view of the virtual scene.

[0107] The menu 521 may also include a selectable option 534 that may be selected to initiate recording of a potential collaborative inspection and / or annotations of the virtual scene and / or model 505. For example, in response to detecting an input selecting the selectable option 534, the electronic device 101 may initiate recording of annotations and / or comments provided via a microphone and directed toward the virtual scene. The recording may span a period of time, and the electronic device 101 may facilitate insertion of the annotations into the virtual scene and / or model 505. While the recording is ongoing, the electronic device 101 may detect an input to stop recording, and in response, may stop recording, thereby defining an endpoint of a "review session" defined by the time of the recording. After recording the review session, the electronic device 101 may load the review session and facilitate playback of the review session (e.g., using selectable options 522 and 524).

[0108] In some examples, when displaying menu 502 and / or 521, electronic device 101 may simultaneously display selectable option 540. Electronic device 101 may detect input selecting selectable option 540, and in response, may initiate display of a virtual scene, the virtual scene being represented by model 505 at a greater than 3D environment 500. Figure 5A The immersion level in the virtual scene may be represented by the immersion level of the immersion level in the electronic device 101. It should be understood that the immersion level may include the extent to which the virtual content included in the virtual scene occupies the viewport of the electronic device 101. For example, the scale of the virtual scene may increase while the scale of the model 505 remains unchanged. Additionally or alternatively, the scale of the virtual scene and / or the scale of the model 505 may be increased (and / or the device 101 may stop displaying the model 505 and instead display the virtual scene occupying the viewport of the electronic device 101). For example, the virtual scene may occupy a larger portion of the viewport of the electronic device 101 than before the optional option 540 is selected. For example, the virtual scene may be displayed by completely consuming the viewport of the electronic device 101, and the user's viewpoint relative to the virtual scene may be changed. The updated viewpoint relative to the virtual scene may be similar to as if the user were standing in the physical equivalent of the virtual scene. Therefore, selecting the optional option 540 may cause the electronic device 101 to view the virtual scene in a more realistic manner than before the optional option 540 is selected. Figure 5A s to display an immersive view of the virtual scene at a larger scale, making it easier to examine more closely what is included in the virtual scene.

[0109] exist Figure 5B , electronic device 101 detects input represented by hand 538 forming an air pinch gesture. Attention 536 may correspond to a target of the user input, such as a location on the ground of model 505. Since selectable option 510 is selected, electronic device 101 initiates anchoring of the annotation within model 505. For example, in Figure 5C In response to detecting Figure 5B The display of visual indication 541 is initiated by inputting as shown. FIG. 5B to FIG. 5D , the electronic device 101 detects that the mid-air pinch gesture formed by the hand 538 is maintained. FIG. 5C to FIG. 5D , the electronic device 101 detects the movement of the hand 538. In response to detecting the movement of the hand 538, the electronic device 101 may initiate Figure 5D The guide wire 542 shown has a first end anchored to a Figure 5B At the location indicated by visual indication 541 in .

[0110] In some examples, the length and / or orientation of the guide line changes based on the movement of hand 538. For example, the guide line can move to extend toward a point defined by the mid-air pinch gesture. While the mid-air gesture is maintained, the end point of the guide line (e.g., the unanchored end of the guide line) can move in one or more directions and / or move one or more distances that are similar or the same as the component of the movement of hand 538. Thus, the guide line can be scaled up, down, and / or oriented based on the movement of hand 538.

[0111] exist Figure 5E In the embodiment, the electronic device 101 detects the cessation of the mid-air pinch formed by the hand 538. In response to detecting the cessation of the mid-air pinch, the electronic device 101 displays the anchored Figure 5D Annotation 544 at the second end of the guide line described. The second anchor point may correspond to the location where electronic device 101 detects that the guide line ends when the air pinch stops. In some examples, annotation 544 may include text and / or images provided by a user of electronic device 101 (e.g., typed text, voice-to-text input, and / or an image inserted into annotation 544). In some examples, electronic device 101 prompts the user to insert content into annotation 544 in response to detecting that the air pinch stops.

[0112] exist Figure 5E In the embodiment, the electronic device 101 detects input pointing to the selectable option 528. In response to detecting the input, the electronic device 101 may initiate display of the annotation list 546, such as Fig. 5F As shown. In some examples, annotation list 546 may include a list of annotations inserted into the virtual scene corresponding to model 505. In some examples, annotation list 546 may correspond to a single review session. In some examples, annotation list 546 may include annotations from multiple review sessions, each review session involving reviewing the virtual scene represented by model 505. For example, list 546 includes multiple list elements, which include annotations 546a (e.g., "John S. just now can we add more boxes?"), 546b (e.g., "John S. 10 minutes let's add more props here"), 546c (e.g., "Lulu W. 22 February add horses to the west end"), and 546d (e.g., "Lulu W. 22 February lighting needs improvement").

[0113] like Fig. 5FAs shown, annotations 546a to 546d can be displayed respectively by a visual indication of the author of the annotation (such as an image representing the author and / or the author's name). Additionally, annotations 546a to 546d include information indicating the time when the annotation was inserted. For example, annotation 546a corresponds to annotation 544 that was most recently entered by the user of electronic device 101 (e.g., "just now").

[0114] In some examples, in response to detecting input pointing to selectable option 528, electronic device 101 displays annotation markers 548a to 548d. It should be understood that annotation markers 548a to 548d may correspond to annotations 546a to 546d, respectively (e.g., annotation marker 548a corresponds to annotation 546a, annotation marker 548b corresponds to annotation 546b, and so on). Annotation markers 548a to 548d respectively indicate where the annotation is inserted into the virtual scene, and each annotation marker may be included in model 505 and / or overlay the model.

[0115] exist Fig. 5F In the embodiment, when attention 550 is directed to selectable option 530, electronic device 101 detects input provided by hand 538b. In response to detecting the input, electronic device 101 displays menu 552, such as Figure 5G In some examples, menu 552 includes a plurality of selectable options, including selectable options 552a to 552c, respectively associated with views of model 505. For example, currently selected Figure 5G , and displays a representation of the annotation by overlaying the model 505 .

[0116] Optional option 552b may correspond to a pre-visualization viewing mode of the model 505, which, when selected, may cause the electronic device 101 to display virtual content used to create and / or use the model 505. For example, the virtual content may include text, images, models, video shot lists, and / or storyboards related to the content, which may be generated using the virtual scene represented by the model 505. Optional option 552c may correspond to a reference image viewing mode, which, when selected, may cause the electronic device 101 to display reference images and / or two-dimensional and three-dimensional objects used as visual references to create content (such as creating virtual models of physical objects, textures, and / or creating animations).

[0117] exist Figure 5G, when attention 554 is directed to selectable option 522, electronic device 101 detects input from hand 538b. In response to detecting the input, electronic device 101 initiates display of a user interface for a review session. Additionally or alternatively, electronic device 101 may initiate playback of the review session. Review session playback may, for example, include displaying annotation markers, annotations, displaying media, playing back audio, etc., at a time corresponding to when such virtual and / or media content is provided to the ongoing review session.

[0118] In some examples, the user interface of the review session includes a plurality of visual indicators and / or selectable options, such as those included in control menu 558. Control menu 558 includes selectable option 560 that can be selected to stop playback of the review session.

[0119] In order to indicate the progress of the playback of the review session, the control menu 558 may include a progress bar 562. The progress bar 562 may include a visual playback head that moves along the lateral dimension of the progress bar 562 to illustrate the playback progress relative to the duration of the record. Additionally or alternatively, the progress bar 562 may include a visual indication covering the progress bar 562, thereby indicating the time when those annotations are inserted into the virtual scene. In some examples, the annotation markers 548a to 548d are displayed by covering the model 505, regardless of the time when the annotations are entered into the scene. In some examples, when the review session playback reaches the time when the annotations associated with the annotation markers 548a to 548d are inserted into the virtual scene, the electronic device 101 displays the annotation markers 548a to 548d. Additionally or alternatively, when the review session playback reaches the time corresponding to the annotation, the electronic device 101 may display the annotation, such as annotation 556.

[0120] like Figure 5H The illustrated annotation 556 includes information identifying the author of the annotation 556, a timestamp of the annotation 556, and / or additional or alternative media (e.g., analog markup entered by the author when creating the annotation, a screenshot captured when creating the annotation, and / or a video captured when creating the annotation). Additionally or alternatively, a path 566 of a user of a device (e.g., a user of the electronic device 101 or another electronic device 101 collaborating with the electronic device 101) when creating the annotation may be displayed or animated. In some examples, the path 566 is gradually animated as the review session playback progresses to visually indicate the progress of the author of the annotation 556.

[0121] In some examples, the control menu 558 may include a selectable option 564 that, when selected, initiates display of subtitles that transcribe the voice annotations directed to the virtual scene. For example, before or during playback of a review session, the electronic device 101 may detect an input selecting the selectable option 564. In response to detecting such input, the electronic device 101 may initiate display of subtitles when the review session reaches a time when the voice annotations are directed to the virtual scene.

[0122] A view of a three-dimensional environment is typically visible to a user through a virtual viewport via one or more display generation components (e.g., a display or a pair of display modules that provide stereoscopic content to different eyes of the same user), the virtual viewport having a viewport boundary that defines the range of the three-dimensional environment visible to the user via the one or more display generation components. In some examples, the area defined by the viewport boundary is smaller than the user's visual range in one or more dimensions (e.g., based on the user's visual range, the size, optical properties or other physical characteristics of the one or more display generation components, and / or the position and / or orientation of the one or more display generation components relative to the user's eyes). In some examples, the area defined by the viewport boundary is larger than the user's visual range in one or more dimensions (e.g., based on the user's visual range, the size, optical properties or other physical characteristics of the one or more display generation components, and / or the position and / or orientation of the one or more display generation components relative to the user's eyes). The viewport and viewport boundary typically move with the movement of one or more display generation components (e.g., with the user's head for a head-mounted device, or with the user's hand for a handheld device such as a tablet or smart phone).

[0123] The user's viewpoint determines what is visible in the viewport, and the viewpoint typically specifies a position and orientation relative to the three-dimensional environment, and as the viewpoint moves, the view of the three-dimensional environment will also move in the viewport. For head-mounted devices, the viewpoint is typically based on the position and orientation of the user's head, face, and / or eyes to provide a view of the three-dimensional environment that is perceptually accurate and provides an immersive experience when the user is using the head-mounted device. For handheld or fixed devices, the viewpoint moves as the handheld or fixed device moves and / or as the user's positioning relative to the handheld or fixed device changes (e.g., the user moves toward, away from, up, down, right, and / or left). For devices including display generation components with virtual pass-through, portions of the physical environment that are visible (e.g., displayed and / or projected) via one or more display generation components are based on the field of view of one or more cameras in communication with the display generation components, which one or more cameras typically move with movement of the display generation components (e.g., with movement of the user's head for a head-mounted device, or with movement of the user's hands for a handheld device such as a tablet or smart phone) because the user's viewpoint moves with movement of the field of view of the one or more cameras (and the appearance of one or more virtual objects displayed via the one or more display generation components is updated based on the user's viewpoint (e.g., the display position and pose of the virtual objects are updated based on movement of the user's viewpoint)). For display generation components with optical transmittance, portions of the physical environment that are visible via one or more display generation components (e.g., optically visible through one or more partially or fully transparent portions of the display generation components) are based on the user's field of view through the partially or fully transparent portions of the display generation components (e.g., moves as the user's head moves for a head-mounted device, or moves as the user's hands move for a handheld device such as a tablet or smart phone) because the user's viewpoint moves as the user moves through the field of view of the partially or fully transparent portions of the display generation components (and the appearance of one or more virtual objects is updated based on the user's viewpoint).

[0124] In some examples, a representation of the physical environment (e.g., displayed via virtual pass-through or optical pass-through) may be partially or completely obscured by the virtual environment. In some examples, the amount of the virtual environment displayed (e.g., the amount of the physical environment that is not displayed) is based on the immersion level of the virtual environment (e.g., relative to the representation of the physical environment). For example, increasing the immersion level optionally causes more of the virtual environment to be displayed, replacing and / or obscuring more of the physical environment, and decreasing the immersion level causes less of the virtual environment to be displayed, thereby revealing portions of the physical environment that were previously not displayed and / or obscured. In some examples, at a particular immersion level, one or more first background objects (e.g., in the representation of the physical environment) are visually de-emphasized more than one or more second background objects (e.g., dimmed, blurred, and / or displayed with increased transparency), and one or more third background objects cease to be displayed.

[0125] In some examples, the immersion level includes an associated degree to which virtual content (e.g., a virtual environment and / or virtual content) displayed by a computer system obscures background content (e.g., content other than the virtual environment and / or virtual content) surrounding / behind the virtual content, optionally including the number of items of background content displayed and / or visual characteristics of the displayed background content (e.g., color, contrast, and / or opacity), the angular range of the virtual content displayed via the display generation component (e.g., 60 degrees for content displayed at low immersion, 120 degrees for content displayed at medium immersion, or 180 degrees for content displayed at high immersion), and / or the proportion of the field of view displayed via the display generation component consumed by the virtual content (e.g., 33% of the field of view consumed by the virtual content at low immersion, 66% of the field of view consumed by the virtual content at medium immersion, or 100% of the field of view consumed by the virtual content at high immersion). In some examples, the background content is included in the background in which the virtual content is displayed (e.g., background content in a representation of the physical environment). In some examples, background content includes a user interface (e.g., a user interface corresponding to an application generated by the computer system), virtual objects that are not associated with or included in the virtual environment and / or virtual content (e.g., files or representations of other users generated by the computer system, etc.), and / or real objects (e.g., transparent objects representing real objects in the physical environment surrounding the user, which are visible so that they are displayed via the display generation component and / or visible via transparent or translucent components of the display generation component because the computer system does not block / impede their visibility through the display generation component). In some examples, at a low immersion level (e.g., a first immersion level), the background, virtual and / or real objects are displayed in an unobstructed manner. For example, a virtual environment with a low immersion level is optionally displayed simultaneously with background content, which is optionally displayed at full brightness, color, and / or translucency.

[0126] In some examples, at a higher immersion level (e.g., a second immersion level that is higher than the first immersion level), background, virtual and / or real objects are displayed in an obstructed manner (e.g., dimmed, blurred, or removed from the display). For example, a corresponding virtual environment with a high immersion level is displayed without simultaneously displaying background content (e.g., in a full screen or fully immersive mode). As another example, a virtual environment displayed at a medium immersion level is displayed simultaneously with background content that is dimmed, blurred, or otherwise de-emphasized. In some examples, the visual characteristics of background objects differ between background objects. For example, at a particular immersion level, one or more first background objects are more visually de-emphasized (e.g., dimmed, blurred, and / or displayed with increased transparency) than one or more second background objects, and one or more third background objects cease to be displayed. In some examples, no immersion level or zero immersion level corresponds to ceasing to display the virtual environment, but instead displaying a representation of the physical environment (optionally with one or more virtual objects, such as applications, windows, or virtual three-dimensional objects), and the representation of the physical environment is not obscured by the virtual environment. Adjusting the immersion level using physical input elements provides a fast and efficient method of adjusting immersion, which enhances the operability of the computer system and makes the user-device interface more efficient.

[0127] Therefore, according to the above, some examples of the present disclosure relate to a method performed at a computer system that communicates with one or more input devices and a display. In some examples, the method includes: when the computer system is participating in a communication session including one or more participants, and when a virtual scene is displayed from the user's viewpoint within a three-dimensional environment of a user of the computer system, obtaining first information via the one or more input devices, the first information including a request to display a first annotation at a location within the three-dimensional environment, wherein the correspondence between the first annotation and the corresponding location within the three-dimensional environment is determined based on the context of the user's interaction with the three-dimensional environment. In some examples, in response to obtaining the first information, and based on determining that the first annotation corresponds to a first location in the virtual scene, the method may include: displaying a first representation of the first annotation at the first location. In some examples, in response to obtaining the first information, and based on determining that the first annotation corresponds to a second location in the virtual scene, the method may include: displaying the first representation of the first annotation at the second location.

[0128] Additionally or alternatively, in some examples, the first representation includes a positioning marker that includes a graphical representation indicating one or more portions of the virtual scene.

[0129] Additionally or alternatively, in some examples, the first representation includes an analog mark having one or more visual characteristics based on an analog marking tool selected when obtaining the first information.

[0130] Additionally or alternatively, in some examples, the context of the user's interaction with the three-dimensional environment is based on the target of the user's attention. In some examples, the method may also include: before displaying the first representation of the first annotation, displaying a visual indication of the target of the user's attention via the display.

[0131] Additionally or alternatively, in some examples, the target of the user's attention is based on a location of the user's gaze relative to the three-dimensional environment.

[0132] Additionally or alternatively, in some examples, the target of the user's attention is based on a spatial relationship between one or more parts of the user's body, the user's viewpoint, and the three-dimensional environment.

[0133] Additionally or alternatively, in some examples, the first information is provided by a participant of the communication session different from the user of the computer system, and the first annotation has one or more visual characteristics that visually indicate that the participant provided the first information.

[0134] Additionally or alternatively, in some examples, the one or more visual characteristics include a color of the first representation.

[0135] Additionally or alternatively, in some examples, the method may also include: when displaying the first representation of the first annotation, detecting input directed to the first representation of the first annotation via the one or more input devices; and in response to detecting the input directed to the first representation, presenting contextual information included in the first information.

[0136] Additionally or alternatively, in some examples, the presentation of the contextual information includes presenting audio associated with the first annotation recorded by a participant in the communication session.

[0137] Additionally or alternatively, in some examples, the presentation of the contextual information includes displaying text associated with the first annotation recorded by a participant in the communication session.

[0138] Additionally or alternatively, in some examples, the presentation of the contextual information includes displaying a representation of a participant who provided the first information moving within the three-dimensional environment.

[0139] Additionally or alternatively, in some examples, the method may further include: when the first representation of the first annotation is included in the three-dimensional environment, obtaining second information different from the first information, requesting display of a user interface, the user interface being intended to display multiple representations of multiple annotations including the first annotation. In some examples, the method may further include: in response to obtaining the second information, displaying via the display at least a portion of the multiple representations of the multiple annotations, including a second representation of the first annotation different from the first representation of the first annotation.

[0140] Additionally or alternatively, in some examples, one or more of the plurality of representations are visually grouped according to one or more characteristics of corresponding annotations of the plurality of representations.

[0141] Additionally or alternatively, in some examples, the one or more characteristics include an originator of a corresponding annotation in the plurality of annotations.

[0142] Additionally or alternatively, in some examples, the one or more characteristics include one or more times at which information corresponding to respective ones of the plurality of annotations was obtained.

[0143] Additionally or alternatively, in some examples, the one or more characteristics include a category of information included in a corresponding annotation in the plurality of annotations.

[0144] Additionally or alternatively, in some examples, the first information includes an air gesture performed by one or more parts of the user's body.

[0145] Additionally or alternatively, in some examples, the method may further include: when the computer system is participating in the communication session including the one or more participants, and when the virtual scene is displayed from the viewpoint of the user of the computer system, obtaining second information via the one or more input devices, the second information including a request to include a second annotation within the three-dimensional environment. In some examples, the method may further include: in response to obtaining the second information, presenting feedback indicating a spatial relationship between the location of the second annotation within the three-dimensional environment and the viewpoint of the user.

[0146] Additionally or alternatively, in some examples, the feedback includes audio generated using a three-dimensional ambient effect that simulates sensing of a physical audio source that generates audio based on the positioning of the second annotation.

[0147] Additionally or alternatively, in some examples, the method may also include: when the computer system is participating in the communication session including the one or more participants, displaying a representation of a first participant of the one or more participants of the communication session at a second location within the three-dimensional environment via the display.

[0148] Additionally or alternatively, in some examples, the representation of the first participant is an expressive avatar that includes one or more simulated body parts that correspond to physical body parts of the first participant.

[0149] Additionally or alternatively, in some examples, the method may further include: when displaying the representation of the first participant at the second location within the three-dimensional environment, obtaining second information, the second information including a request to move the representation of the first participant within the three-dimensional environment. In some examples, the method may further include: in response to obtaining the second information, moving the representation of the first participant from the second location to a third location according to the request to move the representation of the first participant.

[0150] Additionally or alternatively, in some examples, the method may further include: when displaying the representation of the first participant at the second location within the three-dimensional environment, obtaining second information, the second information including a request to display a target of the first participant's attention. In some examples, the method may further include: in response to obtaining the second information, displaying a visual indication of the attention at a third location within the three-dimensional environment via the display.

[0151] Additionally or alternatively, in some examples, the visual appearance of the first representation of the first annotation corresponds to content included in the first annotation.

[0152] Additionally or alternatively, in some examples, the correspondence between the first annotation and the corresponding location within the three-dimensional environment is based on a spatial relationship between one or more parts of the user's body relative to the three-dimensional environment.

[0153] Additionally or alternatively, in some examples, the context of the user's interaction with the three-dimensional environment is associated with one or more words spoken by the user of the computer system included in the first information.

[0154] Additionally or alternatively, in some examples, the first location is a predetermined distance from the viewpoint of the user.

[0155] Additionally or alternatively, in some examples, the method may further include: when displaying the virtual scene from the viewpoint of the user, obtaining second information, the second information including a request to capture a two-dimensional image of the virtual scene, wherein when the second information is obtained, a portion of the three-dimensional environment is visible via the viewport of the computer system. In some examples, the method may further include: in response to obtaining the second information, displaying via the display a virtual object including a two-dimensional image of the three-dimensional environment, the two-dimensional image including the portion of the three-dimensional environment.

[0156] Additionally or alternatively, in some examples, the displaying of the virtual object includes displaying an animation, the animation including displaying the virtual object at a first size corresponding to the viewport of the computer system in response to the second information. In some examples, the animation includes, after displaying the virtual object, shrinking the virtual object to a second size, and maintaining visibility of the portion of the three-dimensional environment via the viewport while shrinking the virtual object.

[0157] Additionally or alternatively, in some examples, the simulated thickness of the virtual object is less than a simulated thickness threshold.

[0158] Additionally or alternatively, in some examples, the method may further include: when displaying the virtual object at a visual significance level that is a first visual significance level relative to the three-dimensional environment, and when the viewpoint of the user is the first viewpoint, detecting, via the one or more input devices, a change of the viewpoint of the user to a second viewpoint. In some examples, the method may further include: in response to detecting the change of the viewpoint, reducing the visual significance level to a second visual significance level that is less than the first visual significance level.

[0159] Additionally or alternatively, in some examples, the method may further include: when displaying the virtual object, detecting, via the one or more input devices, a request to display a simulated marker within the virtual object, the simulated marker comprising movement of one or more parts of the user's body. In some examples, the method may further include: in response to detecting the request to display the simulated marker, displaying the simulated marker according to the movement.

[0160] Additionally or alternatively, in some examples, the method may further include: when displaying the virtual object, detecting, via the one or more input devices, a request to select virtual content included in the virtual object. In some examples, the method may further include: in response to the request to select virtual content, displaying within the virtual object a visual indication of a target of the request.

[0161] Some examples of the present disclosure relate to an electronic device that includes one or more processors, a memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for executing a method as described herein.

[0162] Some examples of the present disclosure relate to a non-transitory computer-readable storage medium that stores one or more programs that include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform a method as described herein.

[0163] Some examples of the present disclosure relate to an electronic device comprising one or more processors, a memory, and means for performing a method as described herein.

[0164] Some examples of the present disclosure relate to an information processing apparatus for use in an electronic device, the information processing apparatus comprising means for performing a method as described herein.

[0165] Some examples of the present disclosure relate to an electronic device that communicates with one or more input devices and a display, the electronic device comprising: one or more processors; memory; and one or more programs, the one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing a method. In some examples, the method may include: when the computer system is participating in a communication session including one or more participants, and when a virtual scene is displayed from the user's viewpoint within a three-dimensional environment of a user of the computer system, obtaining first information via the one or more input devices, the first information comprising a request to display a first annotation at a location within the three-dimensional environment, wherein the correspondence between the first annotation and the corresponding location within the three-dimensional environment is determined based on the context of the user's interaction with the three-dimensional environment. In some examples, the method may include: in response to obtaining the first information, displaying a first representation of the first annotation at the first location according to determining that the first annotation corresponds to a first location in the virtual scene; and displaying the first representation of the first annotation at the second location according to determining that the first annotation corresponds to a second location in the virtual scene.

[0166] Some examples of the present disclosure relate to a non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device in one or more input devices and a display, cause the electronic device to perform a method. In some examples, the method may include: when the computer system is participating in a communication session including one or more participants, and when a virtual scene is displayed from the user's viewpoint within a three-dimensional environment of a user of the computer system, obtaining first information via the one or more input devices, the first information including a request to display a first annotation at a location within the three-dimensional environment, wherein the correspondence between the first annotation and the corresponding location within the three-dimensional environment is determined based on the context of the user's interaction with the three-dimensional environment. In some examples, the method may include: in response to obtaining the first information, displaying a first representation of the first annotation at the first location based on determining that the first annotation corresponds to a first location in the virtual scene; and displaying the first representation of the first annotation at the second location based on determining that the first annotation corresponds to a second location in the virtual scene.

[0167] For the purpose of explanation, the foregoing description is described by reference to specific examples. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible based on the above teachings. The examples are selected and described in order to best illustrate the principles of the invention and its practical application, so as to enable others skilled in the art to best use the invention and the various described examples with various modifications suitable for the specific purpose contemplated.

Claims

1. A method comprising: At a computer system in communication with one or more input devices and a display: When the computer system is participating in a communication session including one or more participants, and when a virtual scene is displayed from a viewpoint of a user of the computer system within a three-dimensional environment of the user: obtaining, via the one or more input devices, first information, the first information comprising a request to display a first annotation at a location within the three-dimensional environment, wherein the correspondence of the first annotation to the corresponding location within the three-dimensional environment is determined based on a context of the user's interaction with the three-dimensional environment; as well as In response to obtaining the first information: Based on determining that the first annotation corresponds to a first location in the virtual scene, displaying a first representation of the first annotation at the first location; as well as Based on determining that the first annotation corresponds to a second location in the virtual scene, the first representation of the first annotation is displayed at the second location. 2 . The method of claim 1 , wherein the first representation comprises a positioning marker comprising a graphical representation indicating one or more portions of the virtual scene. 3 . The method of claim 1 , wherein the first representation comprises an analog mark having one or more visual characteristics based on an analog marking tool selected when obtaining the first information.

4. The method of claim 1 , wherein the context of the user's interaction with the three-dimensional environment is based on a target of the user's attention, the method further comprising: Prior to displaying the first representation of the first annotation, a visual indication of the target of the user's attention is displayed via the display. The method of claim 4 , wherein the target of the user's attention is based on a location of the user's gaze relative to the three-dimensional environment.

6. The method of claim 4, wherein the target of the user's attention is based on a spatial relationship between one or more parts of the user's body, the user's viewpoint, and the three-dimensional environment.

7. The method of claim 1, wherein the first information is provided by a participant of the communication session different from the user of the computer system, and the first annotation has one or more visual characteristics that visually indicate that the participant provided the first information.

8. The method according to claim 1, further comprising: while displaying the first representation of the first annotation, detecting, via the one or more input devices, input directed to the first representation of the first annotation; as well as In response to detecting the input directed to the first representation, contextual information included in the first information is presented.

9. The method of claim 8, wherein the presenting of the contextual information comprises presenting audio associated with the first annotation recorded by a participant of the communication session.

10. The method according to claim 1, further comprising: obtaining, when the first representation of the first annotation is included within the three-dimensional environment, second information different from the first information, the second information requesting display of a user interface directed to displaying a plurality of representations of a plurality of annotations including the first annotation; as well as In response to obtaining the second information, at least a portion of the plurality of representations of the plurality of annotations is displayed via the display, including a second representation of the first annotation that is different from the first representation of the first annotation.

11. The method of claim 1, wherein the first information comprises an air gesture performed by one or more parts of the user's body.

12. The method according to claim 1, further comprising: while the computer system is participating in the communication session including the one or more participants, and while the virtual scene is displayed from the viewpoint of the user of the computer system, obtaining, via the one or more input devices, second information, the second information including a request to include a second annotation within the three-dimensional environment; as well as In response to obtaining the second information, feedback is presented indicating a spatial relationship between a location of the second annotation within the three-dimensional environment and the viewpoint of the user.

13. The method according to claim 1, further comprising: While the computer system is participating in the communication session including the one or more participants, a representation of a first participant of the one or more participants of the communication session is displayed via the display at a second location within the three-dimensional environment.

14. The method of claim 1, wherein a visual appearance of the first representation of the first annotation corresponds to content included in the first annotation.

15. The method of claim 1, wherein the correspondence between the first annotation and the corresponding location within the three-dimensional environment is based on a spatial relationship between one or more parts of the user's body relative to the three-dimensional environment.

16. The method of claim 1, wherein the context of the user's interaction with the three-dimensional environment is associated with one or more words spoken by the user of the computer system included in the first information.

17. The method of claim 1, wherein the first location is a predetermined distance from the viewpoint of the user.

18. The method of claim 1, further comprising: obtaining, while displaying the virtual scene from the viewpoint of the user, second information including a request to capture a two-dimensional image of the virtual scene, wherein when the second information is obtained, a portion of the three-dimensional environment is visible via a viewport of the computer system; as well as In response to obtaining the second information, a virtual object including the two-dimensional image of the three-dimensional environment is displayed via the display, the two-dimensional image including the portion of the three-dimensional environment.

19. An electronic device, the electronic device communicating with one or more input devices and a display, the electronic device comprising: one or more processors; Memory; and One or more programs, the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 18.

20. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by one or more processors of an electronic device that communicates with one or more input devices and a display, cause the electronic device to perform a method according to any one of claims 1 to 18.

Images