Presenting enhanced video see-through in a three-dimensional environment

Abstract

Methods and apparatus for presenting enhanced video see-through in a three-dimensional environment. In some examples, a first electronic device is in communication with one or more input devices. In some examples, the electronic device identifies a region within a three-dimensional environment, captures one or more first images associated with the identified region within the three-dimensional environment via the one or more input devices, identifies respective portions of the one or more first images that correspond to the identified region, and generates one or more second images based on the identified respective portions of the one or more first images, wherein the one or more second images have enhanced visual characteristics relative to visual characteristics of the one or more first images.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 885,927, filed September 22, 2025; U.S. Provisional Application No. 63 / 755,993, filed February 7, 2025; and U.S. Patent Application No. 19 / 447,930, filed January 13, 2026, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0002] The present invention relates to systems and methods for rendering enhanced video perspective in a three-dimensional environment. Background Technology

[0003] Some computer graphics environments provide two-dimensional (2D) and / or three-dimensional (3D) environments, where at least some of the objects displayed for the user to view are virtual and computer-generated. In some examples, these computer graphics environments provide enhanced video perspective. Summary of the Invention

[0004] Some examples of this disclosure relate to systems and methods for rendering enhanced video perspective in a three-dimensional environment. In some examples, electronic devices communicate with one or more input devices. In some examples, the electronic devices identify a region within a three-dimensional environment, capture one or more first images associated with the identified region within the three-dimensional environment via the one or more input devices, identify corresponding portions of the one or more first images corresponding to the identified region, and generate one or more second images based on the identified corresponding portions of the one or more first images, wherein the one or more second images have enhanced visual characteristics relative to the one or more first images. In some examples, the first electronic device communicates with one or more input devices, one or more displays, and a second electronic device. In some examples, when the first electronic device renders user interface elements including one or more first images in a three-dimensional environment via the one or more displays, the first electronic device identifies a region within the one or more first images and renders one or more second images based on the identified region via the one or more displays. In some examples, the one or more second images have enhanced visual characteristics relative to the corresponding visual characteristics of the one or more first images.

[0005] A full description of these examples is provided in the accompanying drawings and detailed embodiments, and it should be understood that the content of this invention does not limit the scope of this disclosure in any way. Attached Figure Description

[0006] To better understand the various examples described herein, reference should be made to the following detailed embodiments and accompanying drawings. Throughout the drawings, similar reference numerals generally refer to corresponding parts.

[0007] Figure 1 Examples of electronic devices that present a three-dimensional environment according to some examples of this disclosure are illustrated.

[0008] Figures 2A to 2B A block diagram illustrating an example architecture for an electronic device according to some examples of this disclosure is shown.

[0009] Figures 3A to 3G , Figures 4A to 4C and Figures 5A to 5C Examples of rendering enhanced video perspective in a three-dimensional environment are illustrated according to some examples of this disclosure.

[0010] Figure 6 Example processes for generating enhanced video perspective in a three-dimensional environment are illustrated according to some examples of this disclosure.

[0011] Figure 7 The flowchart illustrates an example process for rendering enhanced video perspective in a three-dimensional environment, according to some examples of this disclosure.

[0012] Figures 8A to 8J Examples of rendering enhanced video in a three-dimensional environment are illustrated according to some examples of this disclosure.

[0013] Figure 9 The flowchart illustrates an example process for rendering enhanced video in a three-dimensional environment according to some examples of this disclosure. Detailed Implementation

[0014] Some examples of this disclosure relate to methods and apparatus for generating and rendering enhanced video perspective in a three-dimensional environment. In some examples, electronic devices communicate with one or more input devices. In some examples, the electronic devices identify regions within the three-dimensional environment. In some examples, the electronic devices capture one or more first images associated with the identified regions within the three-dimensional environment via one or more input devices. In some examples, the electronic devices identify corresponding portions of the one or more first images corresponding to the identified regions. In some examples, the electronic devices generate one or more second images based on the identified corresponding portions of the one or more first images. In some examples, the one or more second images have enhanced visual characteristics relative to the visual characteristics of the one or more first images. In some examples, a first electronic device communicates with one or more input devices, one or more displays, and a second electronic device. In some examples, when the first electronic device renders user interface elements of one or more first images in a three-dimensional environment including the second electronic device via the one or more displays, the first electronic device identifies regions within the one or more first images and renders one or more second images based on the identified regions via the one or more displays. In some examples, the one or more second images have enhanced visual characteristics relative to the corresponding visual characteristics of the one or more first images.

[0015] In some examples, a 3D object is displayed in a computer-generated 3D environment with a specific orientation that controls one or more behaviors of the 3D object (e.g., when the 3D object moves within the 3D environment). In some examples, the orientation of the 3D object displayed in the 3D environment is selected by the user of the electronic device or automatically by the electronic device. For example, when initiating the rendering of a 3D object in a 3D environment, the user may select a specific orientation of the 3D object, or the electronic device may automatically select the orientation of the 3D object (e.g., based on the type of the 3D object).

[0016] In some examples, 3D objects may be displayed in a 3D environment with world-locked orientation, body-locked orientation, tilt-locked orientation, or head-locked orientation, as described below. As used herein, an object displayed in a 3D environment with body-locked orientation has a distance and orientation offset relative to a portion of the user's body (e.g., the user's torso). Alternatively, in some examples, body-locked objects have a fixed distance from the user, and the orientation of the content does not reference any part of the user's body (e.g., it may be displayed in the same basic direction relative to the user, regardless of head and / or body movement). Additionally or alternatively, in some examples, body-locked objects may be configured to always maintain gravity or horizon (e.g., perpendicular to gravity) alignment, such that changes in the head and / or body's scrolling direction will not cause the body-locked object to move within the 3D environment. Conversely, translational movement in either configuration will cause the body-locked object to reposition within the 3D environment to maintain the distance offset.

[0017] As used in this article, objects displayed in a head-locked orientation within a 3D environment have a distance and orientation offset relative to the user's head. In some examples, the head-locked object moves within the 3D environment as the user's head moves (when the user's viewpoint changes).

[0018] As used in this article, objects displayed in a world-locked orientation in a 3D environment do not have a distance or orientation offset relative to the user.

[0019] As used herein, an object displayed in a tilt-locked orientation in a 3D environment (referred to herein as a tilt-locked object) has a distance offset relative to a user (such as a part of the user's body, e.g., the user's torso, or the user's head). In some examples, the tilt-locked object is displayed in a fixed orientation relative to the 3D environment. In some examples, the tilt-locked object moves according to a polar (e.g., spherical) coordinate system centered on a pole passing through the user (e.g., the user's head). For example, the tilt-locked object moves in the 3D environment based on the movement of the user's head within a spherical space surrounding the user's head (e.g., centered on the user's head). Thus, if the user tilts their head relative to gravity (e.g., up or down in the pitch direction), the tilt-locked object will follow the head tilt and move radially along the sphere, such that the tilt-locked object is repositioned in the 3D environment with the same distance offset relative to the user as before the head tilt, while optionally maintaining the same orientation relative to the 3D environment. In some examples, if the user moves their head relative to gravity in a rolling direction (e.g., clockwise or counterclockwise), the tilt-locked object is not repositioned in the 3D environment.

[0020] Figure 1An electronic device 101 according to some examples of this disclosure is illustrated, which presents a three-dimensional environment (e.g., an extended reality (XR) environment or a computer-generated reality (CGR) environment, optionally including representations of physical and / or virtual objects). In some examples, such as Figure 1 As shown, electronic device 101 is a head-mounted display or other head-mountable device configured to be worn on the head of a user of electronic device 101. See below for reference. Figure 2A An example of an architecture block diagram to describe electronic device 101. For example... Figure 1 As shown, electronic device 101 and table 106 are located in a physical environment. The physical environment may include physical features, such as physical surfaces (e.g., floor, wall) or physical objects (e.g., table, lamp, etc.). In some examples, electronic device 101 may be configured to detect and / or capture images of the physical environment including table 106 (exemplified in the field of view of electronic device 101).

[0021] In some examples, such as Figure 1 As shown, the electronic device 101 includes one or more internal image sensors 114a disposed toward the user's face (e.g., as referenced below). Figures 2A to 2B (The described eye-tracking camera). In some examples, an internal image sensor 114a is used for eye tracking (e.g., detecting the user's gaze). The internal image sensor 114a is optionally arranged on the left and right portions of the display 120 to enable eye tracking of the user's left and right eyes. In some examples, the electronic device 101 also includes external image sensors 114b and 114c facing outwards from the user to detect and / or capture the physical environment of the electronic device 101 and / or movement of the user's hands or other body parts.

[0022] In some examples, display 120 has a field of view visible to the user. In some examples, the user-visible field of view is the same as that of external image sensors 114b and 114c. For example, when display 120 is optionally part of a head-mounted device, the field of view of display 120 is optionally the same as or similar to that of the user's eyes. In some examples, the user-visible field of view differs from that of external image sensors 114b and 114c (e.g., is narrower than that of external image sensors 114b and 114c). In other examples, the field of view of display 120 may be smaller than that of the user's eyes. The user's viewpoint determines what is visible in the field of view, and the viewpoint typically specifies the position and orientation relative to the three-dimensional environment. As the user's viewpoint shifts, the field of view of the three-dimensional environment also shifts accordingly. In some examples, electronic device 101 may be an optically transparent device, through which display 120 is a transparent or translucent display through which parts of the physical environment can be directly viewed. In some examples, display 120 may be included within a transparent lens and may overlap entirely or partially with the transparent lens. In other examples, the electronics may be a video see-through device, wherein display 120 is an opaque display configured to display an image of the physical environment using images captured by external image sensors 114b and 114c. Although Figure 1 A single display is shown, but it should be understood that display 120 optionally includes more than one display. For example, display 120 optionally includes a pair of stereoscopic displays (e.g., a left display panel and a right display panel for the user's left and right eyes, respectively), the pair of stereoscopic displays having display outputs that are combined (e.g., by the user's brain) to create Figure 1 The view showing the content. In some examples, as referenced below. Figures 2A to 2B In more detail, the display 120 includes or corresponds to a transparent or translucent surface (e.g., a lens) that is not equipped with display capabilities (e.g., and therefore cannot generate and display virtual objects 104), and alternatively presents a direct view of the physical environment in the user's field of view (e.g., the user's eye's field of view).

[0023] In some examples, electronic device 101 is configured to display (e.g., in response to a trigger) virtual object 104 in a three-dimensional environment. Virtual object 104 is... Figure 1 The cube illustrated represents a cube that does not exist in the physical environment but is displayed in a three-dimensional environment and positioned on top of table 106 (e.g., a real-world table or a representation thereof). Optionally, in response to detecting a flat surface of table 106 in the physical environment 100, virtual object 104 is displayed on the surface of table 106 in the three-dimensional environment displayed via display 120 of electronic device 101.

[0024] It should be understood that virtual object 104 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or other three-dimensional virtual objects) in a three-dimensional environment. For example, a virtual object may represent an application or user interface displayed in a three-dimensional environment. In some examples, a virtual object may represent content corresponding to an application and / or content displayed in a three-dimensional environment via a user interface. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g., air gestures, such as air pinch gestures, air tap gestures, and / or air touch gestures), allowing the user to virtually touch, tap, move, rotate, or otherwise interact with virtual object 104.

[0025] As discussed in this article, one or more air pinch gestures performed by the user (e.g., using...) Figure 1 The hand (103) is detected by one or more input devices of electronic device 101 and interpreted as one or more user inputs for the content displayed by electronic device 101. Additionally or alternatively, in some examples, the one or more user inputs interpreted by electronic device 101 for the content displayed by electronic device 101 (e.g., virtual object 104) are detected via one or more hardware input devices (e.g., controllers, touchpads, proximity sensors, buttons, sliders, knobs, etc.) rather than via one or more input devices configured to detect air gestures (such as one or more air pinch gestures) performed by the user. Such descriptions are intended to be exemplary and not limiting; the user may optionally use different air gestures and / or other forms of input to provide user input.

[0026] In some examples, electronic device 101 may be configured to communicate with a second electronic device (such as a companion device). For example, as Figure 1 As illustrated, electronic device 101 optionally communicates with electronic device 160. In some examples, electronic device 160 corresponds to a mobile electronic device, such as a smartphone, tablet computer, smartwatch, laptop computer, or other electronic device. In some examples, electronic device 160 corresponds to a non-mobile electronic device, which is typically stationary and not easily moved in its physical environment (e.g., a desktop computer, server, etc.). See below for reference. Figure 2B The following are additional examples of architectural diagrams used to describe electronic device 160. In some examples, electronic device 101 and electronic device 160 are associated with the same user. For example, in Figure 1In this configuration, electronic device 101 may be positioned on a user's head (e.g., mounted on the head), and electronic device 160 may be positioned near electronic device 101, such as in the user's hand 103 (e.g., hand 103 holding electronic device 160), in the user's pocket or bag, or on a surface near the user. Electronic devices 101 and 160 may optionally be associated with the same user account (e.g., the user is logged into the user account on both electronic devices 101 and 160). See below for further details. Figures 2A to 2B Additional details regarding the communication between electronic device 101 and electronic device 160 are provided.

[0027] In some examples, the display of an object in a 3D environment is triggered by interaction with one or more user interface objects in the 3D environment, or the display of an object in a 3D environment enables interaction with one or more user interface objects in the 3D environment. For example, initiating the display of an object in a 3D environment may include interaction with one or more virtual option / capability representations displayed in the 3D environment. In some examples, the electronic device may track the user's gaze as an input signal to identify one or more virtual option / capability representations available for selection when the object display in the 3D environment is initiated. For example, a gaze may be used to identify one or more virtual option / capability representations as targets for selection using another selection input. In some examples, a virtual option / capability representation may be selected using hand-tracking input detected via an input device communicating with the electronic device. In some examples, an object displayed in a 3D environment may be moved and / or reoriented in the 3D environment based on movement input detected via an input device.

[0028] In the following description, an electronic device that communicates with one or more displays and one or more input devices is described. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as touch-sensitive surfaces, physical keyboards, mice, joysticks, hand-tracking devices, eye-tracking devices, styluses, etc. Furthermore, as mentioned above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed among two or more devices. Therefore, as used in this disclosure, information on or displayed by the electronic device is optionally used to describe information output by the electronic device for display on a separate display device (touch-sensitive or non-touch-sensitive). Similarly, as used in this disclosure, input received on the electronic device (e.g., touch input received on a touch-sensitive surface of the electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on a separate input device from which the electronic device receives input information.

[0029] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, phone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, TV channel browsing applications, and / or digital video player applications.

[0030] Figures 2A to 2B Block diagrams illustrating example architectures for electronic devices according to some examples of this disclosure are shown. In some examples, electronic device 201 and / or device 260 include one or more electronic devices. For example, electronic device 201 may be a portable device, an auxiliary device for communicating with another device, a head-mounted display, a head-mounted speaker, etc. In some examples, electronic device 201 corresponds to the above reference. Figure 1 The described electronic device 101. In some examples, electronic device 260 corresponds to the above reference. Figure 1 The described electronic device 160.

[0031] like Figure 2A As illustrated, electronic device 201 optionally includes one or more sensors, such as one or more hand tracking sensors 202, one or more position sensors 204A, and one or more image sensors 206A (optionally corresponding to...). Figure 1 The electronic device 201 may include an internal image sensor 114a and / or external image sensors 114b and 114c, one or more touch-sensitive surfaces 209A, one or more motion and / or orientation sensors 210A, one or more eye-tracking sensors 212, one or more microphones 213A or other audio sensors, one or more body-tracking sensors (e.g., torso and / or head-tracking sensors), etc. The electronic device 201 may optionally include one or more output devices, such as one or more display generating components 214A (optionally corresponding to...). Figure 1 The electronic device 201 may include a display 120, one or more speakers 216A, one or more tactile output devices (not shown), etc. The electronic device 201 optionally includes one or more processors 218A, one or more memories 220A, and / or communication circuitry 222A. One or more communication buses 208A are optionally used for communication between the components of the electronic device 201 mentioned above.

[0032] Additionally, electronic device 260 optionally includes components that are the same as or similar to those of electronic device 201. For example, such as Figure 2BAs shown, electronic device 260 optionally includes one or more position sensors 204B, one or more image sensors 206B, one or more touch-sensitive surfaces 209B, one or more orientation sensors 210B, one or more microphones 213B, one or more display generating components 214B, one or more speakers 216B, one or more processors 218B, one or more memories 220B, and / or communication circuitry 222B. One or more communication buses 208B are optionally used for communication between the aforementioned components of electronic device 260.

[0033] Electronic devices 201 and 260 are optionally configured to communicate via a wired or wireless connection between the two electronic devices (e.g., via communication circuits 222A, 222B). For example, as Figure 2A As indicated, electronic device 260 can be used as an accessory to electronic device 201. For example, in some examples, electronic device 260 processes sensor inputs from electronic devices 201 and 260, and / or uses the display generation component 214A of electronic device 201 to generate content for display.

[0034] Communication circuits 222A and 222B optionally include circuitry for communicating with electronic devices and networks such as the Internet, intranets, wired and / or wireless networks, cellular networks, and wireless local area networks (LANs). Communication circuits 222A and 222B optionally include circuitry for using near-field communication (NFC) and / or devices such as Bluetooth. ® The communication circuits 222A and 222B are for short-range communication. In some examples, the communication circuits 222A and 222B include or support Wi-Fi (e.g., the 802.11 protocol), Ethernet, Ultra Wideband (“UWB”), high-frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), or any other communication protocol, or any combination thereof.

[0035] One or more processors 218A, 218B include one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, one or more processors 218A, 218B include one or more microprocessors, one or more central processing units, one or more application-specific integrated circuits, one or more field-programmable gate arrays, one or more programmable logic devices, or combinations of such devices. In some examples, memories 220A and / or 220B are non-transitory computer-readable storage media (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions configured to be executed by one or more processors 218A, 218B to perform the techniques, processes, and / or methods described herein. In some examples, memories 220A and / or 220B may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium can be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, and device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on compact disc (CD), digital versatile optical disc (DVD), or Blu-ray technology, and persistent solid-state storage devices such as flash memory, solid-state drives, etc.

[0036] In some examples, one or more display generating components 214A, 214B include a single display (e.g., a liquid crystal display (LCD), an organic light-emitting diode (OLED), or other type of display). In some examples, one or more display generating components 214A, 214B include multiple displays. In some examples, one or more display generating components 214A, 214B may include a touch-enabled display (e.g., a touchscreen), a projector, a holographic projector, a retinal projector, a transparent or translucent display, etc. In some examples, the electronic device does not include one or more display generating components 214A or 214B. For example, some electronic devices include transparent or translucent lenses or other surfaces not configured to display or present virtual content in place of one or more display generating components 214A or 214B. However, it should be understood that in such instances, electronic device 201 and / or electronic device 260 are optionally equipped with Figure 2A and Figure 2BOne or more of the other components illustrated herein, such as one or more hand-tracking sensors 202, one or more eye-tracking sensors 212, one or more image sensors 206A, and / or one or more motion and / or orientation sensors 210A. Alternatively, in some examples, one or more display generation components 214A or 214B are provided separately from electronic devices 201 and / or 260. For example, one or more display generation components 214A, 214B communicate with electronic device 201 (and / or electronic device 260) but are not integrated with electronic device 201 and / or electronic device 260 (e.g., not integrated within the housing of electronic devices 201, 260). In some examples, electronic devices 201 and 260 include one or more touch-sensitive surfaces 209A and 209B, respectively, for receiving user input, such as tap input and swipe input or other gestures (e.g., hand-based or finger-based gestures). In some examples, one or more display generating components 214A, 214B and one or more touch-sensitive surfaces 209A, 209B form one or more touch-sensitive displays (e.g., touchscreens integrated with each of the electronic devices 201 and 260, or touchscreens external to each of the electronic devices 201 and 260 and communicating with each of the electronic devices 201 and 260).

[0037] Electronic devices 201 and 260 optionally include one or more image sensors 206A and 206B, respectively. The image sensors 206A and 206B optionally include one or more visible light image sensors (such as charge-coupled device (CCD) sensors) and / or complementary metal-oxide-semiconductor (CMOS) sensors operable to acquire images of physical objects from a real-world environment. The image sensors 206A and 206B also optionally include one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensors 206A and 206B also optionally include one or more cameras configured to capture movement of physical objects in the real-world environment. The image sensors 206A and 206B also optionally include one or more depth sensors configured to detect the distance between the physical object and the electronic devices 201 and 260. In some examples, information from one or more depth sensors allows a device to identify objects in a real-world environment and distinguish them from other objects in the real-world environment. In some examples, one or more depth sensors allow a device to determine the texture and / or shape of objects in a real-world environment. In some examples, one or more image sensors 206A or 206B are included in an electronic device different from electronic devices 201 and / or 260. For example, one or more image sensors 206A, 206B communicate with electronic devices 201, 260 but are not integrated with electronic devices 201, 260 (e.g., not integrated within the housing of electronic devices 201, 260). Specifically, in some examples, one or more cameras of one or more image sensors 206A, 206B are integrated with and / or coupled to one or more devices separate from electronic devices 201 and / or 260 (e.g., but communicating with electronic devices 201 and / or 260), such as one or more input and / or output devices including one or more image sensors 206A, 206B (e.g., one or more speakers and / or one or more microphones, such as headphones or headsets). In some examples, electronic device 201 or electronic device 260 corresponds to a headphone speaker (e.g., headphones or earbuds). In such instances, electronic device 201 or electronic device 260 is equipped with Figure 2A and Figure 2B A subset of other components illustrated herein. In some such examples, electronic device 201 or electronic device 260 is equipped with one or more image sensors 206A, 206B, one or more motion and / or orientation sensors 210A, 210B, and / or speakers 216A, 216B.

[0038] In some examples, electronic devices 201 and 260 combine a CCD sensor, an event camera, and a depth sensor to detect the physical environment surrounding them. In some examples, one or more image sensors 206A and 206B include a first image sensor and a second image sensor. The first and second image sensors work cooperatively and are optionally configured to capture different information about physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor, and the second image sensor is a depth sensor. In some examples, electronic devices 201 and 260 use one or more image sensors 206A and 206B to detect the location and orientation of electronic devices 201 and 260 and / or one or more display generating components 214A and 214B in the real-world environment. For example, electronic devices 201 and 260 use one or more image sensors 206A and 206B to track the location and orientation of one or more display generating components 214A and 214B relative to one or more stationary objects in the real-world environment.

[0039] In some examples, electronic devices 201 and 260 include one or more microphones 213A and 213B, or other audio sensors, respectively. Electronic devices 201 and 260 optionally use one or more microphones 213A and 213B to detect sound from a user and / or the user's real-world environment. In some examples, the one or more microphones 213A and 213B include microphone arrays (e.g., multiple microphones) that optionally work together, such as to identify ambient noise or locate sound sources in a real-world environmental space.

[0040] Electronic devices 201 and 260 each include one or more position sensors 204A and 204B for detecting the positions of electronic device 201 and / or one or more display generating components 214A, and the positions of electronic device 260 and / or one or more display generating components 214B, respectively. For example, the one or more position sensors 204A, 204B may include a Global Positioning System (GPS) receiver that receives data from one or more satellites, enabling electronic devices 201, 260 to determine their absolute position in the physical world.

[0041] Electronic devices 201 and 260 each include one or more orientation sensors 210A and 210B for detecting the orientation and / or movement of electronic device 201 and / or one or more display generating components 214A, and the orientation and / or movement of electronic device 260 and / or one or more display generating components 214B, respectively. For example, electronic devices 201 and 260 use one or more orientation sensors 210A and 210B to track changes in the positioning and / or orientation of electronic devices 201 and 260 and / or one or more display generating components 214A and 214B, such as changes relative to physical objects in a real-world environment. The one or more orientation sensors 210A and 210B optionally include one or more gyroscopes and / or one or more accelerometers.

[0042] In some examples, electronic device 201 includes one or more hand-tracking sensors 202 and / or one or more eye-tracking sensors 212. It should be understood that although electronic device 201 is referred to as a hand-tracking or eye-tracking sensor, the electronic device may additionally or optionally include one or more other body-tracking sensors, such as one or more leg-tracking sensors, one or more torso-tracking sensors, and / or one or more head-tracking sensors. The one or more hand-tracking sensors 202 are configured to track the localization and / or position of one or more portions of a user's hand, and the movement of one or more portions of the user's hand relative to a three-dimensional environment, relative to one or more display generation components 214A, and / or relative to another defined coordinate system. The one or more eye-tracking sensors 212 are configured to track the localization and movement of a user's gaze (e.g., the user's attention, more generally including the eyes, face, or head) relative to the real world or three-dimensional environment and / or relative to one or more display generation components 214A. In some examples, one or more hand-tracking sensors 202 and / or one or more eye-tracking sensors 212 are implemented together with one or more display generation components 214A. In some examples, one or more hand-tracking sensors 202 and / or one or more eye-tracking sensors 212 are implemented separately from one or more display generation components 214A. In some examples, the electronic device 201 optionally does not include one or more hand-tracking sensors 202 and / or one or more eye-tracking sensors 212. In some such examples, one or more display generation components 214A may be utilized by the electronic device 260 to provide a three-dimensional environment, and the electronic device 260 may utilize input and other data collected via one or more other sensors of the electronic device 201 (e.g., one or more position sensors 204A, one or more image sensors 206A, one or more touch-sensitive surfaces 209A, one or more motion and / or orientation sensors 210A, and / or one or more microphones 213A or other audio sensors) as input and data processed by one or more processors 218B of the electronic device 260. Additionally or alternatively, the electronic device 260 optionally does not include Figure 2B Other components shown include one or more position sensors 204B, one or more image sensors 206B, one or more touch-sensitive surfaces 209B, etc. In some such examples, one or more display generation components 214A may be utilized by electronic device 260 to provide a three-dimensional environment, and electronic device 260 may utilize input and other data collected via one or more motion and / or orientation sensors 210A (and / or one or more microphones 213A) of electronic device 201 as input.

[0043] In some examples, one or more hand-tracking sensors 202 (and / or other body-tracking sensors, such as leg, torso, and / or head-tracking sensors) may use one or more image sensors 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that capture real-world 3D information from one or more body parts, including one or more body parts (e.g., a human user's hand, leg, or torso). In some examples, sufficient resolution is available to distinguish the hand to differentiate the fingers and their corresponding positions. In some examples, one or more image sensors 206A are positioned relative to the user to define the field of view and interaction space of one or more image sensors 206A, wherein the finger / hand positions, orientations, and / or movements captured by the image sensors are used as input (e.g., to differentiate from the user's stationary hand or other hands of other people in the real-world environment). Tracking the fingers / hands used for input (e.g., gestures, touches, taps, etc.) can be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0044] In some examples, one or more eye-tracking sensors 212 include at least one eye-tracking camera (e.g., an IR camera) and / or an illumination source (e.g., an IR light source, such as an LED) that emits light toward the user's eyes. The eye-tracking camera may be pointed at the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by the respective eye-tracking camera and illumination source, and focus / gaze can be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by one or more respective eye-tracking cameras / illumination sources.

[0045] Electronic devices 201 and 260 are not limited to Figures 2A to 2B The components and configurations may include, but are not limited to, a few more components, other components, or additional components in various configurations. In some examples, electronic device 201 and / or electronic device 260 may be implemented in various ways among multiple electronic devices (e.g., as a system). In some such examples, each (or more) electronic device may include one or more of the same components discussed above, such as various sensors, one or more display generation components, one or more speakers, one or more processors, one or more memories, and / or communication circuitry. One or more persons using electronic device 201 and / or electronic device 260 are optionally referred to herein as one or more users of the device.

[0046] Now turn our attention to the interaction with one or more virtual objects displayed in a three-dimensional environment presented at the electronic device (e.g., corresponding to electronic device 201). In some examples, and as will be referenced below... Figures 3A to 3G , Figures 4A to 4C and Figures 5A to 5C In more detail, the three-dimensional environment includes representations of portions of the physical environment and / or representations of real physical objects in the physical environment as shown in video perspective by one or more displays of electronic device 101 (e.g., display 120). In some examples, compared to presenting portions of the physical environment and / or real physical objects as unenhanced video perspective or optical perspective (where portions of the physical environment and / or real physical objects are optically visible through one or more portions of one or more displays or completely transparent portions), electronic device 101 visually enhances the representation of portions of the physical environment and / or the representation of real physical objects, such as, for example, presenting the representation of portions of the physical environment and / or the representation of real physical objects via one or more displays with enhanced visual characteristics (such as higher pixel resolution, increased level of detail, improved sharpness, higher quality, increased size, increased contrast, increased sharpness, increased color vibrancy, increased text readability, less noise, etc.).

[0047] Figures 3A to 3G , Figures 4A to 4C and Figures 5A to 5C Examples of rendering enhanced video perspective in a three-dimensional environment are illustrated according to some examples of this disclosure. Figure 3A An electronic device 101 (e.g., ) that presents a three-dimensional environment 300 (e.g., an extended reality (XR) environment, a computer-generated environment, etc.) according to some examples of this disclosure is illustrated. Figure 1 Electronic device 101; and electronic device 201 in Figure 2). The three-dimensional environment 300 is visible from the user's viewpoint of electronic device 101. In some examples, electronic device 101 is a handheld or mobile device, such as a tablet computer, laptop computer, smartphone, wearable device, or head-mounted display. An example of electronic device 101 is described above with reference to the architectural block diagram of Figure 2. Figure 3A As shown, electronic device 101 and content panel 302a (e.g., whiteboard, corkboard, and / or similar items) are located in the physical environment of three-dimensional environment 300. Figure 3A As illustrated, physical posters 304a, 304b, and 304c include various corresponding contents (e.g., text, images, graphics, and / or similar items) and are affixed to content panel 302a. In some examples, electronic device 101 may be configured to capture an area in the physical environment that includes content panel 302a.

[0048] In some examples, the user's viewpoint of electronic device 101 determines the content visible in the viewport (e.g., a view of a three-dimensional environment visible to the user via one or more displays, such as one or more image sensors 206, or a pair of display modules providing stereoscopic content to different eyes of the same user). In some examples, the (virtual) viewport has a viewport boundary that defines the viewport content via one or more displays (e.g., Figures 3A to 3G , Figures 4A to 4C and Figures 5A to 5CThe viewport (display 120) represents the extent of the three-dimensional environment visible to the user. In some examples, the area defined by the viewport boundary is smaller than the user's visual field in one or more dimensions (e.g., based on the user's visual field, the size of one or more displays, optical properties or other physical characteristics, and / or the position and / or orientation of one or more displays relative to the user's eyes). In other examples, the area defined by the viewport boundary is larger than the user's visual field in one or more dimensions (e.g., based on the user's visual field, the size of one or more displays, optical properties or other physical characteristics, and / or the position and / or orientation of one or more displays relative to the user's eyes). The viewport and viewport boundary typically move with the movement of one or more displays (e.g., with the user's head movement in a head-mounted device, or with the user's hand movement in a handheld device such as a tablet or smartphone). The user's viewpoint determines what is visible in the viewport, generally specifying the position and orientation relative to the three-dimensional environment, and as the viewpoint moves, the view of the three-dimensional environment also moves within the viewport. For head-mounted devices, the viewpoint is typically based on the position and orientation of the user's head, face, and / or eyes to provide a perceptually accurate view of the three-dimensional environment and to provide an immersive experience when the user uses the head-mounted device. For handheld or fixed devices, the viewpoint moves as the handheld or fixed device is moved and / or as the user's positioning relative to the handheld or fixed device changes (e.g., the user moves toward the device, away from the device, and above, below, to the right, and / or to the left of the device). For a device that includes one or more displays with video perspective (or, optionally, referred to as virtual perspective), a portion of the physical environment visible (e.g., displayed and / or projected) via the one or more displays is based on the field of view of one or more cameras communicating with the one or more displays, which typically move with the one or more displays (e.g., with the user's head for a head-mounted device, or with the user's hand for a handheld device such as a tablet or smartphone), because the user's viewpoint moves with the field of view of the one or more cameras (and the appearance of one or more virtual objects displayed via the one or more displays is updated based on the user's viewpoint (e.g., the display positioning and pose of virtual objects are updated based on the movement of the user's viewpoint)).For one or more displays with optical perspective, the portion of the physical environment visible through one or more display generating components (e.g., optically visible through one or more portions or completely transparent portions of the display generating components) is based on the user's field of view through the portions or completely transparent portions of the display generating components (e.g., for head-mounted devices, it moves with the user's head, or for handheld devices such as tablets or smartphones, it moves with the user's hand), because the user's viewpoint moves as the user's field of view through the portions or completely transparent portions of the display generating components moves (and the appearance of one or more virtual objects is updated based on the user's viewpoint).

[0049] In some examples, electronic device 101 performs video perspective enhancement operations on regions and / or physical objects within a three-dimensional environment 300. For example, and as... Figure 3A As illustrated, electronic device 101 displays user interface element 306 (e.g., virtual object) via display 120, which is used to identify regions of interest within a three-dimensional environment 300. It should be understood that while the examples described herein are for objects with specific shapes (such as...),... Figure 3A The user interface element 306 is an illustrated rectangular shape, but it can include many different shapes besides a rectangle, such as a circle or other shapes. In some examples, the user interface element 306 is two-dimensional, similar to a user interface window virtual object. In some examples, the user interface element 306 is three-dimensional, similar to a user interface volume virtual object. Figure 3A The diagonal lines in the user interface element 306 shown are for illustrative purposes only, and their actual display may not reflect the diagonal lines or any other pattern. For example, see the following reference. Figure 5A As illustrated and described, the electronic device 101 optionally displays the user interface element 306 with a certain opacity, so that objects within the three-dimensional environment 300 are visible through the user interface element 306.

[0050] In some examples, the electronic device displays the user interface element 306 at a first size and a first position within the three-dimensional environment in response to user input, or automatically, even when no user input requesting the display of user interface element 306 is detected. For example, when the electronic device 101 displays a three-dimensional environment 300 excluding user interface element 306 via the display 120, the electronic device 101 detects user input via one or more input devices, such as voice input from the user of the electronic device 101 corresponding to a request to perform a video perspective enhancement operation. In some examples, in response to detecting this voice input, the electronic device 101 displays the user interface element 306 at a first size and a first position within the three-dimensional environment 300, such as... Figure 3AAs shown. In another example, when electronic device 101 displays a three-dimensional environment 300 excluding user interface element 306 via display 120, in response to determining that one or more physical objects of the first type are located within the three-dimensional environment 300, electronic device 101 automatically displays user interface element 306 within the three-dimensional environment 300 at a first size and a first position, as shown. Figure 3A As shown (e.g., in the absence of detected user input requesting the display of user interface element 306). For example, the first type of physical object optionally includes physical objects having content (e.g., text, characters, and / or images), such as Figure 3A Physical posters 304a, 304b, and 304c are shown in the image. In some examples, electronic device 101 displays user interface element 306 within the three-dimensional environment 300 based on the determination that the three-dimensional environment 300 includes one or more physical objects of a first type. Thus, in some examples, the automatic display of user interface element 306 and the automatic presentation of options for performing video perspective enhancement operations as described herein, based on the determination that the three-dimensional environment 300 includes one or more physical objects of a first type, informs the user of electronic device 101 that one or more physical objects within the three-dimensional environment 300 are candidate objects for video perspective enhancement. This provides options for enhancing and / or providing more readable content, thereby improving user-device interaction, reducing eye strain, and thus avoiding potential physical discomfort caused by viewing content within the three-dimensional environment 300.

[0051] In some examples, and such as Figure 3A As shown, when the user's attention (e.g., gaze) is directed at user interface element 306, electronic device 101 detects user input via one or more input devices, such as an air pinch gesture 308a (e.g., two or more fingers of the user's hand, such as the thumb and index finger, moving together and touching each other). In some examples, air pinch gesture 308a includes gestures such as... Figure 3A The first position within 300 in the three-dimensional environment shown is as follows: Figure 3B The movement of a second position within the three-dimensional environment 300 is shown. In some examples, in response to detecting the user input, the electronic device 101 performs an action to move the user interface element 306 according to the movement of the air pinch gesture. For example, when the user's attention is directed at the user interface element 306, the electronic device 101 moves the user interface element 306 from such a position as... Figure 3A The first position within the 3D environment 300 shown is moved to, as... Figure 3B The second location within the 3D environment 300 shown. Figure 3BIn this context, the second position of user interface element 306 corresponds to the position of physical poster 304b. Additionally and / or alternatively, electronic device 101 moves user interface element 306 in response to user input that differs from user input based on air pinch gestures, such as contact and movement on a touch-sensitive surface (e.g., a touchscreen, touchpad, or touchpad), movement of a physical input device (e.g., movement of a handheld input device, such as a mouse, stylus, controller, or other motion-tracking device that detects the direction and / or magnitude of movement of the physical input device when it is being held in the user's hand, and / or rotation of a physical click wheel or rotatable input device, such as rotation of a digital crown), voice input, the user's gaze, and / or other predefined gestures or inputs described herein.

[0052] In some examples, in response to detecting that the user interface element 306 is in a second position corresponding to the physical poster 304b, the electronic device 101 performs a video perspective enhancement operation, which includes capturing one or more first images associated with a region contained within the three-dimensional environment 300 within the user interface element 306 via the one or more input devices, and generating one or more second images based on the one or more first images for display via the display 120. In some examples, the one or more second images have a higher pixel resolution than the one or more first images. For example, in Figure 3C In this embodiment, the electronic device displays an image 310 within a user interface element 306, which has a higher pixel resolution than the view of the physical poster 304b via the display 120. Therefore, in some examples, by displaying an image 310 with a higher pixel resolution than the view of the physical poster 304b via the display 120, an enhanced view of the content is provided, which reduces eye strain for the user, thus avoiding potential physical discomfort caused by viewing content within a three-dimensional environment 300. In some examples, one or more second images have different enhanced visual characteristics, such as increased level of detail, improved sharpness, higher quality, increased size, increased contrast, increased sharpness, increased color vibrancy, increased text readability, less noise, etc. In some examples, the electronic device 101 displays image 310 in response to determining that one or more criteria are met. For example, one or more criteria include those that are met when electronic device 101 determines that user interface element 306 is in a second position corresponding to the position of physical poster 304b for a time longer than a predetermined time threshold (e.g., 0.5 seconds, 0.7 seconds, 1 second, 3 seconds, 5 seconds, 10 seconds, 20 seconds, or 30 seconds). In some examples, electronic device 101 performs video perspective enhancement operations in response to the display of user interface element 306 without detecting that user interface element 306 is in a position corresponding to a physical object, such as in Figure 3A In the example.

[0053] In some examples, based on the determination that the current position of the user interface element 306 is within a threshold distance (e.g., 0.1 cm, 0.5 cm, 1 cm, 5 cm, 10 cm, 50 cm, or 100 cm) from the position of the physical poster 304b, the electronic device 101 displays the user interface element 306 at a second position corresponding to the position of the physical poster 304b, such as... Figure 3C As shown. Therefore, in some examples, the electronic device 101 performs a "snapping" operation to snap the user interface element 306 onto the physical poster 304b, such that the user interface element 306 is superimposed on the physical poster 304b. In some examples, the electronic device 101 uses object recognition and tracking to identify the physical poster 304b, and optionally alters the size and / or shape of the user interface element 306 to match the size and / or shape of the physical poster 304b, such that the electronic device 101 completely captures the physical poster 304b. In some examples, and as will be referenced below. Figure 5A As described, electronic device 101 captures a portion of a three-dimensional environment (e.g., a portion of physical poster 304b).

[0054] Figure 3C Another view 312 of the three-dimensional environment 300 displayed via display 120 is also illustrated. View 312 illustrates that the user interface element 306b is a virtual object that includes a captured video image 310b of a portion of the three-dimensional environment 300. For example, the captured video image 310b includes content corresponding to the content of a physical poster 304b in the three-dimensional environment 300, and a portion 314b of the user's hand corresponding to the user's hand 314a in the three-dimensional environment 300. Thus, in some examples, the electronic device displays a user interface element / virtual object that includes a captured video image of the three-dimensional environment in order to provide an enhanced "version" of the captured video image in some examples.

[0055] In some examples, electronic device 101 moves user interface elements, including an image 310 with a higher pixel resolution, in response to user input. For example, in Figure 3C In this context, when the user's attention (e.g., gaze) is directed at user interface element 306, electronic device 101 detects user input via one or more input devices, such as an air pinch gesture 308c (e.g., two or more fingers of the user's hand, such as the thumb and index finger, moving together and touching each other). In some examples, air pinch gesture 308c includes gestures such as... Figure 3C The first position within 300 in the three-dimensional environment shown is as follows: Figure 3DThe movement of a second position within the three-dimensional environment 300 is shown. In some examples, in response to detecting the user input, the electronic device 101 performs an action to move the user interface element 306 according to the movement of the air pinch gesture. For example, when the user's attention is directed at the user interface element 306, the electronic device 101 moves the user interface element 306 from such a position as... Figure 3C The second position within the 3D environment 300 shown is moved to, as... Figure 3D The third location within the 3D environment shown (300). Figure 3D In the image, the third position of user interface element 306 is below the position of physical poster 304b. (As shown...) Figure 3D As shown, the electronic device 101 maintains the display of image 310 within the user interface element 306 during and / or in response to moving the user interface element 306 to the third position. Additionally and / or alternatively, the electronic device 101 moves the user interface element 306 including image 310 in response to user input that differs from user input based on air pinch gestures, such as contact and movement on a touch-sensitive surface (e.g., a touchscreen, touchpad, or touchpad), movement of a physical input device (e.g., movement of a handheld input device, such as a mouse, stylus, controller, or other motion-tracking device that detects the direction and / or amplitude of movement of the physical input device when it is being held in the user's hand, and / or rotation of a physical click wheel or rotatable input device, such as rotation of a digital crown), voice input, the user's gaze, and / or other predefined gestures or inputs described herein.

[0056] In some examples, electronic device 101 adjusts the size of user interface element 306, including an image 310 with a higher pixel resolution, in response to user input. For example, in Figure 3E In this embodiment, electronic device 101 detects user input, including air pinching gestures using the user's first hand 308e and second hand 308ee, and movements of pulling the two hands apart, corresponding to a request to increase the size of user interface element 306, including image 310. In some examples, electronic device 101 detects the user input when the user's attention (e.g., gaze) is directed at user interface element 306. Figure 3F As shown, in response to the user input, the electronic device 101 increases the size of the user interface element 306, including the image 310, according to the movement of the user's first hand 308e and second hand 308ee, until the electronic device 101 detects the release of the air pinch gesture, such as... Figure 3F The hand part 308f is shown in the figure.

[0057] In some examples, electronic device 101 displays user interface element 306, including image 310, in a specific orientation relative to the user's viewpoint. For example, when electronic device 101 displays user interface element 306 including image 310 (the user interface element has a first orientation relative to the user's first viewpoint (e.g., facing the first viewpoint)), electronic device 101 detects movement of the user from the first viewpoint to a second viewpoint (e.g., the user turns their head to view user interface element 306 from the second viewpoint). In some examples, when electronic device 101 detects user movement, electronic device 101 detects user input, such as an air pinch gesture 308g (e.g., two or more fingers of the user's hand, such as the thumb and forefinger, move together and touch each other), when the user's attention (e.g., gaze) is directed at user interface element 306. In some examples, in response to detecting this user movement and user input, electronic device 101 performs an action to move user interface element 306 according to the movement of the air pinch gesture. For example, electronic device 101 moves user interface element 306 from such a position as... Figure 3F The first position within the 3D environment 300 shown is moved to, as... Figure 3G The second position within the three-dimensional environment 300 is shown. In some examples, when and / or in response to moving the user interface element 306 to the second position, the electronic device 101 presents the user interface element 306, including the image 310, in a user-facing second viewpoint orientation, such as... Figure 3G As shown. Displaying the user interface element 306, including image 310, in an orientation based on the user's corresponding viewpoint provides an efficient way of presenting content, thereby reducing the amount of input and providing more efficient interaction between the user and electronic device 101. This enhances the operability of electronic device 101, reduces the power consumption of electronic device 101, reduces errors in the interaction between the user and electronic device 101, and reduces the input required to correct such errors. In some examples, electronic device 101 determines that the user's corresponding viewpoint distance from the corresponding position associated with image 310 exceeds a threshold distance (e.g., the distance described above). For example, electronic device 101 determines that the user's viewpoint corresponds to a viewing angle pointing to the right of the content panel 302a, such that image 310 is not within the user's field of view of electronic device 101. In some examples, based on this determination, electronic device 101 presents a notification via display 120 for recentering the user's viewpoint relative to the corresponding position associated with image 310. In some examples, the notification has a reference... Figure 8E A more detailed description of one or more features of notification 824a.

[0058] Figures 4A to 4C Another example of rendering enhanced video perspective in a three-dimensional environment is illustrated according to some examples of this disclosure. Figure 4A An electronic device 101 (e.g., illustrating a three-dimensional environment 400, such as an extended reality (XR) environment, a computer-generated environment, etc., according to some examples of this disclosure) is shown. Figure 1 Electronic device 101 (and electronic device 201 in Figure 2). The three-dimensional environment 400 is visible from the user's viewpoint of electronic device 101. In some examples, electronic device 101 has Figure 3A One or more characteristics and / or one or more functions of the electronic device 101 in the device. For example... Figure 4A As shown, electronic device 101 and table 402 are located in the physical environment of three-dimensional environment 400. Figure 4A As illustrated, physical paper 404 and physical writing instruments 406 (e.g., pencils, pens, styluses, and / or similar items) are located on table 402. In some examples, electronic device 101 may be configured to capture an area in the physical environment that includes paper 404.

[0059] In some examples, electronic device 101 performs video perspective enhancement operations on physical objects within a three-dimensional environment 400. For example, and as... Figure 4A As illustrated, electronic device 101 identifies paper 404, and based on the determination that paper 404 meets one or more criteria (including criteria met when the paper is of a first type having content (e.g., text, characters, and / or images, as described above), electronic device 101 displays, via display 120, a user interface element 408 (e.g., a virtual object) for identifying a region of interest within the three-dimensional environment 400 at a location corresponding to the paper 404 identified by electronic device 101 within the three-dimensional environment 400. In some examples, user interface element 408 has Figure 3A The user interface element 306 in the middle has one or more characteristics and / or one or more functions.

[0060] In some examples, after identifying paper 404 and displaying user interface element 408, electronic device 101 performs a video perspective enhancement operation, which includes capturing one or more first images of paper 404 contained within user interface element 408 via one or more input devices, and generating one or more second images based on the one or more first images for display via display 120. In some examples, the one or more second images have a higher pixel resolution than the one or more first images. For example, in Figure 4BIn this embodiment, electronic device 101 displays a second user interface element 412, which includes an image with a higher pixel resolution than the view via paper 404 on display 120. In some examples, one or more second images have different enhanced visual characteristics, such as increased level of detail, improved sharpness, higher quality, increased size, increased contrast, increased sharpness, increased color vibrancy, increased text readability, less noise, etc. In some examples, such as... Figure 4B As shown, the second user interface element 412 is displayed within the three-dimensional environment 400 at a location different from the corresponding position on the paper 404. In some examples, the electronic device 101 displays the second user interface element 412 at a specific location, allowing the user to view the second user interface element from a more ergonomic viewing position than that associated with viewing the paper 404. For example, in Figure 4B In this embodiment, the second user interface element 412 displayed by the electronic device 101 is aligned with the user's line of sight and adjacent to the paper 404. In some examples, the electronic device 101 determines the user's line of sight based on the user's attention (e.g., gaze) and / or posture. In some examples, when displaying the second user interface element 412 adjacent to the paper 404 and / or in response to displaying the second user interface element adjacent to the paper, the electronic device 101 changes the orientation of the second user interface element 412, which differs from the orientation of the paper 404 (e.g., independent of the orientation of the paper 404). For example, and as... Figure 4B As shown, the electronic device 101 displays the second user interface element 412 in an orientation that points to the viewpoint of the user of the electronic device.

[0061] Therefore, in some examples, by displaying a second user interface element 412 with a pixel resolution image of the paper 404 that is higher than the view of the paper 404 via the display 120 at a corresponding location adjacent to the paper 404 within the three-dimensional environment 400, the presentation of the enhanced content is improved, making it comfortable and ergonomic. This reduces eye strain for the user, thereby avoiding potential physical discomfort caused by viewing content within the three-dimensional environment 400. In some examples, and as... Figure 4CAs shown, electronic device 101 provides a side-by-side view of physical paper and virtual objects (e.g., a second user interface element 412 including enhanced content). In some examples, electronic device 101 captures video images of a corresponding area of ​​the paper within the 3D environment 400 and displays the computer-enhanced video images via the second user interface element 412. For example, the enhanced video images include content corresponding to the content of the paper 404 in the 3D environment 400, and a portion 414 of the user's hand corresponding to the user's hand 410 in the 3D environment 400. Thus, in some examples, electronic device 101 displays an enhanced view of user-viewable content in real time, such that when the user interacts with the paper 404 using the writing instrument 406, the second user interface element 412 includes an enhanced image based on the captured real-time image, such as... Figure 4C As shown. In some examples, the enhanced content includes enlarged content that is larger than the original content on paper 404. In some examples, the enhanced content includes translated content in a user-selected language that differs from the original content on paper 404.

[0062] Figures 5A to 5C Another example of rendering enhanced video perspective in a three-dimensional environment is illustrated according to some examples of this disclosure. Figure 5A An electronic device 101 (e.g., ) that presents a three-dimensional environment 500 (e.g., an extended reality (XR) environment, a computer-generated environment, etc.) according to some examples of this disclosure is illustrated. Figure 1 Electronic device 101 (and electronic device 201 in Figure 2). The three-dimensional environment 500 is visible from the user's viewpoint of electronic device 101. In some examples, electronic device 101 has Figure 3A One or more characteristics and / or one or more functions of the electronic device 101 in the device. For example... Figure 5A As shown, electronic device 101 and table 504 are located in the physical environment of three-dimensional environment 500. Figure 5A As illustrated, monitor 502 is located on table 504. Figure 5A In this process, electronic device 101 determines that monitor 502 is displaying webpage 506a containing content 506b. In some examples, electronic device 101 may be configured to capture an area of ​​the physical environment including monitor 502.

[0063] In some examples, electronic device 101 performs video perspective enhancement operations on regions within a three-dimensional environment 500. For example, and as... Figure 5A As illustrated, electronic device 101 displays user interface element 508 (e.g., virtual object) via display 120, which is used to identify regions of interest within a three-dimensional environment 500. In some examples, user interface element 508 has Figure 3AThe user interface element 508 in the device 101 has one or more characteristics and / or one or more functions. For example, the electronic device 101 displays the user interface element 508 with a certain opacity, such that a portion 506c of the content 506b displayed on the monitor 502 is visible through the user interface element 508.

[0064] In some examples, electronic device 101 performs video perspective enhancement operations based on determining that the position of user interface element 508 meets one or more criteria. For example, one or more criteria include a criterion that is met when electronic device 101 determines that user interface element 508 is in the corresponding position for more than a predetermined time threshold (e.g., 0.5 seconds, 0.7 seconds, 1 second, 3 seconds, 5 seconds, 10 seconds, 20 seconds, or 30 seconds) and there is no movement based on user input (e.g., as described above). In some examples, one or more criteria include a criterion that is met when a region contained within user interface element 508 (e.g., portion 506c of content 506b) is of a first type having content (e.g., text, characters, and / or images) (e.g., as described above). In some examples, performing video perspective enhancement operations includes capturing one or more first images of portion 506c of content 506b contained within user interface element 508 via one or more input devices, and generating one or more second images based on the one or more first images for display via display 120. In some examples, the one or more second images have a higher pixel resolution than the one or more first images. For example, in Figure 5B In the process, electronic device 101 displays an image of portion 506c of content 506b within a user interface element, the image having, for example, previously displayed in Figure 5B The view of content 506b via display 120 is shown to have a higher pixel resolution than the view of content 506b via display 120. Therefore, in some examples, by displaying an image of content 506b portion 506c with a higher pixel resolution than the view of content 506b via display 120, an enhanced view of the content is provided, which reduces eye strain for the user and avoids potential physical discomfort caused by viewing content within a three-dimensional environment 500. In some examples, one or more second images have different enhanced visual characteristics, such as increased level of detail, improved sharpness, higher quality, increased size, increased contrast, increased sharpness, increased color vibrancy, increased text readability, less noise, etc.

[0065] Additionally or alternatively, electronic device 101 moves user interface element 508 based on movement of the user's attention (e.g., gaze). For example, electronic device 101 detects a movement of the user's attention from a portion 506c of webpage 506a to a second portion of content 506b that is different from that portion 506c. In some examples, in response to the detected movement of the user's attention, electronic device 101 moves user interface element 508 to a second position corresponding to the second portion of content 506b and displays the second content at a higher resolution than the original second portion of content 506b. In some examples, display enhancements include displaying content with one or more visual appearances, such as: a size larger than the corresponding size of the original content 506b displayed by monitor 502; a degree of sharpness and / or clarity greater than the corresponding degree of sharpness and / or clarity of the original content 506b displayed by monitor 502; a degree of color vibrancy (e.g., color saturation, richness, vividness) greater than or stronger than the corresponding degree of color vibrancy of the original content 506b displayed by monitor 502; a degree of text readability (e.g., font attributes, contrast, lighting) that is clearer or more understandable than the corresponding degree of text readability of the original content 506b displayed by monitor 502; and / or visual enhancements (e.g., bolding, underlining, highlighting, lighting, and / or similar items). For example, and as... Figure 5C As shown, electronic device 101 displays content 506b larger than the original content 506b displayed by monitor 502 (as previously shown in...). Figure 5A The size shown (as illustrated) displays enhanced content contained within the user interface element 508. In some examples, as the electronic device 101 moves the user interface element 508, the electronic device 101 displays corresponding content with the same (or similar) pixel resolution and / or visual enhancements as described herein.

[0066] It should be understood that the examples shown and described herein are merely exemplary and additional and / or alternative elements may be provided within a three-dimensional environment for rendering enhanced video perspective in that environment. It should be understood that the appearance, shape, form, and size of each of the various user interface elements and objects shown and described herein are exemplary and alternative appearances, shapes, forms, and / or sizes may be provided. For example, virtual objects representing application windows (e.g., user interface elements 306, 408, and / or 508) may be provided with alternative shapes different from those shown, such as rectangular shapes, circular shapes, triangular shapes, etc. Additionally or alternatively, in some examples, the various options, user interface elements, control elements, etc., described herein may be selected and / or manipulated via user input received through one or more input devices communicating with the electronic device (or multiple electronic devices). For example, selection input may be received via a physical input device (such as a mouse, touchpad, keyboard, etc.) communicating with the electronic device (or multiple electronic devices) or a physical button integrated with the electronic device (or multiple electronic devices).

[0067] Figure 6 Example processes for generating enhanced video perspective in a three-dimensional environment are illustrated according to some examples of this disclosure. In some examples, the above references Figure 1 The electronic device 101 (e.g., an electronic device) described in Figure 2 can perform method 600. For example, method 600 includes the electronic device 101 defining a region of interest (ROI) (606). In some examples, as described above, the ROI is contained within a user interface element or a bounding window (or volume) (e.g., Figures 3A to 3G User interface element 306 or Figures 5A to 5C Within the user interface element 508. In some examples, as mentioned above, the ROI corresponds to a physical object (e.g., Figures 4A to 4C (Paper 404 in the document). Additionally or alternatively, electronic device 101 defines multiple ROIs. For example, electronic device 101 in... Figures 5A to 5C The 3D environment 500 displays more than one user interface element 508. In some examples, the electronic device 101 displays two or more ROIs that overlap each other, optionally displaying a larger ROI than the individual, separate ROIs. In some examples, the ROI is based on the user's hand skeleton pose and / or planar anchor points of the 3D environment. In some examples, the hand skeleton pose is captured based on one or more hand tracking sensors 602 of the electronic device 101. In some examples, planar anchor points are created by the electronic device 101 using one or more world sensing 604 and / or environment / surface detection techniques. In some examples, the ROI is defined as a two-dimensional region in a given camera frame. For example, the electronic device 101 may use two-dimensional homography tracking to track, stabilize, and enhance this two-dimensional region.

[0068] In some examples, after defining the ROI, electronic device 101 captures one or more images of the three-dimensional environment captured by multiple image sensors of electronic device 101 (optionally also referred to as main camera 608). In some examples, electronic device 101 may then perform an input frame alignment and correction process 616 on the captured one or more images (e.g., camera image burst / image sequence) to ensure that the captured one or more images are synchronized, consistent, and high-quality images with minimal motion artifacts in order to output a stable image stream, as described in more detail below.

[0069] Additionally or alternatively, electronic device 101 uses one or more images captured from main camera 608 to perform input frame selection and buffering 610. In some examples, performing input frame selection reduces processing time and associated computational complexity by processing only selected frames (such as, for example, frames that meet quality thresholds related to image resolution, color saturation, sharpness, illumination, or other quality measurements before undergoing further processing). In some examples, buffering improves performance, reduces latency, and reduces jitter by utilizing frames from a recent period and associated data. In some examples, after the input frame selection and buffering 610 process, electronic device 101 performs an input frame alignment and correction process 616 for the resulting frames (e.g., camera image burst / image sequence). In some examples, electronic device 101 utilizes one or more alignment techniques, such as feature-based alignment, optical flow-based alignment, homography transformation alignment, or other alignment techniques. In some examples, electronic device 101 utilizes one or more correction techniques, such as stereo / epipolar correction, or other correction processes.

[0070] In some examples, electronic device 101 utilizes one or more device poses from timestamps of the ROI obtained from world tracking unit 614. In some examples, electronic device 101 includes and / or communicates with world tracking unit 614 to obtain reference coordinates in an AR / VR coordinate system for the position in the 3D environment. Electronic device 101 also obtains camera calibration 612 along with its device pose. This camera calibration includes one or more extrinsic and / or intrinsic camera (e.g., image sensor) parameters (e.g., focal length, principal point, tilt, and / or distortion) of electronic device 101 to extract the ROI from camera image bursts / image sequences, thereby generating multiple ROI images / image bursts. In some examples, electronic device 101 uses homography tracking to track the ROI and utilizes an estimated homography matrix to describe the alignment of multiple ROI images / image bursts from different image sensors of electronic device 101. Therefore, electronic device 101 can perform cropping, correction, alignment, and / or other actions for the ROI images / image bursts.

[0071] Electronic device 101 then performs an image enhancement / super-resolution process 618 on multiple ROI images / image bursts to output an enhanced image stream for output rendering 620 to the display 120 of electronic device 101. In some examples, electronic device 101 provides the user with options to display the image stream and / or enhance the image stream at a selected display frame rate (e.g., high frame rate (frames per rate, FPS)). In some examples, electronic device 101 provides the user with selection of one or more parameters for enhancing the image stream, such as selection of specific text, characters, images, and / or similar items.

[0072] Figure 7 The flowchart illustrates example processes for rendering enhanced video perspective in a three-dimensional environment, according to some examples of this disclosure. In some examples, process 700 begins with an electronic device communicating with one or more input devices (e.g., Figure 3A The first electronic device 101 in the system). In some examples, the electronic device is connected to one or more displays (e.g., Figure 3A The display 120 communicates with the device. In some examples, the electronic device identifies (702) an area within a three-dimensional environment, such as, for example, Figures 3A to 3G The region within the three-dimensional environment 300 is captured by user interface element 306. In some examples, the electronic device captures (704) one or more first images associated with the region identified within the three-dimensional environment via one or more input devices, such as... Figures 3A to 3G The image of the physical poster 304b in the image. In some examples, the electronic device identifier (706) is one or more portions of the first image corresponding to the identified area, such as in Figure 6 The method described in method 600. In some examples, the electronic device generates (710) one or more second images based on the identified corresponding portions of one or more first images, wherein the one or more second images have enhanced visual characteristics relative to the visual characteristics of the one or more first images, such as Figures 3B to 3G Image 310 in the image.

[0073] It should be understood that process 700 is an example, and more, fewer, or different operations may be performed in the same or different order. Additionally, the operations in process 700 described above may optionally be performed via a general-purpose processor (e.g., relative to a general-purpose processor). Figures 2A to 2B One or more functional modules operate in an information processing device such as a (described) or dedicated chip, and / or are... Figures 2A to 2B It is achieved through other components.

[0074] Therefore, according to the foregoing, some examples of this disclosure relate to a method comprising, at an electronic device in communication with one or more input devices: identifying a region within a three-dimensional environment; capturing one or more first images associated with the identified region within the three-dimensional environment via the one or more input devices; identifying a corresponding portion of the one or more first images corresponding to the identified region; and generating one or more second images based on the identified corresponding portion of the one or more first images, wherein the one or more second images have enhanced visual characteristics relative to the one or more first images. Additionally or alternatively, identifying the corresponding portion of the one or more first images corresponding to the identified region includes: determining the pose of the electronic device, and performing alignment and correction operations on the one or more first images based on the pose of the electronic device. Additionally or alternatively, identifying the corresponding portion of the one or more first images corresponding to the identified region includes: performing homography tracking of the images relative to the one or more first images. Additionally or alternatively, the enhanced visual characteristics include higher pixel resolution, increased level of detail, improved sharpness, increased size, increased contrast, increased sharpness, increased color vibrancy, increased text readability, and / or less noise.

[0075] Additionally or alternatively, identifying the region within the three-dimensional environment includes: presenting a user interface element within the three-dimensional environment via one or more displays communicating with the electronic device. Additionally or alternatively, in some examples, the method further includes: when presenting the user interface element within the three-dimensional environment, the electronic device detects user input via the one or more input devices, the user input corresponding to a request to move the user interface element to a second region within the three-dimensional environment different from the first region. In some examples, in response to detecting the user input, based on determining that the user input meets one or more criteria, the electronic device: captures one or more third images associated with the second region within the three-dimensional environment via the one or more input devices; identifies a corresponding portion of the one or more third images corresponding to the second region; and generates one or more fourth images based on the identified corresponding portion of the one or more third images, wherein the one or more fourth images have enhanced visual characteristics relative to the one or more third images. In some examples, in response to detecting the user input, based on determining that the user input does not meet the one or more criteria, the electronic device: presents an indication via the one or more displays that the user input does not meet the one or more criteria.

[0076] Additionally or alternatively, in some examples, the method further includes: presenting a user interface element via one or more displays in communication with the electronic device, the user interface element including the one or more second images overlaid on the area identified within the three-dimensional environment. Additionally or alternatively, in some examples, the method further includes: when presenting the user interface element including the one or more second images overlaid on the area identified within the three-dimensional environment, the electronic device detects user input via the one or more input devices, the user input corresponding to a request to move the user interface element including the one or more second images to a second area within the three-dimensional environment different from the area identified. In some examples, in response to detecting the user input, the electronic device presents the user interface element including the one or more second images overlaid on the second area.

[0077] Additionally or alternatively, in some examples, the method further includes: when presenting the user interface element comprising one or more second images overlaid on the area identified within the three-dimensional environment, the electronic device detects user input via the one or more input devices, the user input corresponding to a request to further enhance a portion of the one or more second images. In response to detecting the user input, the electronic device presents the user interface element comprising the further enhanced portion of the one or more second images overlaid on the area. Additionally or alternatively, identifying the area within the three-dimensional environment includes: identifying objects within the three-dimensional environment, and presenting the user interface element within the three-dimensional environment at a location corresponding to a corresponding position of the identified object within the three-dimensional environment via one or more displays communicating with the electronic device.

[0078] Additionally or alternatively, identifying the area within the three-dimensional environment includes: presenting a user interface element at a first location within the three-dimensional environment via one or more displays communicating with the electronic device, and presenting an object at a second location within the three-dimensional environment different from the first location. In some examples, when the user interface element is presented at the first location within the three-dimensional environment and the object is presented at the second location within the three-dimensional environment, the electronic device detects user input via the one or more input devices, the user input corresponding to a request to move the user interface element from the first location within the three-dimensional environment. In some examples, in response to detecting the user input, the electronic device moves the user interface element within the three-dimensional environment according to the user input. In some examples, based on determining that the user input corresponds to moving the user interface element to a third location within the three-dimensional environment different from the second location and within a threshold distance of the second location of the object, the electronic device moves the user interface element to a corresponding location within the three-dimensional environment different from the third location and corresponding to the object. In some examples, based on determining that the user input corresponds to moving the user interface element to a fourth location within the three-dimensional environment outside the threshold distance of the second location of the object, the electronic device moves the user interface element to the fourth location within the three-dimensional environment. Additionally or alternatively, the method further includes: in response to detecting the user input, and based on determining that the user input corresponds to moving the user interface element to the third position within the three-dimensional environment, the electronic device changes the size of the user interface element to present the user interface element at the corresponding position in a size based on the size of the object. Additionally or alternatively, the corresponding position is adjacent to the second position of the object. Additionally or alternatively, moving the user interface element to the corresponding position includes: changing the orientation of the user interface element, which differs from the orientation of the object. Additionally or alternatively, the method further includes: when capturing the one or more first images, the electronic device determines that one or more criteria are met, including criteria satisfied when movement of the user's viewpoint causes the viewpoint to be located at a distance exceeding a threshold distance from the corresponding position associated with capturing the one or more first images. Additionally or alternatively, the method further includes: in response to determining that the one or more criteria are met, the electronic device presents a notification via one or more displays in communication with the electronic device for recentering the user's viewpoint relative to the corresponding position. Additionally or alternatively, the method further includes: when capturing the one or more first images, the electronic device determines that one or more criteria are met, including criteria that are met when movement of the viewpoint of a user of the electronic device causes the viewpoint to be located at a distance exceeding a threshold distance from a corresponding position associated with capturing the one or more first images.Additionally or alternatively, the method further includes: in response to determining that one or more criteria are met, the electronic device sends information associated with the movement of the viewpoint of the user of the electronic device to a second electronic device in communication with the electronic device, wherein the information causes user interface elements displayed by the second electronic device to be moved according to the movement of the viewpoint of the user of the electronic device.

[0079] Now turn our attention to an example user interaction with an enhanced video displayed on a first electronic device (e.g., corresponding to...). Figure 2A The three-dimensional environment presented at the electronic device 201 in the middle.

[0080] Figures 8A to 8J Examples of rendering enhanced video in a three-dimensional environment are illustrated according to some examples of this disclosure. The examples in these figures are used to illustrate the processes described below, including references to... Figure 9 The process described. Although Figures 8A to 8J An example is given of the first electronic device capable of performing the following relative to Figure 9 Various examples of the methods of describing the process are provided, but it should be understood that these examples are not intended to be limiting, and the first electronic device is capable of operating without reference to these examples. Figures 8A to 8J The method described below is to perform the procedure as described in the reference. Figure 9 The process described.

[0081] Users interact with electronic devices in many different ways. In some examples, and such as... Figure 8A As shown, the first electronic device 101a is connected to one or more displays (e.g., display 120a) and one or more input devices (e.g., in...). Figure 1 The image sensors 114a to 114c are described in more detail below. The examples described below provide a manner in which a first electronic device 101a presents enhanced images and / or video based on a second electronic device different from the first electronic device 101a (e.g., described in more detail below). Figure 8IThe images and / or videos received by the second electronic device 101b. For example, the first electronic device 101a presents an enhanced real-time video feed to a remote user of the first electronic device 101a, which is captured at the second electronic device (e.g., at a location different from the corresponding location of the first electronic device 101a). In some examples, the user experience is improved by automatically enhancing one or more features of the real-time video feed (e.g., without detecting user input for explicitly enhancing the real-time video feed), which provides higher visual quality of the real-time video feed in a way that reduces latency, reduces noise, reduces dependence on the second electronic device, provides resolution upscaling, provides region of interest cropping, and / or provides efficient and resource-saving enhanced stability, thereby enhancing the operability of the electronic device. It should be understood that people use electronic devices. When a person uses an electronic device (such as the first electronic device 101a), that person is optionally referred to as a user of the first electronic device 101a.

[0082] In some examples, and such as Figure 8A As shown, the first electronic device 101a is in contact with the second electronic device (e.g., described in more detail below). Figure 8I In a communication session with the second electronic device 101b, the first electronic device 101a and the second electronic device are configured to communicate (e.g., wirelessly, such as via Wi-Fi, a server (e.g., a wireless communication terminal), or any other wireless communication network) to exchange data, instructions, and / or other indications for performing one or more operations at the first electronic device 101a and / or the second electronic device. In some examples, the first electronic device 101a and the second electronic device are not located in the same physical location. In some examples, the first electronic device 101a and the second electronic device optionally correspond to or resemble the above reference. Figure 1 , Figures 3A to 3G The electronic devices 101 and / or discussed Figures 2A to 2B The electronic device 201 is a second electronic device. In some examples, the second electronic device is optionally referred to as a transmitting device. In some examples, the first electronic device 101a is optionally referred to as a remote device. In an exemplary user case, the first electronic device 101a is associated with a customer service representative configured to provide troubleshooting services to a user (e.g., a client) of the second electronic device. The client is physically co-located with the physical machine 832 that is being troubleshooted. In some examples, the first electronic device 101a receives a live video feed from the machine 832. The first electronic device 101a presents the video feed, enabling the customer service representative to remotely inspect the machine 832 and guide the client as described in the reference. Figures 8A to 8JThe described corrective action. For example, describe an example use case where a customer service representative provides a client with instructions to remove an object from machine 832 (e.g., ...). Figure 8A Remote support for object 828, wherein the object is not intended (e.g., should not) to be located within machine 832 when the machine is operating normally. For example, object 828 may optionally include debris, misplaced machine parts, or other external objects. This object can be detected by first electronic device 101a via a real-time video feed, as referenced in [reference missing]. Figures 8E to 8H As described.

[0083] In some examples, and such as Figure 8A As shown, a first electronic device 101a receives one or more first images 804c (e.g., real-time video streams) from a second electronic device's three-dimensional environment, and renders the one or more first images 804c in a user interface 804a (e.g., virtual objects) within the three-dimensional environment 800a of the first electronic device 101a via a display 120a. In some examples, the three-dimensional environment 800a has Figures 3A to 3G One or more characteristics of the 3D environment 300. For example, in Figure 8A In this context, the 3D environment 800a includes multiple virtual objects (e.g., user interface 804a, user interface element 804d, and user interface element 804e) and real-world objects (e.g., lamp 802). In some examples, such as... Figure 8A As shown in the top view 812, the first electronic device 101a is being used by the first user 810 (e.g., worn on the head of the first user), and the user interface 804a is positioned in front of the first user 810 in the three-dimensional environment 800a.

[0084] In some examples, presenting user interface 804a includes presenting user interface element 804d via display 120a, which identifies the one or more first images 804c as associated with a second electronic device (e.g., "Sally's Window"). In some examples, presenting user interface 804a includes presenting user interface element 804e via display 120a, which, when selected, causes the first electronic device 101a to move user interface 804a within a three-dimensional environment 800a. This user interface includes one or more first images 804c, user interface element 804d, and user interface element 804e. In some examples, user interface 804a includes user interface element 804b, which, when selected, causes the first electronic device 101a to present a second user interface element, such as... Figure 8B The second user interface element 814a. For example, in Figure 8AIn this context, when the attention 806a (e.g., gaze) of a first user 810 of the first electronic device 101a is directed at a user interface element 804b, the first electronic device 101a detects an air pinch gesture 808a. In some examples, the detection of the air pinch gesture 808a has the features described in the reference above. Figure 3A The description describes one or more characteristics of detecting an air pinch gesture 308a. In some examples, in response to detecting an air pinch gesture 808a when the attention 806a (e.g., gaze) of a first user 810 of the first electronic device 101a is directed at a user interface element 804b, the first electronic device 101a presents a second user interface element 814a, including a second user interface component 814b, via a display 120a, such as... Figure 8B As shown. In some examples, the second user interface component 814b has Figure 8A One or more characteristics of the user interface element 804e in the document.

[0085] In some examples, the second user interface element 814a is interactive to select and / or define one or more regions of the first image 804c for visual enhancement, as discussed in more detail below. In some examples, the second user interface element 814a optionally serves as a bounding box or container (e.g., of any shape) for the regions of one or more of the first image 804c. For example, in Figure 8B In this context, the second user interface element 814a includes a transparent frame or window-type user interface element that surrounds a first area within one or more first images 804c. In some examples, the first area included in the second user interface component 814b includes content 818, such as... Figure 8B As shown. For example, content 818 optionally includes machine identification information, measurements, diagnostic information, maintenance instructions, a user interface including information generated / output by the machine, and / or similar items. In some examples, content 818 is provided in a language different from the preferred language of the first user of the first electronic device 101a, or is difficult to read, or is presented at a size or scale insufficient for human reading.

[0086] In some examples, the first electronic device 101a presents one or more second images based on a first area including content 818. For example, in Figure 8B In the process, when the attention 806b of the first user 810 is directed at content 818, the first electronic device 101a detects an air pinch gesture 808b and a voice input 816 requesting translation of content 818. In some examples, the detection of the air pinch gesture 808b has the features described in the above reference. Figure 3AThe description describes one or more features of detecting an air pinch gesture 308a. In some examples, in response to the detection of an air pinch gesture 808b and voice input 816 when the attention 806b of a first user 810 of the first electronic device 101a is directed at content 818, the first electronic device 101a presents one or more second images via a display 120a, such as... Figure 8C Image 818b in the second user interface component 814b corresponds to a visual enhancement of the first region included in the second user interface component 814b. For example, image 818b is presented with enhanced visual characteristics relative to the corresponding visual characteristics of the one or more first images 804 including content 818. For example, and as... Figure 8C As shown, presenting image 818b with enhanced visual characteristics includes translating content 818 into the preferred language of a first user 810, and / or presenting content 818 at a size or scale sufficient for human reading, as shown in image 818b. In some examples, the enhanced visual characteristics include the resolution of image 818b, which is higher than the corresponding resolution of content 818. In some examples, the enhanced visual characteristics include the level of detail, sharpness, quality, contrast, color vibrancy, text readability, and / or sharpness of image 818b, which are higher than the corresponding level of detail, sharpness, quality, contrast, color vibrancy, text readability, and / or sharpness of content 818. In another example, the enhanced visual characteristics include the amount of noise in image 818b, which is lower than the corresponding amount of noise in content 818. In some examples, presenting image 818b includes performing a process that includes one or more characteristics, as described in the above reference. Figure 6 The image enhancement / super-resolution 618 process is described.

[0087] In some examples, the first electronic device 101a initiates a process of adjusting the size and / or moving the second user interface element 814a within its three-dimensional environment 800a. For example, in Figure 8C In this process, a first electronic device 101a detects user input corresponding to a request to move a second user interface element 814a. For example, optionally when the attention 806c of a first user 810 of the first electronic device 101a is directed to a second user interface component 814b of the second user interface element 814a, the first electronic device 101a detects a sequence of one or more inputs including an air pinch gesture. In response to detecting the sequence of one or more inputs, the first electronic device 101a selects the second user interface element 814a. In some examples, the first electronic device 101a detects a sequence of one or more inputs including an air pinch gesture moving in space from a first position 808c to a second position 808cc, such as... Figure 8CAs shown. In response to detecting a sequence of one or more inputs, the first electronic device 101a moves the second user interface element 814a according to the movement of the air pinch gesture. For example, when the attention 806d of the first user 810 of the first electronic device 101a is directed to the second user interface component 814b of the second user interface element 814a, the first electronic device 101a moves the second user interface element 814a from a first position within the three-dimensional environment 800a to a second position within the three-dimensional environment 800a, as shown. Figure 8D As shown. In some examples, moving the second user interface element 814a does not cause the first electronic device 101a to maintain the rendering of image 818b (e.g., maintain the rendering of one or more second images with enhanced visual characteristics). For example, and as shown... Figure 8D As shown, the first electronic device 101 displays a second user interface element 814a via a display 120a. This second user interface element includes a second area of ​​one or more first images 804c, the second area being... Figure 8B The first regions of one or more first images 804c displayed are different and do not include enhanced visual features.

[0088] In some examples, when presenting a second user interface element 814a that includes a second region comprising one or more first images 804c (e.g., at a second location), such as Figure 8D As shown, the first electronic device 101a optionally detects the user's attention, such as attention 806d, directed at the content 818. In some examples, in response to detecting the user's attention directed at the content 818, the first electronic device 101a optionally presents an image 818b (e.g., an enhanced version of the content 818), such as... Figure 8C As stated above.

[0089] In some examples, when presenting a second user interface element 814a that includes a second region comprising one or more first images 804c, such as Figure 8DAs shown, a first electronic device 101a detects an object, such as object 828, in one or more first images 804c. For example, and in relation to customer service representative and troubleshooting use cases, the first electronic device 101a detects object 828 within machine 832, which, according to the specifications of machine 832, is not expected to be present and / or should not be present within machine 832. In some examples, the first electronic device 101a determines that object 828 is not intended to be present within machine 832 based on a visual identification process, which includes one or more operations to: identify machine 832; compare one or more first images 804c with a reference model of machine 832 (e.g., design specifications of the intended configuration of machine 832 and / or historically pre-recorded images of machine 832); and / or analyze the comparison to detect the unintended object 828 and / or any other deviation from the reference model of machine 832. In some examples, the first electronic device 101a obtains the specifications, reference model, and / or pre-recorded model images of the machine 832 from a remote server communicating with the first electronic device 101a, from a local processor used to retrieve specific specifications and / or machine information (e.g., maintained by the first electronic device 101a, optionally from a computer-aided design (CAD) and drafting software application or other application running on the first electronic device 101a), and / or from a second electronic device (e.g., optionally sent from the second electronic device along with the one or more first images).

[0090] In some examples, in response to the detection of object 828, the first electronic device 101a presents an indication via display 120a that object 828 has been detected and / or identified by the first electronic device 101a. In some examples, and as in... Figure 8E As shown, the indication that object 828 has been identified includes presenting a user interface element 822 at a location within the three-dimensional environment 800a of the first electronic device 101a via display 120a, the location corresponding to a corresponding location of object 828 identified within the one or more first images. In some examples, presenting user interface element 822 includes applying visual processing to emphasize object 828 relative to other objects or parts in machine 832, such as highlighting the location of object 828, and / or performing the above reference on the area of ​​the one or more first images 804c including object 828. Figure 6The image enhancement / super-resolution 618 process is described. For example, in response to the detection of object 828, a first electronic device 101a presents one or more second images based on the location of the identified object 828, wherein the one or more second images have enhanced visual characteristics (e.g., level of detail, sharpness, quality, contrast, sharpness, color vibrancy, text readability, and / or any of the above characteristics) relative to the one or more first images. For example, in Figure 8E In this process, the first electronic device 101a presents the one or more second images as a magnifying element displayed by the user interface element 822, such that the object 828 appears magnified relative to the surrounding portion of the one or more second images (e.g., the video feed portion of the machine 832).

[0091] In some examples, and such as Figure 8E As shown, when presenting the one or more second images including the user interface element 822 as described above, the first electronic device 101a presents a notification 824a via the display 120a, which, when selected, causes the first electronic device 101a to send the one or more second images to a second electronic device. For example, the notification 824a includes: a first option 824b, which, when selected, causes the first electronic device 101a to send the one or more second images including the user interface element 822 to the second electronic device; and a second option 824c, which, when selected, causes the first electronic device 101a to forgo sending the one or more second images including the user interface element 822 to the second electronic device. In some examples, before presenting the one or more second images including the user interface element 822, the first electronic device 101a presents a notification via the display 120a to share the one or more second images including the user interface element 822 with the second electronic device. In some examples, the notification to share the one or more second images including the user interface element 822 has... Figure 8E One or more features of notification 824a in the document.

[0092] In some examples, the first electronic device 101a changes the view of one or more second images (e.g., areas of one or more first images 804c contained within a second user interface element 814a). For example, in Figure 8E In this context, when the attention 806e of the first user 810 is directed to one or more regions of the second image (e.g., contained within the second user interface element 814a), the first electronic device 101a detects an air pinch gesture 808e requesting magnification of the second user interface element 814a and voice input 820. In some examples, detecting the air pinch gesture 808e has the following characteristics as described above. Figure 3AThe description refers to one or more characteristics of detecting an air pinch gesture 308a. In some examples, in response to the detection of an air pinch gesture 808e and voice input 820 when the attention 806e of a first user 810 of the first electronic device 101a is directed to an area of ​​one or more second images, the first electronic device 101a presents a magnified view of the one or more second images via a display 120a, such as... Figure 8F The second user interface element 814a is shown in some examples. Presenting this magnified view includes: cropping a region of interest contained within the second user interface element 814a from one or more first images 804c (e.g., video feeds from a second electronic device), and scaling the cropped region, as shown in the example. Figure 8F As shown.

[0093] In some examples, and such as Figure 8F As shown, when the second user interface element 814a is presented in a first position of the three-dimensional environment 800a of the first electronic device 101a, the first electronic device 101a detects a sequence of one or more inputs, the sequence of one or more inputs including: an air pinch gesture (optionally when the attention 806f of the first user 810 of the first electronic device 101a is directed to the second user interface component 814b of the second user interface element 814a), and the movement of the air pinch gesture in space from the first position 808f to the second position 808ff, such as Figure 8F As shown. In response to detecting a sequence of one or more inputs, the first electronic device 101a optionally selects a second user interface element 814a for movement, and moves the second user interface element 814a according to the movement of the air pinch gesture. For example, when the attention 806f of the first user 810 of the first electronic device 101a is directed to the second user interface component 814b of the second user interface element 814a, the first electronic device 101a moves the second user interface element 814a from a first position within the three-dimensional environment 800a to a second position within the three-dimensional environment 800a, as shown. Figure 8G As shown.

[0094] In some examples, moving the second user interface element 814a causes the first electronic device 101a to maintain the view or presentation of the magnified view of the one or more second images. For example, and as... Figure 8GAs shown, the first electronic device 101a presents a magnified view of the second user interface element 814a, including the one or more second images, at a second location within the three-dimensional environment 800a via a display 120a, as shown in top view 812. In some examples, the first electronic device 101a also performs actions to: de-dock (e.g., release or detach) the second user interface element 814a from a first location within the three-dimensional environment 800a still occupied by the user interface 804a including the one or more first images 804c; and move the second user interface element 814a to a second location within the three-dimensional environment 800a according to the movement of an air pinch gesture, such as... Figure 8G The top view 812 is shown in the figure.

[0095] In some examples, a first electronic device 101a presents one or more annotations overlaid on one or more second images and one or more first images. In some examples, when the first electronic device 101a presents a second user interface element 814a including one or more second images, the first electronic device 101a detects user input corresponding to a request to add annotations to one or more second images. For example, and as... Figure 8G As shown, detecting user input corresponding to a request to add annotation includes: detecting a sequence of one or more inputs when the attention 806g of the first user 810 is directed to one or more regions of the second image (e.g., contained within the second user interface element 814a), the sequence of one or more inputs including an air pinch gesture 808g and a voice input 838 requesting annotation (e.g., "Add arrow here"). In some examples, detecting the air pinch gesture 808g has the same characteristics as described above. Figure 3A The description describes one or more features of detecting an air pinch gesture 308a. In some examples, in response to the detection of an air pinch gesture 808g and voice input 838 when the attention 806g of a first user 810 of the first electronic device 101a is directed to one or more areas of a second image, the first electronic device 101a displays an arrow icon 836a or a graphic via a display 120a. In some examples, the sequence in which the first electronic device 101a detects the one or more inputs includes the air pinch gesture moving in space from a first position of the air pinch gesture 808g to a second position 808gg, such as... Figure 8G As shown. In response to detecting a sequence of one or more inputs, the first electronic device 101a moves the arrow icon 836a according to the movement of the air pinch gesture. For example, the first electronic device 101a moves the arrow icon 836a to, as shown in the image. Figure 8G The position shown. In some examples, movement and as... Figure 8GAfter the arrow icon 836a is displayed at the indicated location, the first electronic device 101a transmits one or more second images including the arrow icon 836a to the second electronic device, causing the display of the second electronic device to display a representation corresponding to the arrow icon 836a. For example, and as shown... Figure 8I As shown, corresponding to Figure 8G The arrow icon 836a, represented by 836b, is presented as an overlay on machine 832 within the environment of the second electronic device 101b. In some examples, the sending and presentation of one or more annotations (such as the representation 836b corresponding to the arrow icon 836a) enables the first electronic device 101a and / or the first user 810 of the first electronic device 101a to guide the second user to locate object 828. For example, and as... Figure 8G As shown, the hand 826 of the second user was detected at the position corresponding to arrow icon 836a, and... Figure 8H In the middle, the second user's hand 826 is presented as removing object 828 from machine 832.

[0096] Figure 8I and Figure 8J An example is illustrated where a first electronic device 101a moves a user interface 804a, including one or more first images 804, according to the movement of the viewpoint of a user of a second electronic device. For example, Figure 8I The three-dimensional environment 800b of the second user 830 of the second electronic device 101b is further illustrated. For example... Figure 8I As shown, a first portion of the three-dimensional environment 800b of the second electronic device 101b (e.g., included within the user interface 834a) is presented within the three-dimensional environment 800a of the first electronic device 101b via the user interface 804a. In some examples, the presented first portion originates from a first viewpoint of a second user 830 of the second electronic device 101b, which corresponds to the second user 830's head tilted downward relative to the machine 832, as illustrated in the side view of the second user 830. In some examples, the first portion in the three-dimensional environment 800b includes the hand 826 of the second user 830 holding an object 828 (which has been removed from its position within the machine 832, as referenced above). Figure 8H (Similar description). This removal was performed by the second user 830 using a guide indicating the location of object 828 within machine 832, represented by 836b (e.g., corresponding to...). Figure 8HThe arrow icon 836 in the image is generated, positioned, and sent to the second electronic device 101b by the first electronic device 101a. In some examples, the first electronic device 101a presents the user interface 804a at a first viewing angle (e.g., in a position and orientation corresponding to the optimal viewing angle automatically determined by the first electronic device 101a based on image content, user preferences, and / or in a manner that presents relevant features of the machine 832 at the optimal viewing angle).

[0097] In some examples, the first electronic device 101a receives information associated with the movement of the viewpoint of the second user 830 in relation to the second electronic device 101b. For example, this information includes information from sources such as... Figure 8I The first viewpoint shown moves to, as Figure 8J The second viewpoint is shown, where the second viewpoint corresponds to the upward tilt of the second user 830's head relative to the machine 832. In some examples, in response to receiving this information, the first electronic device 101a moves the user interface 804a to a corresponding position within the three-dimensional environment 800a of the first electronic device 101a, and presents a second portion of the three-dimensional environment 800b from the second viewpoint, which corresponds to the upward tilt of the second user 830's head relative to the machine 832, such as... Figure 8J As shown. For example, as Figure 8J As shown, the first electronic device 101a moves the user interface 804a vertically upward relative to its viewpoint in the three-dimensional environment 800a, and updates the display of one or more first images 804c to correspond to portions of the physical environment of the second user 830 contained within the user interface 834a at the second electronic device 101b (e.g., including hand 826 and machine 832). In some examples, the first electronic device 101a aligns the user interface 804a with the updated viewpoint of the second user 830 to ensure that the first user 810 of the first electronic device 101a perceives the corresponding portion (e.g., video feed) in the three-dimensional environment 800b with an orientation corresponding to the field of view of the second user 830, thereby enabling the first user 810 to provide more precise guidance to the second user 830 as one of the advantages.

[0098] Figure 9 The flowchart illustrates example processes for rendering enhanced video in a three-dimensional environment according to some examples of this disclosure. In some examples, process 900 begins with a first electronic device communicating with one or more displays and one or more input devices (e.g., Figure 8A The first electronic device 101a). In some examples, when a user interface element (902) (such as a second electronic device) is displayed in a three-dimensional environment including one or more first images of a three-dimensional environment via the one or more displays, the second electronic device is displayed. Figure 8AWhen a user interface 804a includes one or more first images 804, the first electronic device identifies (904) a region within the one or more first images, such as Figure 8B The area contained in the second user interface element 814a. In some examples, the first electronic device presents (906) one or more second images based on the identified area via the one or more displays, such as Figure 8C Image 818b, wherein the one or more second images have enhanced visual characteristics relative to the corresponding visual characteristics of the one or more first images, such as, for example, Figure 8C Image 818b includes enlarged, translated text.

[0099] It should be understood that process 900 is an example, and more, fewer, or different operations may be performed in the same or different order. Additionally, the operations in process 900 described above may optionally be performed via a general-purpose processor (e.g., relative to a general-purpose processor). Figures 2A to 2B One or more functional modules operate in an information processing device such as a (described) or dedicated chip, and / or are... Figures 2A to 2B It is achieved through other components.

[0100] Therefore, based on the foregoing, some examples of this disclosure relate to a method comprising: at a first electronic device communicating with one or more input devices, one or more displays, and a second electronic device. In some examples, when a user interface element comprising one or more first images in a three-dimensional environment including the second electronic device is presented via the one or more displays, the first electronic device identifies a region within the one or more first images and presents one or more second images based on the identified regions via the one or more displays, wherein the one or more second images have enhanced visual characteristics relative to the corresponding visual characteristics of the one or more first images. Additionally or alternatively, the enhanced visual characteristics include a resolution of the one or more second images that is higher than the corresponding resolution of the one or more first images. Additionally or alternatively, the enhanced visual characteristics include a level of detail, sharpness, quality, contrast, color vibrancy, text readability, or sharpness of the one or more second images that is higher than the corresponding level of detail, sharpness, quality, contrast, color vibrancy, text readability, or sharpness of the one or more first images. Additionally or alternatively, the enhanced visual characteristics include a noise level of the one or more second images that is lower than the corresponding noise level of the one or more first images.

[0101] Additionally or alternatively, identifying a region within one or more first images includes a first electronic device displaying a second user interface element comprising one or more second images within the three-dimensional environment of the first electronic device via one or more displays. Additionally or alternatively, the second user interface element may be optionally configured to initiate a process of resizing and / or moving the second user interface element within the three-dimensional environment of the first electronic device. Additionally or alternatively, identifying a region within one or more first images includes the first electronic device identifying an object within one or more first images and displaying a second user interface element comprising one or more second images via one or more displays, wherein the second user interface element is displayed at a location within the three-dimensional environment of the first electronic device corresponding to the corresponding location of the object identified within the one or more first images. Additionally or alternatively, identifying a region within one or more first images includes the first electronic device detecting user input via one or more input devices, the user input corresponding to a request to capture a region within the one or more first images; and displaying a second user interface element comprising one or more second images within a user interface element via one or more displays based on the user input.

[0102] Additionally or alternatively, in some examples, the method further includes: before presenting one or more second images based on the identified area, the first electronic device presents a notification via one or more displays indicating that the identified area is shared with the second electronic device. Additionally or alternatively, in some examples, the method further includes: when presenting one or more second images based on the identified area, the first electronic device presents a notification via one or more displays, which, when selected, causes the first electronic device to send one or more second images to the second electronic device. Additionally or alternatively, in some examples, the one or more second images are presented as contained within a second user interface element, and the second user interface element is presented at a location in the three-dimensional environment of the first electronic device that differs from the corresponding location of the user interface element including the one or more first images within the three-dimensional environment of the first electronic device.

[0103] Additionally or alternatively, in some examples, the method further includes: when a second user interface element is presented at the location in the three-dimensional environment of the first electronic device, the first electronic device detects user input via one or more input devices, the user input corresponding to a request to move the second user interface element to a second region within the three-dimensional environment of the first electronic device. Additionally or alternatively, in some examples, in response to detecting user input, the first electronic device moves the second user interface element to a second region within the three-dimensional environment of the first electronic device based on the user input, without moving the user interface including one or more first images. Additionally or alternatively, in some examples, the method further includes: when presenting one or more second images based on an identified region, the first electronic device detects user input via one or more input devices, the user input corresponding to a request to add annotations to one or more second images. Additionally or alternatively, in some examples, in response to detecting user input, the first electronic device adds annotations to one or more second images based on the user input, and sends the one or more second images including the annotations to the second electronic device. Additionally or alternatively, the one or more displays include a head-mounted display. Additionally or alternatively, in some examples, the method further includes: when presenting user interface elements of one or more first images in a three-dimensional environment including the second electronic device, the first electronic device receives information associated with movement of the viewpoint of a user of the second electronic device via one or more input devices. Additionally or alternatively, in some examples, in response to receiving the information, the first electronic device moves the user interface elements of one or more first images in the three-dimensional environment including the second electronic device to a position within the three-dimensional environment of the first electronic device according to the movement of the viewpoint of the user of the second electronic device.

[0104] Some examples of this disclosure relate to an electronic device comprising: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described above.

[0105] Some examples of this disclosure relate to a non-transitory computer-readable storage medium that stores one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the methods described above.

[0106] Some examples of this disclosure relate to an electronic device that includes one or more processors, a memory, and components for performing any of the methods described above.

[0107] Some examples of this disclosure relate to an information processing apparatus used in an electronic device, the information processing apparatus including components for performing any of the methods described above.

[0108] This disclosure envisions, in some examples, that the data utilized may include personal information data that uniquely identifies a particular person or can be used to contact or locate a particular person. Such personal information data may include demographic data, content consumption activity, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information. Specifically, as described herein, one aspect of this disclosure is tracking user engagement with content.

[0109] This disclosure recognizes that the use of such personal information data in the techniques of this invention can be used to benefit users. For example, personal information data can be used to display suggested text that changes based on changes in user engagement. For example, the suggested text can be updated based on changes in a user's reading preferences and / or health history.

[0110] This disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with robust privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to privacy policies and measures that are recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable entity purposes and should not be shared or sold outside of these legitimate purposes. Furthermore, such collection / sharing should be conducted only after receiving informed consent from users. Additionally, such entities should consider taking any necessary steps to protect and safeguard the right to access such personal information data and ensure that other entities with access to personal information data comply with the privacy policies and procedures of other entities. Furthermore, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and privacy practices. Moreover, policies and measures should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including considerations of specific jurisdictions. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.

[0111] Regardless of the foregoing, this disclosure also envisions examples of users selectively blocking the use or access to personal information data. That is, this disclosure contemplates providing hardware and / or software components to prevent or block access to such personal information data. For example, the inventive technology can be configured to allow a user to opt-in or opt-out during or at any time after registering for the service. In another example, a user may choose not to enable the recording of personal information data in a specific application (e.g., a first application and / or a second application). In addition to providing "opt-in" and "opt-out" options, this disclosure also contemplates providing notifications related to access to or use of personal information. For example, a user may be notified when collection is initiated that their personal information data will be accessed, and subsequently reminded again before access to the personal information data on one or more devices.

[0112] Furthermore, the intent of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by restricting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0113] For purposes of explanation, the foregoing description has been presented with reference to specific examples. However, the illustrative arguments above are not intended to be exhaustive or to limit this disclosure to the precise form disclosed. Many modifications and variations are possible in accordance with the foregoing teachings. The examples were chosen and described to best elucidate the principles of this disclosure and its practical application, thereby enabling others skilled in the art to make optimal use of this disclosure with various modifications suitable for the particular intended purpose, as well as the various described examples.

Claims

1. A method, the method comprising: At an electronic device that communicates with one or more input devices: Identify areas within a 3D environment; Capture one or more first images associated with the region identified within the three-dimensional environment via the one or more input devices; Identify the corresponding portion of the one or more first images that corresponds to the identified region; as well as One or more second images are generated based on the identified corresponding portions of the one or more first images, wherein the one or more second images have enhanced visual characteristics relative to the one or more first images.

2. The method of claim 1, wherein identifying the corresponding portion of the one or more first images corresponding to the identified region comprises: Determine the orientation of the electronic device; as well as Alignment and correction operations are performed on the one or more first images based on the posture of the electronic device.

3. The method of claim 1, wherein identifying the corresponding portion of the one or more first images corresponding to the identified region comprises: Perform homography tracking of the image relative to the one or more first images.

4. The method of claim 1, wherein the enhanced visual characteristics include higher pixel resolution, increased level of detail, improved sharpness, increased size, increased contrast, increased sharpness, increased color vibrancy, increased text readability, and / or less noise.

5. The method of claim 1, wherein identifying the region within the three-dimensional environment comprises: User interface elements are presented in the three-dimensional environment via one or more displays that communicate with the electronic device.

6. The method according to claim 5, further comprising: When the user interface element is presented in the three-dimensional environment, user input is detected via the one or more input devices, and the user input corresponds to a request to move the user interface element to a second area in the three-dimensional environment that is different from the first area. In response to the detection of the user input: Based on the determination that the user input meets one or more criteria: Capture one or more third images associated with the second region within the three-dimensional environment via the one or more input devices; Identify the corresponding portion of the one or more third images that corresponds to the second region; as well as One or more fourth images are generated based on the identified corresponding portions of the one or more third images, wherein the one or more fourth images have enhanced visual characteristics relative to the visual characteristics of the one or more third images; as well as Based on the determination that the user input does not meet the one or more criteria, an indication that the user input does not meet the one or more criteria is presented via the one or more displays.

7. The method according to claim 1, further comprising: User interface elements are presented via one or more displays that communicate with the electronic device, the user interface elements including the one or more second images overlaid on the area identified within the three-dimensional environment.

8. The method according to claim 7, further comprising: When presenting the user interface element comprising one or more second images superimposed on the area identified in the three-dimensional environment, user input is detected via the one or more input devices, the user input corresponding to a request to move the user interface element comprising the one or more second images to a second area in the three-dimensional environment that is different from the area identified in the first area; as well as In response to the detection of the user input, the user interface element comprising the one or more second images overlaid on the second area is presented.

9. The method according to claim 7, further comprising: When presenting the user interface element including one or more second images overlaid on the area identified in the three-dimensional environment, user input is detected via the one or more input devices, the user input corresponding to a request to further enhance a portion of the one or more second images; as well as In response to the detection of the user input, the user interface element is presented as a further enhanced portion of the one or more second images overlaid on the area.

10. The method of claim 1, wherein identifying the region within the three-dimensional environment comprises: Identify objects within the three-dimensional environment; as well as User interface elements are presented at locations corresponding to the positions of the objects identified in the three-dimensional environment, via one or more displays that communicate with the electronic device.

11. The method of claim 1, wherein identifying the region within the three-dimensional environment comprises: User interface elements are presented at a first location within the three-dimensional environment via one or more displays that communicate with the electronic device, and objects are presented at a second location within the three-dimensional environment that is different from the first location. When the user interface element is presented at the first position in the three-dimensional environment and the object is presented at the second position in the three-dimensional environment, user input is detected via the one or more input devices, the user input corresponding to a request to move the user interface element from the first position in the three-dimensional environment; as well as In response to the detection of the user input: Moving the user interface element in the three-dimensional environment based on the user input includes: Based on the determination that the user input corresponds to a third position within the three-dimensional environment that is different from the second position and within a threshold distance of the second position of the object, the user interface element is moved to a corresponding position that is different from the third position and corresponds to the object; as well as Based on the determination that the user input corresponds to a fourth position within the three-dimensional environment, which is outside the threshold distance of the second position of the object, the user interface element is moved to the fourth position within the three-dimensional environment.

12. The method according to claim 11, further comprising: In response to the detection of the user input: Based on the determination that the user input corresponds to moving the user interface element to the third position within the three-dimensional environment: The size of the user interface element is changed so that it is presented at the corresponding location at a size based on the size of the object.

13. The method according to claim 11, wherein: The corresponding position is adjacent to the second position of the object; and Moving the user interface element to the corresponding position includes changing the orientation of the user interface element, which is different from the orientation of the object.

14. The method according to claim 1, further comprising: When capturing the one or more first images, it is determined that one or more criteria are met, including criteria that are met when the movement of the viewpoint of the user of the electronic device causes the viewpoint to be located at a distance exceeding a threshold distance from the corresponding position associated with capturing the one or more first images; as well as In response to determining that one or more criteria are met, a notification is presented via one or more displays in communication with the electronic device to recenter the user's viewpoint relative to the corresponding position.

15. The method according to claim 1, further comprising: When capturing the one or more first images, it is determined that one or more criteria are met, including criteria that are met when the movement of the viewpoint of the user of the electronic device causes the viewpoint to be located at a distance exceeding a threshold distance from the corresponding position associated with capturing the one or more first images; as well as In response to determining that one or more criteria are met, information associated with the movement of the viewpoint of the user of the electronic device is sent to a second electronic device in communication with the electronic device, wherein the information causes user interface elements displayed by the second electronic device to be moved according to the movement of the viewpoint of the user of the electronic device.

16. An electronic device, the electronic device comprising: One or more processors; Memory; as well as One or more programs, the programs being stored in the memory and configured to be executed by the one or more processors, the programs including instructions for performing the method according to any one of claims 1 to 15.

17. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 15.

Images