Standard-based opportunistic manipulation of displayed content

Abstract

The invention relates to standard-based opportunistic manipulation of displayed content. Some examples of the present disclosure relate to a system and method for displaying content, such as a representation of a content item, in a three-dimensional environment presented at an electronic device and updating the display of the content in response to movement of the electronic device and meeting a set of criteria. Examples of the present disclosure relate to improving the user experience by automatically updating a representation of the content item when certain conditions are met, such as when the orientation of the electronic device relative to the three-dimensional environment is appropriate (e.g., the criteria for updating the representation of the content item are met).

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 642,597, filed May 3, 2024, and U.S. Patent Application No. 19 / 185,742, filed April 22, 2025, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] This disclosure relates in general to systems and methods for displaying content such as representations of content items or user interface elements and manipulating that content based on satisfying associated standards. Background Technology

[0004] Some computer graphics environments provide two-dimensional and / or three-dimensional environments, where at least some of the objects displayed for user viewing are virtual and computer-generated. In some examples, a physical environment (e.g., including one or more physical objects) is optionally presented in the three-dimensional environment along with one or more virtual objects. Some computer graphics environments provide two-dimensional and / or three-dimensional environments (e.g., extended reality environments), where at least some of the objects displayed for user viewing are virtual and computer-generated. In some examples, objects displayed in the three-dimensional environment (e.g., including virtual user interfaces, such as virtual navigation user interfaces) are configured to be interactive (e.g., via direct or indirect input provided by the user). In some examples, objects are displayed in the three-dimensional environment with a corresponding visual appearance (e.g., including virtual user interfaces, the level of detail of the virtual user interface, the number of user interface objects included in the virtual user interface, the size of the virtual user interface, etc.). In some examples, objects are configured to move within the three-dimensional environment based on movement of the user's viewpoint (e.g., movement of the user's head and / or torso). In some examples, after the user's viewpoint moves, an unexpected or unintended view of the object is presented to the user in the three-dimensional environment (e.g., including an unexpected or unintended visual appearance). Summary of the Invention

[0005] Some examples of this disclosure relate to systems and methods for displaying and updating the display of content, such as representations of content items, in a computer-generated environment. In some examples, an electronic device captures a portion of one or more physical environments (e.g., indoor and / or outdoor environments) within the field of view of one or more cameras, and presents representations of one or more physical objects and content items within the one or more physical environments via one or more displays. In some examples, the electronic device detects movement of the electronic device and, in response, updates the representation of the content item based on determining that one or more criteria are met. In some examples, updating the representation of the content item may include scaling the size of the representation of the content item or cropping or trimming the content item based on meeting the one or more criteria. In some examples, the updating of the representation of the content item may be sequential or discrete and is limited to a range of movement thresholds relative to a predefined reference frame.

[0006] A full description of these examples is provided in the accompanying drawings and detailed embodiments, and it should be understood that the content of this invention does not limit the scope of this disclosure in any way. Attached Figure Description

[0007] To better understand the various examples described herein, reference should be made to the following detailed embodiments and accompanying drawings. Throughout the drawings, similar reference numerals generally refer to corresponding parts.

[0008] Figure 1 Examples of electronic devices that present extended real-world environments according to some examples of this disclosure are illustrated.

[0009] Figures 2A to 2B A block diagram illustrating an example architecture of an electronic device according to some examples of this disclosure is shown.

[0010] Figure 3A Examples of electronic devices according to this disclosure are illustrated, which are displaying a representation of a content item but have not yet made any movement (e.g., and / or detected any movement) and have not yet met any criteria associated with updating the representation of the content item.

[0011] Figure 3B Examples of electronic devices according to this disclosure are illustrated, which have scrolling direction movement (relative to) in a physical environment before satisfying one or more criteria associated with the representation of updated content items. Figure 3A (Example).

[0012] Figure 3B-1 Examples of electronic devices according to this disclosure are illustrated, which are similar to Figure 3B Examples include visual cues.

[0013] Figure 3B-2 Examples of electronic devices according to this disclosure are illustrated, which are similar to Figure 3B Examples include visual cues located in predefined areas of content items.

[0014] Figures 3C to 3E Examples of electronic devices with a scrolling direction in a physical environment are illustrated according to some examples of this disclosure, the electronic devices satisfying one or more criteria associated with the representation of updated content items.

[0015] Figure 3F Examples of movement in a rolling direction (relative to) in a physical environment according to this disclosure are illustrated. Figure 3A (Example) An electronic device that no longer meets one or more criteria associated with the representation of the updated content item.

[0016] Figures 3G to 3I Examples of electronic devices according to this disclosure are illustrated, which have pitch direction movement in a physical environment.

[0017] Figures 3I to 3K Examples of electronic devices according to this disclosure are illustrated, which have yaw direction movement in a physical environment.

[0018] Figures 4A to 4B Examples of electronic devices according to this disclosure are illustrated, which display representations of content items without predetermined visual cues in corner or boundary areas and / or without playback features.

[0019] Figures 5A to 5C Examples of electronic devices according to this disclosure are illustrated, which have been moving sequentially around a yaw direction in a physical environment.

[0020] Figure 6 This is a flowchart illustrating some examples of an example process according to the present disclosure, which is used to display content and automatically update content such as the representation of content items based on the detection of movement of an electronic device and according to the satisfaction of one or more criteria. Detailed Implementation

[0021] Some examples of this disclosure relate to systems and methods for displaying and updating the display of content, such as representations of content items, in a computer-generated environment. In some examples, an electronic device captures a portion of one or more physical environments (e.g., indoor and / or outdoor environments) within the field of view of one or more cameras, and presents representations of one or more physical objects and content items within the one or more physical environments via one or more displays. In some examples, the electronic device presents content items overlaid on a view of the one or more physical environments via one or more transparent or semi-transparent displays. In some examples, the electronic device detects movement of the electronic device and, in response, updates the representation of the content item based on determining that one or more criteria are met. In some examples, updating the representation of the content item may include scaling the size of the representation of the content item or cropping or trimming the content item based on meeting the one or more criteria. In some examples, the updating of the representation of the content item may be sequential or discrete and is limited to a range of movement thresholds relative to a predefined reference frame.

[0022] In some examples, 3D objects are displayed in a computer-generated 3D environment with a specific orientation that controls one or more behaviors of the 3D object (e.g., when the 3D object moves within the 3D environment). In some examples, the orientation in which the 3D object is displayed in the 3D environment is selected by the user of the electronic device or is selected automatically by the electronic device. For example, when initiating the rendering of a 3D object in a 3D environment, the user may select a specific orientation of the 3D object, or the electronic device may automatically select the orientation of the 3D object (e.g., based on the type of the 3D object).

[0023] In some examples, 3D objects may be displayed in a 3D environment with world-locked orientation, body-locked orientation, tilt-locked orientation, or head-locked orientation, as described below. As used herein, an object displayed in a 3D environment with body-locked orientation has a distance and orientation offset relative to a portion of the user's body (e.g., the user's torso). Alternatively, in some examples, body-locked objects have a fixed distance from the user, and the orientation of the content does not reference any part of the user's body (e.g., it may be displayed in the same basic direction relative to the user, regardless of head and / or body movement). Additionally or alternatively, in some examples, body-locked objects may be configured to always maintain gravity or horizon (e.g., perpendicular to gravity) alignment, such that changes in the head and / or body's scrolling direction will not cause the body-locked object to move within the 3D environment. Instead, translational movement in either configuration will cause the body-locked object to reposition within the 3D environment to maintain the distance offset.

[0024] As used in this article, objects displayed in a head-locked orientation within a 3D environment have a distance and orientation offset relative to the user's head. In some examples, the head-locked object moves within the 3D environment as the user's head moves (when the user's viewpoint changes).

[0025] As used in this article, objects displayed in a world-locked orientation in a 3D environment do not have a distance or orientation offset relative to the user.

[0026] As used herein, an object displayed in a tilt-locked orientation in a 3D environment (referred to herein as a tilt-locked object) has a distance offset relative to a user (such as a part of the user's body, e.g., the user's torso, or the user's head). In some examples, the tilt-locked object is displayed in a fixed orientation relative to the 3D environment. In some examples, the tilt-locked object moves according to a polar (e.g., spherical) coordinate system centered on a pole passing through the user (e.g., the user's head). For example, the tilt-locked object moves in the 3D environment based on the movement of the user's head within a spherical space surrounding the user's head (e.g., centered on the user's head). Thus, if the user tilts their head relative to gravity (e.g., up or down in the pitch direction), the tilt-locked object will follow the head tilt and move radially along the sphere, such that the tilt-locked object is repositioned in the 3D environment with the same distance offset relative to the user as before the head tilt, while optionally maintaining the same orientation relative to the 3D environment. In some examples, if the user moves their head relative to gravity in a rolling direction (e.g., clockwise or counterclockwise), the tilt-locked object is not repositioned in the 3D environment.

[0027] Figure 1 An electronic device 101 is illustrated according to some examples of this disclosure, which presents a three-dimensional environment (e.g., an extended reality (XR) environment or a computer-generated reality (CGR) environment that optionally includes representations of physical and / or virtual objects). In some examples, such as Figure 1 As shown, electronic device 101 is a head-mounted display or other head-mounted device configured to be worn on the head of a user of electronic device 101. See below for reference. Figure 2A An example of an architecture block diagram to describe electronic device 101. For example... Figure 1 As shown, electronic device 101 and table 106 are located in a physical environment. The physical environment may include physical features such as physical surfaces (e.g., floor, wall) or physical objects (e.g., table, lamp, etc.). In some examples, electronic device 101 may be configured to detect and / or capture images of the physical environment including table 106 (exemplified in the field of view of electronic device 101).

[0028] In some examples, such as Figure 1As shown, the electronic device 101 includes one or more internal image sensors 114a oriented toward the user's face (e.g., referred to below). Figures 2A to 2B (The described eye-tracking camera). In some examples, an internal image sensor 114a is used for eye tracking (e.g., detecting the user's gaze). The internal image sensor 114a is optionally arranged on the left and right portions of the display 120 to enable eye tracking of the user's left and right eyes. In some examples, the electronic device 101 also includes external image sensors 114b and 114c facing outwards from the user to detect and / or capture the physical environment of the electronic device 101 and / or movement of the user's hands or other body parts.

[0029] In some examples, display 120 has a field of view visible to the user. In some examples, the user-visible field of view is the same as the field of view of external image sensors 114b and 114c. For example, when display 120 is optionally part of a head-mounted device, the field of view of display 120 is optionally the same as or similar to the field of view of the user's eyes. In some examples, the user-visible field of view is different from the field of view of external image sensors 114b and 114c (e.g., narrower than the field of view of external image sensors 114b and 114c). In other examples, the field of view of display 120 may be smaller than the field of view of the user's eyes. The user's viewpoint determines what is visible in the field of view, and the viewpoint typically specifies the position and orientation relative to the three-dimensional environment. As the user's viewpoint shifts, the field of view of the three-dimensional environment will also shift accordingly. In some examples, electronic device 101 may be an optically transparent device, through which display 120 is a transparent or translucent display through which parts of the physical environment can be directly viewed. In some examples, display 120 may be included within a transparent lens and may overlap entirely or partially with the transparent lens. In other examples, the electronics may be a video pass-through device, wherein display 120 is an opaque display configured to display an image of the physical environment using images captured by external image sensors 114b and 114c. Although Figure 1 A single display is shown, but it should be understood that display 120 optionally includes more than one display. For example, display 120 optionally includes displays that are integrated (e.g., by the user's brain) to create Figure 1 The content view shown is displayed on a stereoscopic display pair (e.g., a left display panel and a right display panel for the user's left and right eyes, respectively). Some examples are shown below. Figures 2A to 2B In more detail, the display 120 includes or corresponds to a transparent or translucent surface (e.g., a lens) that is not equipped with display capabilities (e.g., and therefore cannot generate and display the virtual object 104), and alternatively presents a direct view of the physical environment in the user's field of view (e.g., the user's eye's field of view).

[0030] In some examples, electronic device 101 is configured to display virtual object 104 in a three-dimensional environment (e.g., in response to a trigger). Virtual object 104 is... Figure 1 The illustrated cube represents a virtual object that does not exist in the physical environment but is displayed in a three-dimensional environment as positioned on top of table 106 (e.g., a real-world table or a representation thereof). Optionally, in response to detecting a plane of table 106 in the physical environment 100, virtual object 104 may be displayed on the surface of table 106 in the three-dimensional environment displayed via display 120 of electronic device 101.

[0031] It should be understood that virtual object 104 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or other three-dimensional virtual objects) in a three-dimensional environment. For example, a virtual object may represent an application or user interface displayed in a three-dimensional environment. In some examples, a virtual object may represent content corresponding to an application and / or displayed via a user interface in a three-dimensional environment. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g., air gestures, such as air pinch gestures, air tap gestures, and / or air touch gestures), allowing the user to virtually touch, tap, move, rotate, or otherwise interact with virtual object 104.

[0032] As discussed in this article, by the user (e.g., using Figure 1 One or more air pinch gestures performed by the hand (103) are detected by one or more input devices of the electronic device 101 and interpreted as one or more user inputs pointing to content displayed by the electronic device 101. Additionally or alternatively, in some examples, one or more user inputs interpreted by the electronic device 101 as pointing to content displayed by the electronic device 101 (e.g., virtual object 104) are detected via one or more hardware input devices (e.g., controllers, touchpads, proximity sensors, buttons, sliders, knobs, etc.) rather than via one or more input devices configured to detect air gestures (such as one or more air pinch gestures) performed by the user. This description is exemplary and not limiting; the user may optionally use different air gestures and / or other forms of input to provide user input.

[0033] In some examples, electronic device 101 may be configured to communicate with a second electronic device (such as an accompanying device). For example, as Figure 1As illustrated, electronic device 101 optionally communicates with electronic device 160. In some examples, electronic device 160 corresponds to a mobile electronic device, such as a smartphone, tablet computer, smartwatch, laptop computer, or other electronic device. In some examples, electronic device 160 corresponds to a non-mobile electronic device, which is typically stationary and not easily moved within a physical environment (e.g., a desktop computer, server, etc.). See below for reference. Figure 2B The following are additional examples of architectural diagrams used to describe electronic device 160. In some examples, electronic device 101 and electronic device 160 are associated with the same user. For example, in Figure 1 In this embodiment, electronic device 101 may be positioned (e.g., mounted) on the user's head, and electronic device 160 may be positioned near electronic device 101, such as in the user's hand 103 (e.g., hand 103 holding electronic device 160), in the user's pocket or bag, or on a surface near the user. Electronic devices 101 and 160 may optionally be associated with the same user account (e.g., the user logs into that user account on both electronic devices 101 and 160). See below for further details. Figures 2A to 2B Additional details regarding the communication between electronic device 101 and electronic device 160 are provided.

[0034] In some examples, displaying an object in a 3D environment is triggered by or enables interaction with one or more user interface objects in the 3D environment. For example, initiating the display of an object in a 3D environment may include interaction with one or more virtual option / function representations displayed in the 3D environment. In some examples, when initiating the display of an object in a 3D environment, the electronic device may track the user's gaze as input for identifying one or more virtual option / function representations as targets for selection. For example, a gaze may be used to identify one or more virtual option / function representations as targets for selection using another selection input. In some examples, a virtual option / function representation may be selected using hand-tracking input detected via an input device communicating with the electronic device. In some examples, an object displayed in a 3D environment may move and / or reorient itself in the 3D environment based on movement input detected via an input device.

[0035] In the following description, an electronic device that communicates with one or more displays and one or more input devices is described. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as touch-sensitive surfaces, physical keyboards, mice, joysticks, hand-tracking devices, eye-tracking devices, styluses, etc. Furthermore, as mentioned above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed among two or more devices. Therefore, as used in this disclosure, information on or displayed by the electronic device is optionally used to describe information output by the electronic device for display on a separate display device (touch-sensitive or non-touch-sensitive). Similarly, as used in this disclosure, input received on the electronic device (e.g., touch input received on a touch-sensitive surface of the electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on a separate input device from which the electronic device receives input information.

[0036] The device typically supports a variety of applications, such as one or more of the following: drawing applications, rendering applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, phone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, TV channel browsing applications, and / or digital video player applications.

[0037] Figures 2A to 2B Block diagrams illustrating example architectures of electronic devices according to some examples of this disclosure are provided. In some examples, electronic device 201 and / or device 260 include one or more electronic devices. For example, electronic device 201 may be a portable device, an auxiliary device for communicating with another device, a head-mounted display, a head-mounted speaker, etc. In some examples, electronic device 201 corresponds to the above reference. Figure 1 The described electronic device 101. In some examples, electronic device 260 corresponds to the above reference. Figure 1 The described electronic device 160.

[0038] like Figure 2A As illustrated, electronic device 201 optionally includes one or more sensors, such as one or more hand tracking sensors 202, one or more position sensors 204A, and one or more image sensors 206A (optionally corresponding to...). Figure 1The electronic device 201 may include an internal image sensor 114a and / or external image sensors 114b and 114c, one or more touch-sensitive surfaces 209A, one or more motion and / or orientation sensors 210A, one or more eye-tracking sensors 212, one or more microphones 213A or other audio sensors, one or more body tracking sensors (e.g., torso and / or head tracking sensors), etc. The electronic device 201 may optionally include one or more output devices, such as one or more display generation components 214A (optionally corresponding to...). Figure 1 The electronic device 201 may include a display 120, one or more speakers 216A, one or more tactile output devices (not shown), etc. The electronic device 201 optionally includes one or more processors 218A, one or more memories 220A, and / or communication circuitry 222A. One or more communication buses 208A are optionally used for communication between the components of the electronic device 201 mentioned above.

[0039] Additionally, electronic device 260 optionally includes components that are the same as or similar to those of electronic device 201. For example, such as Figure 2B As shown, electronic device 260 optionally includes one or more position sensors 204B, one or more image sensors 206B, one or more touch-sensitive surfaces 209B, one or more orientation sensors 210B, one or more microphones 213B, one or more display generating components 214B, one or more speakers 216B, one or more processors 218B, one or more memories 220B, and / or communication circuitry 222B. One or more communication buses 208B are optionally used for communication between the components of electronic device 260 mentioned above.

[0040] Electronic devices 201 and 260 are optionally configured to communicate via a wired or wireless connection between the two electronic devices (e.g., via communication circuits 222A, 222B). For example, as Figure 2A As indicated, electronic device 260 can be used as an accessory device to electronic device 201. For example, in some examples, electronic device 260 processes sensor inputs from electronic devices 201 and 260 and / or uses the display generation component 214A of electronic device 201 to generate content for display.

[0041] Communication circuits 222A and 222B optionally include circuitry for communicating with electronic devices, networks (such as the Internet, intranets, wired and / or wireless networks, cellular networks, and wireless local area networks (LANs)). Communication circuits 222A and 222B optionally include circuitry for using near-field communication (NFC) and / or short-range communication (such as...) The communication circuits 222A and 222B communicate with each other. In some examples, the communication circuits 222A and 222B include or support Wi-Fi (e.g., the 802.11 protocol), Ethernet, Ultra Wideband (“UWB”), high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems) or any other communication protocol or any combination thereof.

[0042] One or more processors 218A, 218B include one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, one or more processors 218A, 218B include one or more microprocessors, one or more central processing units, one or more application-specific integrated circuits, one or more field-programmable gate arrays, one or more programmable logic devices, or combinations of such devices. In some examples, memories 220A and / or 220B are non-transitory computer-readable storage media (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions configured to be executed by processors 218A, 218B to perform the techniques, processes, and / or methods described herein. In some examples, memories 220A and / or 220B may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium can be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, and device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on compact disc (CD), digital versatile optical disc (DVD), or Blu-ray technology, and persistent solid-state storage devices (such as flash memory, solid-state drives, etc.).

[0043] In some examples, one or more display generating components 214A, 214B include a single display (e.g., a liquid crystal display (LCD), an organic light-emitting diode (OLED), or other type of display). In some examples, one or more display generating components 214A, 214B include multiple displays. In some examples, one or more display generating components 214A, 214B may include a touch-enabled display (e.g., a touchscreen), a projector, a holographic projector, a retinal projector, a transparent or translucent display, etc. In some examples, the electronic device does not include one or more display generating components 214A or 214B. For example, as an alternative to one or more display generating components 214A or 214B, some electronic devices include transparent or translucent lenses or other surfaces not configured to display or present virtual content. However, it should be understood that in such cases, electronic device 201 and / or electronic device 260 are optionally equipped with Figure 2A and Figure 2B One or more of the other components illustrated and described herein, such as one or more hand-tracking sensors 202, one or more eye-tracking sensors 212, one or more image sensors 206A, and / or one or more motion and / or orientation sensors 210A. Alternatively, in some examples, one or more display generation components 214A or 214B are provided separately from electronic devices 201 and / or 260. For example, one or more display generation components 214A, 214B communicate with electronic device 201 (and / or electronic device 260) but are not integrated with electronic device 201 and / or electronic device 260 (e.g., within the housing of electronic devices 201, 260). In some examples, electronic devices 201 and 260 include one or more touch-sensitive surfaces 209A and 209B, respectively, for receiving user input, such as tap input and swipe input or other gestures (e.g., hand-based gestures or finger-based gestures). In some examples, one or more display generating components 214A, 214B and touch-sensitive surfaces 209A, 209B form one or more touch-sensitive displays (e.g., touchscreens integrated with each of the electronic devices 201 and 260 or touchscreens external to each of the electronic devices 201 and 260 that communicate with each of the electronic devices 201 and 260).

[0044] Electronic devices 201 and 260 optionally include one or more image sensors 206A and 206B, respectively. The image sensors 206A and 206B optionally include one or more visible light image sensors (such as charge-coupled device (CCD) sensors) and / or complementary metal-oxide-semiconductor (CMOS) sensors operable to acquire images of physical objects from a real-world environment. The image sensors 206A and 206B also optionally include one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensors 206A and 206B also optionally include one or more cameras configured to capture movement of physical objects in the real-world environment. The image sensors 206A and 206B also optionally include one or more depth sensors configured to detect the distance between the physical object and the electronic devices 201 and 260. In some examples, information from one or more depth sensors allows a device to identify objects in a real-world environment and distinguish them from other objects in the real-world environment. In some examples, one or more depth sensors allow a device to determine the texture and / or shape of objects in a real-world environment. In some examples, one or more image sensors 206A or 206B are included in an electronic device different from electronic devices 201 and / or 260. For example, one or more image sensors 206A, 206B communicate with electronic devices 201, 260 but are not integrated with electronic devices 201, 260 (e.g., within the housing of electronic devices 201, 260). Specifically, in some examples, one or more cameras of one or more image sensors 206A, 206B are integrated with and / or coupled to one or more separate devices from electronic devices 201 and / or 260 (e.g., but communicate with electronic devices 201 and / or 260), such as one or more input and / or output devices (e.g., one or more speakers and / or one or more microphones, such as headphones or headsets) that include one or more image sensors 206A, 206B. In some examples, electronic device 201 or electronic device 260 corresponds to a headphone speaker (e.g., headphones or earbuds). In such cases, electronic device 201 or electronic device 260 is equipped with Figure 2A and Figure 2B A subset of the other components illustrated and described herein. In some such examples, electronic device 201 or electronic device 260 is equipped with one or more image sensors 206A, 206B, one or more motion and / or orientation sensors 210A, 210B, and / or speakers 216A, 216B.

[0045] In some examples, electronic devices 201 and 260 combine a CCD sensor, an event camera, and a depth sensor to detect the physical environment surrounding them. In some examples, one or more image sensors 206A and 206B include a first image sensor and a second image sensor. The first and second image sensors work cooperatively and are optionally configured to capture different information about physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor, and the second image sensor is a depth sensor. In some examples, electronic devices 201 and 260 use one or more image sensors 206A and 206B to detect the location and orientation of electronic devices 201 and 260 and / or one or more display generating components 214A and 214B in the real-world environment. For example, electronic devices 201 and 260 use one or more image sensors 206A and 206B to track the location and orientation of one or more display generating components 214A and 214B relative to one or more stationary objects in the real-world environment.

[0046] In some examples, electronic devices 201 and 260 include one or more microphones 213A and 213B or other audio sensors, respectively. Electronic devices 201 and 260 optionally use one or more microphones 213A and 213B to detect sound from a user and / or the user's real-world environment. In some examples, the one or more microphones 213A and 213B include microphone arrays (e.g., multiple microphones) that optionally operate in cooperation, such as to identify ambient noise or locate sound sources in the space of a real-world environment.

[0047] Electronic devices 201 and 260 each include one or more position sensors 204A and 204B for detecting the positions of electronic device 201 and / or one or more display generating components 214A and electronic device 260 and / or one or more display generating components 214B, respectively. For example, the one or more position sensors 204A, 204B may include a Global Positioning System (GPS) receiver that receives data from one or more satellites and allows electronic devices 201, 260 to determine the absolute position of the electronic device in the physical world.

[0048] Electronic devices 201 and 260 each include one or more orientation sensors 210A and 210B for detecting the orientation and / or movement of electronic device 201 and / or one or more display generating components 214A and the orientation and / or movement of electronic device 260 and / or one or more display generating components 214B, respectively. For example, electronic devices 201 and 260 use one or more orientation sensors 210A and 210B to track changes in the positioning and / or orientation of electronic devices 201 and 260 and / or one or more display generating components 214A and 214B, such as changes relative to physical objects in a real-world environment. The one or more orientation sensors 210A and 210B optionally include one or more gyroscopes and / or one or more accelerometers.

[0049] In some examples, electronic device 201 includes one or more hand tracking sensors 202 and / or one or more eye tracking sensors 212. It should be understood that, although referred to as hand tracking sensors or eye tracking sensors, electronic device 201 may additionally or alternatively include one or more other body tracking sensors, such as one or more leg tracking sensors, one or more torso tracking sensors, and / or one or more head tracking sensors. One or more hand tracking sensors 202 are configured to track the localization and / or position of one or more portions of a user's hand, and / or the movement of one or more portions of the user's hand relative to a three-dimensional environment, relative to one or more display generation components 214A, and / or relative to another defined coordinate system. One or more eye tracking sensors 212 are configured to track the localization and movement of a user's gaze (e.g., the user's attention, more generally including the eyes, face, or head) relative to the real world or three-dimensional environment and / or relative to one or more display generation components 214A. In some examples, one or more hand tracking sensors 202 and / or one or more eye tracking sensors 212 are implemented together with one or more display generation components 214A. In some examples, one or more hand-tracking sensors 202 and / or one or more eye-tracking sensors 212 are implemented separately from one or more display generation components 214A. In some examples, the electronic device 201 optionally does not include one or more hand-tracking sensors 202 and / or one or more eye-tracking sensors 212. In some such examples, one or more display generation components 214A may be used by the electronic device 260 to provide a three-dimensional environment, and the electronic device 260 may utilize input and other data collected via other one or more sensors of the electronic device 201 (e.g., one or more position sensors 204A, one or more image sensors 206A, one or more touch-sensitive surfaces 209A, one or more motion and / or orientation sensors 210A and / or one or more microphones 213A or other audio sensors) as input and data processed by one or more processors 218B of the electronic device 260. Additionally or alternatively, the electronic device 260 optionally does not include Figure 2B Other components shown include one or more position sensors 204B, one or more image sensors 206B, and / or one or more touch-sensitive surfaces 209B. In some such examples, one or more display generation components 214A may be used by electronic device 260 to provide a three-dimensional environment, and electronic device 260 may utilize input and other data collected via one or more motion and / or orientation sensors 210A (and / or one or more microphones 213A) of electronic device 201 as input.

[0050] In some examples, one or more hand-tracking sensors 202 (and / or other body-tracking sensors, such as leg-tracking sensors, torso-tracking sensors, and / or head-tracking sensors) may use one or more image sensors 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that capture 3D information from the real world, including one or more body parts (e.g., a human user's hand, leg, torso). In some examples, sufficient resolution is available to distinguish the hand to differentiate the fingers and their corresponding positions. In some examples, one or more image sensors 206A are positioned relative to the user to define a field of view and interaction space for one or more image sensors 206A, in which the finger / hand positions, orientations, and / or movements captured by the image sensors are used as input (e.g., to differentiate from the user's resting hand or other hands of other people in the real-world environment). Tracking the fingers / hands used for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0051] In some examples, one or more eye-tracking sensors 212 include at least one eye-tracking camera (e.g., an IR camera) and / or an illumination source (e.g., an IR light source, such as an LED) that emits light toward the user's eyes. The eye-tracking camera may be pointed at the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by the respective eye-tracking camera and illumination source, and focus / gaze can be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by one or more respective eye-tracking cameras / illumination sources.

[0052] Electronic devices 201 and 260 are not limited to Figures 2A to 2B The components and configurations may include, but are not limited to, a few components, other components, or additional components in various configurations. In some examples, electronic device 201 and / or electronic device 260 may be implemented in various ways among multiple electronic devices (e.g., as a system). In some such examples, each(s) of the electronic devices may include one or more of the same components discussed above, such as various sensors, one or more display generation components, one or more speakers, one or more processors, one or more memories, and / or communication circuitry. One or more persons using electronic device 201 and / or electronic device 260 are optionally referred to herein as one or more users of the devices.

[0053] The focus now is on interaction with one or more virtual objects (e.g., representations of content items) displayed in a three-dimensional environment (e.g., an extended reality environment) presented at an electronic device (e.g., corresponding to electronic device 201). As used herein, content items include any displayable content, such as images (e.g., photographs, graphics, etc.), videos (television programs, movies, live broadcasts, etc.), user interface elements, etc. Examples of this disclosure relate to improving the user experience by automatically manipulating the display of the representation of content items in response to the detection of movement of the electronic device when certain conditions are met, which causes portions of the physical environment, the three-dimensional environment, and / or the representation of the content items displayed via a display generation component to update according to the movement of the electronic device.

[0054] Figures 3A to 3K Electronic devices representing display content items according to some examples of this disclosure are illustrated. Electronic device 301 may be similar to electronic devices 101 or 201 discussed above, and / or may be a head-mounted system / device and / or a projection-based system / device (including hologram-based systems / devices) configured to generate and present a three-dimensional environment, such as, for example, a head-up display (HUD), a head-mounted display (HMD), a window with integrated display capabilities, or a display formed as a lens (e.g., similar to a contact lens) designed to be placed on a human eye. Figures 3A to 3E In the examples, a user optionally wears electronic device 301 in a three-dimensional environment 350, which may be defined by the X, Y, and Z axes viewed from the perspective of the electronic device (e.g., the viewpoint associated with the user of electronic device 301). Electronic device 301 may be configured to move based on the movement of the user (e.g., the user's head) (e.g., in six degrees of freedom), such that electronic device 301 may move in the X, Y, or Z direction, the roll direction, the pitch direction, and / or the yaw direction. Although the X, Y, and Z directions are described, electronic device 301 may use any suitable coordinate system to track its position and / or orientation. In some examples, electronic device 301 may be located within an area of ​​an indoor environment (e.g., in a specific room). In some examples, electronic device 301 may be moved to a new area within the indoor environment (e.g., moved to a different room). In some examples, the field of view of one or more cameras of electronic device 301 is updated as electronic device 301 is moved. Although Figures 3A to 3E The example illustrates an example of electronic device 301 rotating counterclockwise and updating content item 310 in response to that rotation, but in other examples, the electronic device may rotate clockwise and perform a similar update to the content item.

[0055] Figure 3AExamples of electronic devices according to this disclosure are illustrated, which are displaying a representation of a content item but have not yet made any movement (e.g., and / or detected any movement) and have not yet met any criteria associated with updating the representation of content item 310. For example... Figure 3A As shown, electronic device 301 can be positioned in a physical environment (e.g., an indoor environment) that includes multiple real-world objects. Figure 3A In some examples, electronic device 301 may be oriented toward physical objects within an indoor physical environment 375 (such as window 312) and may present a representation of the physical objects. In some examples, the three-dimensional environment 350 presented using electronic device 301 optionally includes a captured portion of the physical environment 375 surrounding electronic device 301. In some examples, the user's field of view may be a subset of the field of view of one or more cameras, and the field of view of one or more cameras may include portions of the three-dimensional environment 350 that are larger than the user's field of view. In other examples, the user's field of view may be equivalent to the field of view of one or more transparent or semi-transparent displays, and a portion of the three-dimensional environment 350 may be presented within the field of view of one or more transparent or semi-transparent displays. Therefore, although in some cases the visible field of view presented to the user in the electronic device may be described herein as being provided by one or more cameras (e.g., of electronic device 301), it should be understood that the presented field of view is not limited to this, and the field of view may alternatively be based on the field of view of one or more semi-transparent or transparent displays. Therefore, in some examples, the representation of physical objects in the field of view of one or more cameras may include portions of the physical environment viewed through the transparent or semi-transparent displays of electronic device 301.

[0056] In some examples, electronic device 301 may display a representation of content item 310 and evaluate one or more criteria associated with updating the representation of content item 310 in all indoor environments, only in limited indoor environments (e.g., a residence or office), or only in certain rooms of a residence or office. In other examples, electronic device 301 may display a representation of content item 310 and evaluate one or more criteria associated with updating the representation of content item 310 in other indoor environments (such as a hotel room, a friend's house, a non-public space, etc.) or outdoor environments.

[0057] The representation of content item 310 can be displayed at a scale predetermined by system settings or user preferences. In some examples, the content item can be so-called "playing content," causing the display to continuously update the content being presented. In some examples, the playing content item being presented can be a movie, TV series, television program, music video, or any other content item that includes visual content. In some examples, the representation of the content item can be a user interface element of a currently running application that includes visual content. In other examples, the representation of the content item may not include playing content, but can instead be an image (e.g., a photograph) captured or downloaded on electronic device 301.

[0058] In some examples, the displayed representation of content item 310 occupies a portion of the three-dimensional environment 350 and has an initial size and / or a first visual state (e.g., upon receiving a launch request, or when the representation is automatically launched). As shown, the representation of content item 310 has a rectangular shape. It should be understood that in some examples, the representation of content item 310 may have a circular shape or other shapes suitable for the type of content being displayed. In some examples, the initial size of the representation of the content item may be predetermined based on system settings. Alternatively, the first visual state and / or initial size of the representation of the content item may be customized and / or personalized based on user preferences, needs, and / or intentions.

[0059] In some examples, the user, electronic device, and / or one or more physical objects in the indoor or outdoor physical environment may move around within the environment. In some examples, the electronic device detects movement of itself, one or more physical objects in the indoor or outdoor physical environment, and / or the user, and upon detecting such movement, changes the field of view (including representations of one or more physical objects within the field of view of one or more cameras). Depending on the changed field of view, previously invisible physical objects may optionally become visible in the changed field of view.

[0060] In some examples, the display of content item 310 may be adjusted in size (e.g., decrease or increase size) or in angle (e.g., update the orientation of the content item relative to the electronic device in response to an offset in the angle or orientation of the electronic device).

[0061] In some examples, the presentation of one or more content items 310 may be bound to and / or associated with corresponding predetermined and / or user-defined locations or corresponding physical objects in the physical environment 375, such that the presentation of one or more content items occurs only when the corresponding location or corresponding physical object in the physical environment 375 is visible in the field of view of one or more cameras of the electronic device, and / or only when the electronic device is within a distance threshold from the corresponding predetermined and / or user-defined location in the physical environment 375 or within a distance threshold from the corresponding physical object in the physical environment 375. Changes to the presentation of one or more content items (e.g., reducing or increasing area, aspect ratio, etc.) may be a function of the distance between the electronic device 301 and the associated predetermined and / or user-defined location and / or physical object when one or more content items are fixed in place.

[0062] Alternatively, in some examples, in response to the detection of movement of electronic device 301 and / or user, one or more content items 310 may dynamically update and / or move according to the detected movement, such that one or more content items maintain their presentation within the three-dimensional environment 350. In some examples, in response to the detection of movement of electronic device 301 and / or user, one or more content items may transition from being presented in a first visual state to being presented in a second visual state different from the first visual state in the three-dimensional environment 350. In some examples, the transition of content items may be delayed by a time (e.g., 0.5 seconds or 1 second) to maintain the impression of responsive content items while avoiding potential user dizziness from more transient visual feedback. Generally, the display of content items may transition from a first visual state to a second visual state. In some examples, the content items are picture-in-picture (PiP) content. Although in Figure 3A Not shown in the example, but as PiP content, the representation of content item 310 is optionally displayed in a smaller size, which optionally partially or completely covers a larger content item that is different from the representation of content item 310.

[0063] In some examples, electronic device 301 selectively alters the visual state of the representation of content item 310 in the three-dimensional environment 350 based on the movement of the electronic device. For example, in Figure 3AIn this context, the representation of content item 310 may be tilt-locked in the three-dimensional environment 350 (as defined above). In some examples, because the representation of content item 310 is tilt-locked (e.g., displayed in a fixed orientation relative to the three-dimensional environment), the representation of content item 310 may not be repositioned in the three-dimensional environment 350 in response to movement of electronic device 301 (e.g., clockwise or counterclockwise scrolling of the device). In some examples, the representation of content item 310 may be considered to be rotated in the opposite direction to the rotation of the electronic device to counteract the rotation of the electronic device and maintain its fixed orientation relative to the three-dimensional environment 350. As described above, in some examples, in response to determining that one or more criteria associated with updating the representation of content item 310 have been met (e.g., detecting that movement of electronic device 301 exceeds a movement threshold (e.g., an angle threshold)), electronic device 301 transitions between displaying the representation of content item 310 in the three-dimensional environment 350 in a first visual state to displaying the representation of content item 310 in a second visual state different from the first visual state, as discussed in more detail below. In some examples, if electronic device 301 determines that one or more criteria associated with the representation of updated content item 310 have not yet been met, electronic device 301 maintains the representation of content item 310 in a first visual state.

[0064] In some examples, determining that one or more criteria are met can trigger an automatic update of the representation of content item 310 to improve the user experience by variably displaying the desired content item in the updated view with minimal user input (e.g., without gestures, navigating the user interface, pressing buttons, etc.). Several non-limiting example criteria associated with updating the representation of content item 310 will now be discussed. Figure 3A In some examples, one or more criteria associated with updating the representation of content item 310 may include criteria that are met when the movement of electronic device 301 is at or above a movement threshold. In some examples, if the movement of electronic device 301 exceeds the movement threshold, electronic device 301 may switch from displaying a representation of content item 310 in a first visual state to displaying a representation of content item 310 in a second visual state different from the first visual state. Figures 3A to 3E As illustrated in the illustrations, in some examples, reference ray 321 (to which the motion threshold is measured) corresponds to a ray that is perpendicular to both gravity and ray 323 (which is also perpendicular to gravity and extends away from the electronic device 301 to a point on the horizon of the physical environment in the user's field of view) (e.g., from...). Figure 3A From this perspective, ray 323 is guided "into the page". (As...) Figures 3A to 3EAs shown in the illustration, reference ray 321 (to which the movement threshold is measured) corresponds to a ray that points approximately to the right and is parallel to the x-axis 329 of the electronic device 301. Therefore, reference ray 321 is independent of the orientation of the electronic device 301 in the three-dimensional environment 350.

[0065] In some examples, a reference ray 321 is established based on the calibration of electronic device 301 (to which a movement threshold is measured). For example, when content is first initiated on electronic device 301 (e.g., such as in...). Figure 3A In this context, following a previous user interaction corresponding to a request to initiate content associated with a representation of content item 310, or at some other time during operation, the electronic device 301 may prompt the user (e.g., visually (e.g., via visual cues, such as text prompts) and / or auditorily (e.g., via audio output)) to face forward and look straight ahead in the three-dimensional environment 350, because the user's natural (e.g., comfortable) forward-facing head tilt (e.g., along one or both of the "tilt" axis and the "roll" axis) may not necessarily be perpendicular to gravity and parallel to the horizon. Once the user has complied with the prompt, the user may provide input to the electronic device 301 to set the reference ray 321 parallel to the x-axis 329 of the electronic device (but not necessarily parallel to the horizon). In other examples, the user may provide input to the electronic device 301 at any time or after other prompts (but not necessarily prompts to face forward and look straight ahead), regardless of the current orientation of the electronic device. This allows, for example, a user to set the reference ray 321 parallel to the x-axis of the electronic device 301, even when the device is heavily tilted relative to the horizontal line of the three-dimensional environment 350, such as when oriented in a side-lying position (e.g., heavily rolled to the left or right).

[0066] In some examples, the movement threshold corresponds to an angular movement threshold. In some examples, if electronic device 301 detects a sufficient change in the angle between the x-axis 329 of electronic device 301 and the reference ray 321 (e.g., illustrated in Figure 320) (e.g., greater than 3 degrees, 5 degrees, 8 degrees, 10 degrees, etc.), the angular movement of electronic device 301 may exceed a counterclockwise angular movement threshold (threshold-ccw) 325 or a clockwise angular movement threshold (threshold-cw) 327. Exceeding the angular movement threshold in either scrolling direction (e.g., clockwise or counterclockwise relative to the reference ray 321) may trigger a transition from displaying a representation of content item 310 in a first visual state to displaying a representation of content item 310 in a second visual state different from the first visual state.

[0067] In other examples, the angular movement threshold does not distinguish angular direction but corresponds to a magnitude in polar coordinates relative to reference ray 321. For example, the angular movement threshold may be set to a magnitude of 10 polar coordinates relative to reference ray 321 in Figure 320. In this example, a 10-degree clockwise roll (e.g., -10 polar coordinates relative to the reference ray) or a 10-degree counterclockwise roll (e.g., +10 polar coordinates relative to the reference ray) satisfies the angular movement threshold because the magnitude of the polar coordinates is 10 polar coordinates in both cases. In other words, if electronic device 301 detects an angular movement of electronic device 301 in any rolling direction relative to the reference ray with a magnitude greater than the angular movement threshold, it can be determined that the movement of electronic device 301 exceeds the angular movement threshold.

[0068] It should be understood that in some examples, a total motion threshold may be established, which may include an angular motion threshold and / or additional or alternative thresholds, such as a distance threshold, a time threshold, a velocity threshold, an acceleration threshold, a jerk threshold, or motion relative to a ray in other directions (e.g., yaw, pitch, or roll). Based on determination that the angular motion threshold and any other additional or alternative thresholds have been met (e.g., exceeded), electronic device 301 may trigger a transition from displaying a representation of content item 310 in a first visual state to displaying a representation of content item 310 in a second visual state different from the first visual state. However, based on determination that the angular motion threshold and any other additional or alternative thresholds have not been met, electronic device 301 does not transition to displaying a representation of content item 310 in the second visual state.

[0069] Figure 3B Examples of electronic devices according to this disclosure are illustrated, which have moved (e.g., rotated) in a physical environment (relative to) Figure 3A (Example) but one or more criteria associated with the representation of the updated content item 310 have not yet been met. Figure 3B As shown, electronic device 301 has been removed from its... Figure 3A The object rotates from its previous position but remains positioned within a physical environment (e.g., an indoor environment) that includes multiple real-world objects. Figure 3B In the example, electronic device 301 has changed its orientation relative to window 312 to align with... Figure 3A The electronic device 301 can be pointed at at different angles (e.g., the electronic device 301 has been moved counterclockwise by +5 polar coordinates relative to the reference ray 321 in Figure 320), thereby changing the field of view of its display provided by one or more of its cameras. Therefore, in some examples, the electronic device 301 may present one or more updated representations of a physical object based on the updated field of view provided by one or more cameras or based on the updated field of view of one or more translucent or transparent displays.

[0070] In some examples, one or more criteria associated with the representation of updated content item 310 may include content type criteria that are met when the content type corresponds to a predetermined content item type (e.g., a movie, a TV program, or a broadcast content item). Figure 3B In one example, movement of electronic device 301 is detected (e.g., device 301 has rotated) and a movement threshold is met, but the content type criterion is not met because the content item represented by the representation of content item 310 is not a predetermined content item type (e.g., the content item associated with the representation of content item 310 is a non-playing or static image). Therefore, the representation of content item 310 may not transition from a first visual state to a second visual state. In some examples, one or more criteria associated with updating the representation of content item 310 may include criteria that are met when the content orientation type corresponds to a predetermined content orientation type (e.g., tilt-lock orientation or head-lock orientation). Generally, even if the movement of electronic device 301 meets the movement criteria for updating the representation of content item 310, the electronic device may not trigger a transition in the display of the content item's representation from a first visual state to a second visual state if one or more other criteria associated with updating the representation of content item 310 are not met.

[0071] Figure 3B-1 Examples of electronic devices according to this disclosure are illustrated, which are compatible with... Figure 3B The difference in the examples lies in their display of a representation of content item 310 that includes visual cues. In some examples, one or more criteria associated with updating the representation of content item 310 may include visual cue criteria that are met when electronic device 301 determines that the associated content includes a predefined visual cue. Visual cues may include identified people, man-made objects (e.g., a specific building or physical structure), or naturally occurring objects (e.g., a geographic location or object). For example, if the content item is a photograph capturing one or a group of faces, and one or more of those faces are identified as predefined visual cues (e.g., object 330 includes a representation of a face identified by facial recognition software as corresponding to a predefined visual cue), then electronic device 301 may determine that visual cues criteria have been met. Figure 3B-1 As shown, the representation of content item 310 may include visual cues (e.g., object 330), and electronic device 301 can recognize object 330 within the representation of content item 310 to correspond to the visual cues. (Note that although object 330 appears to have emphasis) Figure 3B-1 The dashed line represents the boundary of object 330, but for illustrative purposes, this depiction only outlines the importance of object 330 as an identified visual cue within the representation of content item 310; in reality, object 330 may not be displayed in such a defined manner.

[0072] Figure 3B-2 Examples of electronic devices according to this disclosure are illustrated, which are compatible with... Figure 3B The difference in the examples lies in their display of content item 310, which includes a visual cue located within a predefined area of ​​the content item (e.g., one or more border areas and / or corner areas). In some examples, such as Figure 3B-2 As illustrated, meeting the visual cueing standard may also require that the visual cue be located within a predefined area of ​​the content item (e.g., one or more boundary areas and / or corner areas). In a specific example of the representation of content item 310 being presented in the user's field of vision, the boundary area included in the visual cueing standard may constitute the area between a 10-degree and a 15-degree viewing angle from the center of the representation of content item 310 (where viewing angle is a measure of the angular size of an object or scene perceived by the observer's eye). In another specific example, the boundary area included in the visual cueing standard may constitute the area between the outer perimeter of the representation of the content item (e.g., the representation of content item 310) and the perimeter of a concentric rectangle whose area is 90% of the area of ​​the representation of content item 310. In some examples, the corner segments included in the visual cueing standard may include the area within a pre-determined distance (e.g., 1 inch or 2 inches, or one-tenth of a side of the representation of content item 310) from each intersection of two adjacent sides of the outer perimeter of the representation of content item 310, such that the overall corner area included in the visual cueing standard may be defined as the sum of all four corner segments.

[0073] One advantage of predefined area criteria is that updates to content item 310 can be modified if a visual cue (which can be assumed to be important to the user) appears in a predefined area, so that most or all of the representation of the content item remains visible, rather than having potentially important parts cut off, cropped, or otherwise removed from the view (along with the visual cue). (As used herein, cut-off or cropping are used interchangeably to describe preventing or removing one or more corner or boundary areas of content item 310 from being displayed.) For example, one or more corners or boundaries of the representation of content item 310 may remain visible even after the content item is updated. Thus, in some examples, a combination of identified visual cues located in a predefined area may be referred to as anti-cut-off / cropping criteria. Anti-cut-off / cropping criteria can also be advantageous when content item 310 has playback characteristics (e.g., television programs, movies, videos, etc.), where it can be assumed that all parts of content item 310 are potentially important and should not be cut off or cropped. Thus, in some examples, the identification of playback characteristics in content item 310 can form another basis for satisfying anti-cut-off / cropping criteria.

[0074] refer to Figure 3B-1If the representation of content item 310 does not include the identified visual cue, then the visual cue criterion may remain unmet. Similarly, refer to Figure 3B-2 If the identified visual cue within the representation of content item 310 is not located in one or more predefined areas (e.g., object 330 is located near the center of the representation of content item 310, rather than in one or more corner areas), the visual cue criterion may also remain unmet. Generally, electronic device 301 may determine that the representation of content item does not include any identified visual cue (e.g., one or more faces), or may include identified visual cues not located in predefined areas. Therefore, electronic device may not transition the display of the representation of content item 310 from a first visual state to a second visual state.

[0075] Figure 3C Examples of objects that have been moved in a physical environment (e.g., relative to the present disclosure) are illustrated according to some examples of this disclosure. Figure 3B (Example rotation) electronic device, and electronic device 301 has determined that one or more criteria associated with the representation of the updated content item 310 have been met. Figure 3C As shown, although electronic device 301 has been removed from its... Figure 3B The object rotates from its previous position, but remains positioned within a physical environment (e.g., an indoor environment) that includes multiple real-world objects. Figure 3C In the example, electronic device 301 has been removed from Figure 3B The orientation of the electronic device 301 is changed by an angle (e.g., the electronic device 301 has been moved counterclockwise by an additional +5 polar coordinate degrees relative to the reference ray 321 in Figure 320), thereby changing the field of view of its display provided by one or more of its cameras. Therefore, in some examples, the electronic device 301 may present one or more updated representations of a physical object based on an updated field of view provided by one or more cameras or based on an updated field of view of one or more translucent or transparent displays. The electronic device 301 may also present a representation of a content item 310 rotated relative to the orientation of the electronic device 301, such that the content item maintains a fixed orientation relative to the three-dimensional environment.

[0076] In some examples, one or more criteria associated with updating the representation of content item 310 may include size criteria that are met when the initial size of the representation of content item 310 (e.g., upon receiving a startup request, or upon automatic startup) exceeds a minimum portion of the representation of content item 310 (e.g., 50% of the user's field of view). Figure 3CIn the example, movement of electronic device 301 is detected (e.g., device 301 has rotated), and the electronic device has determined that the movement criterion has been met, and also that the size criterion associated with the partial size of the representation of content item 310 has been met, since the representation of content item 310 has an initial size of a minimum portion exceeding half of the user's field of view. If the satisfaction of the movement criterion and the size criterion indicates that all criteria for updating the representation of content item 310 are met, then electronic device 301 may trigger a transition of the representation of content item 310 from a first visual state to a second visual state.

[0077] exist Figure 3C In some examples, because the representation of content item 310 is tilt-locked and maintains a fixed orientation relative to the 3D environment 350, once the device has rotated, the electronic device 301 updates the 3D environment 350, which includes one or more visual representations of one or more virtual objects and one or more content items 310, to fit the updated field of view. In some examples, the first visual state before the update of the representation of content item 310 is associated with the initial size of the representation of content item 310 (e.g., upon receiving a startup request, or when the representation is automatically launched). In some examples, the transition from the representation of content item 310 to the updated second visual state corresponds to scaling the display of the representation of the content item to a second size different from the initial size. In some examples, the second size is smaller than the initial size. For example, in Figure 3C In this example, when electronic device 301 rotates counterclockwise beyond a movement threshold while simultaneously satisfying both movement and size criteria, the representation of content item 310 (which is tilt-locked) can be scaled down in size so that the entire border of the updated representation of content item 310 remains visible in the updated field of view. Even though the content can be scaled in this example, the user can still access the full border of the content item (e.g., if the content is a television program, the user can easily view the perimeter of the representation of content item 310 without any cropping). In other examples, the initial size is smaller than the second size. For example, if Figures 3A to 3C If the movement of the electronic device 301 is reversed, thereby exceeding the movement threshold in the clockwise direction and also satisfying the size criterion, then the representation of the content item can be scaled up in size so that the entire border of the updated representation 310 of the content item remains visible in the updated field of view (e.g., without cropping) and occupies a large portion of the user's field of view.

[0078] Figures 3D to 3E Examples of electronic devices according to this disclosure are illustrated, which have continued to move in physical environment 375 (e.g., relative to...). Figure 3C (Or, as in the example in the aforementioned figures, continue rotating counterclockwise) and it has been determined that one or more criteria associated with the representation of the updated content item 310 have been met. Figures 3D to 3E In the example, although electronic device 301 has been removed from its... Figure 3C The object rotates from its previous position, but remains positioned within a physical environment (e.g., an indoor environment) that includes multiple real-world objects. Figures 3D to 3E In both examples, electronic device 301 has been removed. Figure 3C The orientation of the electronic device 301 is changed by an angle (e.g., the electronic device 301 has been moved counterclockwise by an additional +5 polar coordinate degrees relative to the reference ray 321 in Figure 320), thereby changing the field of view of its display provided by one or more of its cameras. The electronic device 301 may present a representation of the content item 310 rotated relative to the orientation of the electronic device 301, such that the content item maintains a fixed orientation relative to the three-dimensional environment.

[0079] In some examples, when electronic device 301 continues to move after exceeding an initial movement threshold, and while one or more criteria associated with the representation of the updated content item remain satisfied, the representation of content item 310 can continue to transition to an additional visual state. For example, electronic device 301 may experience... Figure 3C The additional counter-clockwise movement in the middle eventually reaches Figure 3D The state depicted in the diagram. Therefore, the representation of content item 310 can continue to transition to a new visual state based on movement. In some examples, additional visual state transitions may occur after the electronic device 301 moves further and one or more additional movement thresholds are sequentially satisfied. For example, Figure 3C and Figure 3D The representation of content item 310 in the image can correspond to the continuously updated visual state. In this example, electronic device 301 receives information from its current state. Figure 3C The orientation in Figure 3D Subsequent angular shifts in orientation can satisfy only an additional shift threshold to trigger the transition of the representation of content item 310 to the updated visual state. However, if from Figure 3C The initial electronic device 301 did not detect enough movement to reach [the target]. Figure 3D If the angle depicted in the diagram triggers an additional motion threshold, the representation of content item 310 may not transition to a new visual state. Alternatively, in other examples, the additional visual state transition of the representation of content item 310 may appear to be substantially continuous, wherein the electronic device 301 detects that multiple smaller motion thresholds are met and result in transitions through multiple visual states.

[0080] In some examples, if the direction of movement is maintained, the angular rotation of electronic device 301 (e.g., angular rotation relative to reference ray 321) may reach a critical angular threshold (e.g., +45 polar degrees relative to the reference ray) that differs from one or more of the aforementioned angular movement thresholds. This critical angular threshold limits any further updates to the representation of content item 310. For example, when electronic device 301 reaches its... Figure 3DAfter the orientation is achieved, electronic device 301 can reach Figure 3E The critical angle threshold in the example is ccw331. In this example, the electronic device 301 can be further rotated counterclockwise to achieve its position... Figure 3E The orientation of the content item 310 is such that, since the critical angle threshold -ccw 331 is already satisfied, any additional rotation will not trigger further scaling down of the representation of the content item 310 (e.g., it will not trigger further updates to the representation of the content item 310). In some examples, any rotation exceeding the critical angle threshold -ccw 331 may reverse previous updates to the representation of the content item 310. A similar constraint on updating the representation of the content item 310 can be implemented using the critical angle threshold -cw 333 for clockwise rotation of the electronic device 301. In other examples, a critical angle threshold can be established when the magnitude of the rotation of the electronic device exceeds a threshold, regardless of whether the rotation is clockwise or counterclockwise.

[0081] In some examples, one or more criteria associated with updating the representation of content item 310 may include criteria that are met when the context of the 3D environment corresponds to a predetermined context (e.g., lack of or reduced movement of electronic device 301, or lack of or reduced user interaction with the user interface for a period longer than a time delay (e.g., 5 seconds, 10 seconds, 15 seconds, etc.), detection of the user sitting, etc.). Although not explicitly shown in the figures, the user may expect to pay attention back to the representation of content item 310 after movement of electronic device 301, and the detection of meeting predetermined context criteria may trigger an update to the representation of content item 310, which facilitates refocusing. User attention to a given representation of content item 310 can be achieved by reducing or stopping user interaction with other content items or reducing or stopping user and / or electronic device movement. In some examples, various sensors in electronic device 301 may detect a reduction (or lack) in user interaction with content items or a reduction (or lack) in electronic device movement and initiate a timer or other elapsed time mechanism. When a threshold time is met, the representation of content item 310 may be updated. For example, electronic device 301 may update and restore the representation of content item 310 to its initial (e.g., larger) size, or enlarge the border of the representation of content item 310 to make the representation of content item 310 conform to the widest possible aspect ratio with maximum visibility.

[0082] The aforementioned examples of electronic devices primarily focus on rolling movement and electronic devices relative to each other. Figure 3A The corresponding behaviors of the examples. Several non-limiting examples relating to the movement of the electronic device in the pitch and / or yaw directions and the corresponding behaviors of the electronic device will now be discussed. Figure 3F Examples of electronic devices according to this disclosure are illustrated, which have been moved (e.g., tilted) in a physical environment (relative to) Figure 3A(Example) but one or more criteria associated with the representation of the updated content item 310 have not yet been met. Figure 3F As shown, electronic device 301 has been removed from its... Figure 3A The previous position in the image is tilted, but it remains positioned within a physical environment (e.g., an indoor environment) that includes multiple real-world objects. Figure 3F In the example, electronic device 301 has been tilted in the pitch direction as described above (e.g., electronic device 301 has been moved clockwise by +5 polar coordinate degrees relative to reference ray 361 in figure 360), thereby changing the field of view of its display provided by one or more of its cameras, and thus presenting one or more updated representations of the physical object based on the updated field of view provided by one or more cameras or based on the updated field of view of one or more translucent or transparent displays.

[0083] In some examples, when no user input is detected for moving or exiting the representation of content item 310 in the displayed 3D environment 350, the electronic device 301 may continue to present the representation of content item 310 indefinitely at a predetermined and / or user-defined location in the physical environment 375. In these examples, movement of the electronic device may cause an update to the representation of content item 310; however, the presentation of the representation of content item 310 may occur as long as the predetermined and / or user-defined location in the physical environment 375 remains visible in the field of view of one or more cameras of the electronic device 301. In some examples, movement of the electronic device may not necessarily cause an update to content item 310. For example, as... Figure 3F As shown, the electronic device is tilted in the pitch direction, but the representation of content item 310 remains in the same position relative to the physical environment 375 and with respect to the surroundings. Figure 3A Presented with the same size and shape as shown.

[0084] In some examples, the representation of content item 310 is head-locked, tilt-locked, and / or horizontally locked, as defined above, and optionally has elasticity. For example, when the representation of content item 310 is head-locked and elastic, the electronic device 101 optionally makes the representation of content item 310 visually represent head-locked content according to an elasticity model. In some examples, the elasticity model physically implements the user's interaction in the three-dimensional environment 375 such that the interaction is governed by physical laws, such as laws related to springs. For example, the user's head position and / or head orientation optionally corresponds to the position of a first end of a spring (e.g., simulating the first end of a spring attached to an object), and the representation of content item 310 optionally corresponds to the mass of a second end attached to the spring that is different from the first end of the spring (e.g., opposite to the first end). When the head position and / or orientation is a first head position and / or a first orientation corresponding to a first position of a first end of the spring and the representation of content item 310 corresponds to a mass attached to a second end of the spring, the electronic device optionally detects head movement (e.g., head rotation) from the first head position and / or the first head orientation to a second head position and / or the second head orientation. In response to detecting head rotation, the electronic device optionally models the deformation of the spring (e.g., based on the amount and / or the speed of head rotation) and moves the representation of content item 310 according to the energy release caused by the movement of the spring relative to the second head position and / or the second head orientation toward an equilibrium position (e.g., a stable equilibrium position). The speed at which the representation of content item 310 follows head rotation is optionally a function of the distance between the position of the representation of content item 310 when the electronic device detects head rotation and the position of the representation of content item 310 corresponding to a relaxed position of the spring (e.g., an equilibrium position), which is optionally the same position relative to the user's new viewpoint caused by head rotation as the position of the representation of content item 310 relative to the user's viewpoint before head rotation was detected. In some examples, the speed of the representation of content item 310 decreases as it moves toward a relaxed position in response to head rotation. In some examples, the user's head rotates a first amount within a first time interval, and the movement of the representation of content item 310 to its new position relative to the user's new viewpoint is performed within a second time interval greater than the first time interval. Thus, when the representation of content item 310 is head-locked and has elasticity 322, based on detected head movement, the electronic device 101 optionally displays first virtual content, such as reference content, that moves within the three-dimensional environment according to the user's head movement and according to an elastic model that mimics lazy following movement behavior. Figures 3F to 3K As shown and described.

[0085] Applying elastic behavior to horizontally locked or head-locked content as the user moves (e.g., rotates their head) is useful for smoothing the movement of initial virtual content within a 3D environment. This smoothing improves the user experience by reducing motion sickness or dizziness caused by inelastic behavior. Alternatively or additionally, a time delay can be used instead of an elastic model.

[0086] like Figure 3F As shown, for example, when the electronic device 301 is tilted in the pitch direction, the representation of the content item 310 may appear to be closer to the top side of the display (e.g., initially appearing to be fixed in place relative to the three-dimensional environment 375 for a short period of time before being updated according to the elastic model).

[0087] Figures 3G to 3H Examples of electronic devices according to this disclosure are illustrated, which have continued to move (e.g., tilt) (e.g., relative to) physical environment 375. Figure 3F (Or, as in the example in the aforementioned figures, continue to tilt in the pitch direction) and it has been determined that one or more criteria associated with the representation of the updated content item 310 have been met. Figures 3G to 3H In the example, although electronic device 301 has been removed from its... Figure 3F The object rotates from its previous position, but remains positioned within a physical environment (e.g., an indoor environment) that includes multiple real-world objects. Figures 3G to 3H In the example, electronic device 301 has been removed from Figure 3F The angle in the image changes its orientation (e.g., the electronic device 301 has been moved clockwise by an additional +10 polar degrees relative to the reference ray 361 in figure 360), thereby changing the field of view of its display provided by one or more of its cameras. The electronic device 301 may present a representation of the content item 310, which can be considered as initially sliding relative to the movement of the electronic device 301 in the pitch direction until reaching the boundary of the user's viewpoint, wherein the representation of the content item 310 is updated to be displayed with different border sizes and / or moved within the three-dimensional environment 375 to remain displayed within the user's viewpoint.

[0088] In some examples, the representation of content item 310 may undergo one or more updates to its visual state (e.g., resizing). In some examples, updates to the visual state of the visual representation of content item 310 may include any updates to its orientation and placement relative to the 3D environment 375, or any other updates without changing its orientation and placement relative to the 3D environment 375. In some examples, the full border of the visual representation of content item 310 remains displayed within the user's viewpoint unless the user provides input to change the locking behavior of the representation of content item 310.

[0089] In some examples, when one or more criteria become satisfied to cause an update to the representation of content item 310, an effective update to the representation of content item 310 may occur only after a time delay and / or with an inertial follow-up elasticity. This may result in the representation of content item 310 being displayed in an intermediate visual state (e.g., between a first visual state and a second visual state), where a portion of the representation of content item 310 is clipped by a partial size depending on the degree of movement in the pitch direction. In some examples, the electronic device 301 may transition from displaying the representation of content item 310 in a first visual state to displaying the representation of content item 310 in a second visual state different from the first visual state, and all corners (e.g., or full borders) of the representation of content item 310 in the second visual state remain visible within the boundaries of the depicted three-dimensional environment 350 (e.g., or within the user's viewpoint). Figure 3G (Shrinking). In some examples, such as Figure 3G As shown, the second visual state represented by content item 310 is scaled up or down to occupy the widest aspect ratio on the display without changing its center and without any clipping. In some examples, when implementing lazily following elasticity, the edges of the display act as hard edges that do not allow content to move off-screen. In other words, given that elastic content behavior may allow part of the content to be off-screen, but when the content would otherwise be off-screen according to the elastic model, hard edge behavior will lock that content to the edge of the display. Additionally, it should be understood that although a specific edge is shown in this example, hard edge techniques may be applied to one or more edges depending on the direction of motion (e.g., the top and left edges could be hard edges for rapid pitch / yaw movements to the lower right).

[0090] In some examples, the representation of content item 310 and its updated version are head-locked, tilt-locked, and / or horizontally locked, as defined above, exhibiting elasticity. In some examples, after detecting a first instance of movement (e.g., continuous movement), electronic device 301 also detects the end of movement (e.g., continuous lack of movement or movement below a threshold), and the updated representation of content item 310 moves toward a balanced position relative to the user's updated viewpoint after exceeding a time threshold for the most recent inactivity event. In other words, the spring-like behavior of the elastic model returns to a relaxed state. In some examples, optionally in response to exceeding a time threshold after continuous lack of movement, the updated representation of content item 310 may undergo a further transition to a new visual state, which may include realigning with a new orientation of electronic device 301 (e.g., after any previous movement of the electronic device) and / or occupying a larger portion of the field of view or returning to its initial size from the first visual state. For example, as... Figure 3HAs shown, after a series of events including the continuous tilting of the electronic device 301 in the pitch direction and correspondingly transforming the representation of the content item 310 into a new visual state, and subsequently after a time delay exceeding the duration of continuous lack of movement after the transformation, the representation of the content item reverses its scale back to its initial size and / or can be re-aligned or reoriented within the user's updated viewpoint.

[0091] The elasticity model described herein can be a function of time and distance. For example, a relatively increased rotational movement speed increases the likelihood of potential clipping (without the techniques described herein). For instance, for a first rotational movement within a first time period, elasticity may cause content to be clipped (e.g., partially off-screen), while for a second rotational movement greater than the first rotational movement within a second time period longer than the first time period, elasticity may not cause clipping. For ease of illustration, Figures 3G to 3H The angular movement is referenced to a threshold angle; however, it should be understood that the illustrated shearing and / or resizing behavior may occur alternatively when there is movement at a velocity relatively greater than the threshold (e.g., the threshold angle shown relative to the time period in which shearing might occur). Figures 3G to 3H As shown in Figure 360, in some examples, reference ray 361 (to which the movement threshold is measured) corresponds to a ray that is perpendicular to both gravity and the horizontal line of the physical environment in the user's field of view. Reference ray 361 corresponds to a ray that points approximately to the horizon and is parallel to the y-axis 369 of electronic device 301. Therefore, reference ray 361 is independent of the orientation of electronic device 301 in the three-dimensional environment 350. In some examples, reference ray 361 (to which the movement threshold is measured) is established based on the calibration of electronic device 301. In some examples, if electronic device 301 detects a sufficient change in the angle between its y-axis 369 and reference ray 361 (e.g., illustrated in Figure 360) (e.g., greater than 3 degrees, 5 degrees, 8 degrees, 10 degrees, etc.), the angular movement of electronic device 301 may exceed a clockwise angular movement threshold (threshold-cw) 367 or a counterclockwise angular movement threshold (threshold-ccw) 365 (e.g., thresholds 365 and 367 are not perpendicular to gravity, and they typically correspond to directions below and above the electronic device, respectively). Exceeding the angular movement threshold in either pitch direction (e.g., clockwise or counterclockwise relative to reference ray 361) may trigger a transition from displaying a representation of content item 310 in a first visual state to displaying a representation of content item 310 in a second visual state different from the first visual state.

[0092] Figures 3I to 3K Examples of electronic devices according to this disclosure are illustrated, which have been moved sequentially (e.g., tilted) (e.g., relative to) a yaw direction in a physical environment 375. Figure 3AOr, as in the example in the aforementioned figures, tilting in the yaw direction. Figures 3I to 3K The examples depicted in the series and Figures 3F to 3H The difference in the example depicted is that the device movement is around a yaw direction, which is typically on the right side of the electronic device. For simplicity, the relevant example features and alternatives discussed in the foregoing figures can be applied to... Figures 3I to 3K Examples are depicted in the text. For instance, yaw movement of electronic device 301 is detected (e.g., user movement), and the user's viewpoint shifts from... Figure 3A The instance in the middle is changed to Figure 3I The viewpoint in the text. The transformation of the representation of content item 310 in... Figure 3J The scaling is shown to avoid clipping when a portion of content item 310 is off-screen due to the elasticity of lazy follow behavior. (See example...) Figure 3K As shown, the scaling of the representation of content item 310 is reversed and / or realigned with the new orientation of the electronic device. For example, after a time threshold of no movement (or less than a threshold movement amount or less than a threshold velocity amount to avoid shearing due to lazy following behavior), the size of the representation of content item 310 is adjusted to... Figure 3A The full border dimensions are shown.

[0093] The elasticity model described herein can be a function of time and distance. For example, a relatively increased rotational movement speed increases the likelihood of potential clipping (without the techniques described herein). For instance, for a first rotational movement within a first time period, elasticity may cause content to be clipped (e.g., partially off-screen), while for a second rotational movement greater than the first rotational movement within a second time period longer than the first time period, elasticity may not cause clipping. For ease of illustration, Figures 3I to 3K The angular movement is referenced to a threshold angle; however, it should be understood that the illustrated shearing and / or resizing behavior may occur alternatively when there is movement at a velocity relatively greater than the threshold (e.g., the threshold angle shown relative to the time period in which shearing might occur). Figures 3I to 3KAs shown in Figure 380, in some examples, reference ray 381 (to which the movement threshold is measured) corresponds to a ray that is perpendicular to both gravity and the horizontal line of the physical environment in the user's field of view. Reference ray 381 corresponds to a ray that points approximately to the horizon and is parallel to the y-axis 369 of electronic device 301. Therefore, reference ray 381 is independent of the orientation of electronic device 301 in the three-dimensional environment 350. In some examples, reference ray 381 (to which the movement threshold is measured) is established based on the calibration of electronic device 301. In some examples, if electronic device 301 detects a sufficient change in the angle between its y-axis 369 and reference ray 381 (e.g., illustrated in Figure 380) (e.g., greater than 3 degrees, 5 degrees, 8 degrees, 10 degrees, etc.), the angular movement of electronic device 301 may exceed a clockwise angular movement threshold (threshold-cw) 387 or a counterclockwise angular movement threshold (threshold-ccw) 385 (e.g., all thresholds including thresholds 385 and 387 are perpendicular to gravity, and they typically correspond to the left and right directions of electronic device 301, respectively). Exceeding the angular movement threshold in either yaw direction (e.g., clockwise or counterclockwise relative to reference ray 381) may trigger a transition from displaying a representation of content item 310 in a first visual state to displaying a representation of content item 310 in a second visual state different from the first visual state.

[0094] In some examples, one or more criteria associated with updating the representation of content item 310 may include a rate of change criterion that is satisfied when the rate of change (e.g., velocity, speed, and / or acceleration) of movement (e.g., angular movement) of electronic device 301 exceeds a threshold rate of change (e.g., 0.3 px / ms, 0.5 px / ms, or 1 px / ms), optionally as a supplement to or alternative to criteria based on the movement threshold of electronic device 301. For example, electronic device 301 calculates the rate of change of movement and determines that a rate of change criterion has been met (e.g., as a supplement to or alternative to satisfying movement criteria). When the criteria (including the rate of change criterion) for updating the representation of content item 310 are met, electronic device 301 may trigger a transition of the representation of content item 310 from a first visual state to a second visual state. For example, yaw movement of electronic device 301 (e.g., or user movement) via... Figure 3A and Figure 3IThis can be illustrated by changes in the user's viewpoint. When a rate of change criterion is met (e.g., optionally for a duration of a threshold time period), the representation of content item 310 transitions to a new visual state. In some examples, the transition of the representation of content item 310 may be proportional to the value of the rate of change of movement of electronic device 301 (e.g., the representation of content item 310 shrinks further to provide more padding for the aforementioned elastic model or perceived lazy following). Alternatively, regardless of the value of the rate of change of movement of electronic device 301, the new visual state of the representation of the content item is predetermined and / or defined by the user based on one or more visual characteristics.

[0095] Figures 5A to 5C Examples of electronic devices according to this disclosure are shown, which have been moved (e.g., tilted) in sequence about a yaw direction in a physical environment 575 (e.g., tilted in the yaw direction). Figures 5A to 5C The examples depicted in the series and Figure 3A and Figures 3I to 3K The difference in the examples depicted is that the representation of content items is typically rendered without any cropping, trimming, and / or scaling. In some examples, the representation of content items tracks the movement of electronic device 301. Optionally, the representation of content items tracks the movement of electronic device after a time delay, while maintaining its rendering within the boundaries of the depicted environment 550. For the sake of brevity, the relevant example features and alternatives discussed in the foregoing figures can be applied to... Figures 5A to 5C The example depicted in the text.

[0096] Figure 4A An electronic device for displaying a representation of content item 410 according to some examples of this disclosure is illustrated. This electronic device meets one or more criteria for updating the representation of the content item, but does not include predetermined visual cues in corner or boundary areas, and / or the associated content item does not have playback features (e.g., a television program or movie). In some examples, the lack of predetermined visual cues and / or the lack of playback features in corner or boundary areas means that anti-cutting / cropping criteria have not yet been met, and therefore there is no restriction on cutting or cropping content item 410. In these cases, according to some examples of this disclosure, updating or adjusting the representation of content item 410 from a first visual state to a second visual state can be performed by cutting one or more corner areas of the representation of content item 410, such as... Figure 4A As shown. In some examples, electronic device 301 may determine the amount of shearing of one or more corner regions (e.g., or one or more factors associated therewith) based on the magnitude of roll movement (or pitch movement or yaw movement). For the sake of brevity, the example criteria or example features and alternatives discussed in the foregoing figures may be applied... Figures 4A to 4B The example depicted in the text.

[0097] In some examples, the cut may be limited to corner areas as defined above. In some examples, the resulting border of the content item 310 after the cut boundary area may be hexagonal, rectangular, or square. The cut representation of content item 410 may advantageously preserve the size of the content item (e.g., magnification) at the cost of sacrificing the cut portion of the content item. However, since the cut is performed only if anti-cutting / cropping criteria are not yet met, visual cues located in corners or boundary areas should not be hidden.

[0098] In some examples, it may not be desirable to clip one or more corner regions of the representation of content item 410, and maintaining the approximate shape of the representation of content item 410 (e.g., a rectangular shape) may be preferred. In some examples, according to some examples of this disclosure, the representation of content item 410 can be clipped in a way that allows for... Figure 4B One or more boundary regions outside the shown area 430 are cropped to update or adjust the representation of content item 410 from a first visual state to a second visual state. In some examples, the aspect ratio of the representation of content item 410 is maintained during and / or after the transition (e.g., unless the user provides input to change the aspect ratio). In some examples, the new visual state of the representation of content item 410 may include only the visual content within area 430. In some examples, electronic device 301 may determine the amount by which one or more boundary regions (e.g., or one or more factors associated therewith) are cropped based on the magnitude of the movement (e.g., or determine the size and placement of area 430 such that the visual content of the representation of the content outside area 430 is cropped).

[0099] In some examples, cropping or trimming may occur for larger areas of the content item representation (such as boundary areas as defined above). In some examples, the resulting border of the content item 310 representation after cropping or trimming the boundary area may be limited to a square or rectangular shape. In some examples, the cropped or trimmed corner areas have equal areas. Alternatively, in other examples, the cropped or trimmed corner areas of the content item representation may not have equal areas.

[0100] For example, in response to the movement of electronic device 401 exceeding an angular movement threshold and determining that the anti-cropping / trimming criterion has not been met, electronic device 401 may switch from a first visual state in which the representation of displayed content item 410 is uncropped or cropped to a second visual state that is cropped or trimmed, different from the first visual state. In some examples, the initial size (area) of the representation of content item 410 in the first visual state is larger than the size (area) of the representation of content item 410 in the cropped or trimmed second visual state. Figures 4A to 4BAs shown in the example, the tilt-locked representation of content item 410 can be clipped or cropped so that a smaller portion of the border of the updated representation of content item 410 remains visible in the updated field of view. Figures 4A to 4B In the example, if the movement of the electronic device 401 is reversed such that one or more corner movement thresholds are no longer met, the electronic device may undo the clipping or cropping in one or more corner regions of the representation of the content item 410, so that the entire border of the updated representation of the content item 310 becomes visible again in the updated field of view and occupies a large portion of the user's field of view.

[0101] It should be understood that the examples shown and described herein are merely exemplary and additional and / or alternative elements may be provided in a three-dimensional environment.

[0102] Figure 6 This is a flowchart illustrating example processes according to some examples of this disclosure, which are used to display content and automatically update content such as the representation of content items based on detected movement of an electronic device and according to the satisfaction of one or more criteria. In some examples, process 600 begins with an electronic device communicating with one or more displays, one or more input devices, and optionally one or more cameras and / or one or more accelerometers (e.g., for performing roll, pitch, or yaw movements as described herein). In some examples, the electronic device is optionally a head-mounted display similar to or corresponding to the electronic device 201 of FIG. 2. Figure 6 As shown, in some examples, at 602, when the electronic device displays a representation of a content item in a first visual state within the field of view of one or more cameras via one or more displays, the electronic device detects movement of the electronic device via one or more input devices. For example, as Figure 3B As illustrated, when electronic device 301 displays a representation of content item 310 in a first visual state within the field of view of one or more cameras, electronic device 301 detects movement of electronic device 301.

[0103] In some examples, at 604, in response to the electronic device detecting movement, at 606, based on determining that one or more criteria are met, the electronic device automatically changes the representation of the content item from a first visual state to a second visual state different from the first visual state. For example, as referenced... Figure 3C In response to the detection of movement of electronic device 301 (e.g., relative to reference ray 321), based on the determination of one or more criteria (e.g., exceeding the movement threshold of threshold -ccw325), electronic device 301 moves from its first visual state (e.g., ... Figure 3A The described visual state automatically transitions to a second visual state, different from the first visual state (e.g., Figure 3C (As depicted).

[0104] It should be understood that process 600 is an example, and more, fewer, or different operations may be performed in the same or different order. Additionally, the operations in process 600 described above may optionally be implemented by running one or more functional modules in an information processing device such as a general-purpose processor (e.g., as described relative to FIG. 2) or a dedicated chip and / or by other components of FIG. 2.

[0105] Therefore, based on the foregoing, some examples of this disclosure relate to a method comprising: at an electronic device communicating with one or more cameras and one or more input devices, detecting movement of the electronic device via the one or more input devices while a representation of a content item is displayed via the one or more displays in a first visual state; in response to detecting the movement of the electronic device, transitioning the representation of the content item from the first visual state to a second visual state different from the first visual state, based on determining that one or more criteria are met, including a criterion satisfied when the electronic device detects that the movement of the electronic device is greater than a movement threshold; detecting cessation of the movement of the electronic device via the one or more input devices while the representation of the content item is in the second visual state; and in response to detecting cessation of the movement of the electronic device, transitioning the representation of the content item from the second visual state to a third visual state different from the second visual state. As a supplement or alternative to one or more of the examples above, in some examples, transitioning the representation of the content item from the first visual state to the second visual state includes updating the representation of the content item from being displayed at a first size to being displayed at a second size different from the first size. As a supplement or alternative to one or more of the above examples, in some examples, the second size is smaller than the first size. As a supplement or alternative to one or more of the above examples, in some examples, updating the representation of the content item from being displayed at the first size to being displayed at the second size includes scaling the representation of the content item from the first size to the second size. As a supplement or alternative to one or more of the above examples, in some examples, changing the representation of the content item from the first visual state to the second visual state includes cropping the representation of the content item. As a supplement or alternative to one or more of the above examples, in some examples, the motion threshold is an angular motion threshold satisfied when the electronic device detects that the movement of the electronic device exceeds a predetermined angular rotation. As a supplement or alternative to one or more of the above examples, in some examples, the motion threshold is an angular motion velocity threshold satisfied when the electronic device detects that the movement speed or rate of the electronic device exceeds a predetermined angular velocity or predetermined angular rate. As a supplement or alternative to one or more of the examples above, in some examples, the movement threshold is an angular movement acceleration threshold that is satisfied when the electronic device detects that the movement acceleration of the electronic device exceeds a predetermined angular acceleration. As a supplement or alternative to one or more of the examples above, in some examples, the one or more criteria also include a criterion that is satisfied when the representation of the content item is a predetermined content type.As a supplement or alternative to one or more of the examples above, in some examples, the one or more criteria also include criteria satisfied when the representation of the content item includes a predefined visual cue. As a supplement or alternative to one or more of the examples above, in some examples, the one or more criteria also include anti-clipping / trimming criteria satisfied when the predefined visual cue is located in a predefined area of ​​the content item. As a supplement or alternative to one or more of the examples above, in some examples, based on determining that the one or more criteria are satisfied (including determining that the anti-clipping / trimming criteria are satisfied), the representation of the content item transitions from the first visual state to the second visual state while maintaining the display of one or more corners and boundaries of the content item. As a supplement or alternative to one or more of the examples above, in some examples, the one or more criteria also include criteria satisfied when the representation of the content item is greater than a size threshold. As a supplement or alternative to one or more of the examples above, in some examples, based on determining that one or more critical angle movement thresholds of the electronic device have been satisfied, the transition of the representation of the content item from the first visual state to the second visual state is stopped. As a supplement or alternative to one or more of the above examples, in some examples, detecting the cessation of the movement of the electronic device via the one or more input devices includes detecting that the movement of the electronic device is less than a second threshold for a predetermined period of time. As a supplement or alternative to one or more of the above examples, in some examples, the method further includes: when the representation of the content item is in the second visual state, detecting a threshold period of no user interaction with the displayed user interface via the one or more input devices, and transitioning the representation of the content item from the second visual state to the first visual state. As a supplement or alternative to one or more of the above examples, in some examples, the method further includes: in response to detecting the movement of the electronic device, abandoning the transition of the representation of the content item from the first visual state to the second visual state based on determining that the one or more criteria are not met.

[0106] Some examples of this disclosure relate to an electronic device comprising: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described above.

[0107] Some examples of this disclosure relate to a non-transitory computer-readable storage medium that stores one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the methods described above.

[0108] Some examples of this disclosure relate to an electronic device that includes one or more processors, a memory, and components for performing any of the methods described above.

[0109] Some examples of this disclosure relate to an information processing apparatus used in an electronic device, the information processing apparatus including components for performing any of the methods described above.

[0110] This disclosure anticipates that, in some examples, the data utilized may include personal information that uniquely identifies or can be used to contact or locate specific individuals. Such personal information may include demographic data, content consumption activity, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records relating to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information. Specifically, as described herein, one aspect of this disclosure is the tracking of users' biometric data.

[0111] This disclosure recognizes that the use of such personal information data in the techniques of this invention can be used to benefit users. For example, personal information data can be used to display suggested text that changes based on changes in a user's biometric data. For example, suggested text can be updated based on changes in a user's age, height, weight, and / or medical history.

[0112] This disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with robust privacy policies and / or privacy measures. Specifically, such entities should implement and maintain privacy policies and measures that are recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable entity purposes and should not be shared or sold outside of these legitimate purposes. Furthermore, such collection / sharing should be conducted only after receiving informed consent from users. Additionally, such entities should consider taking any necessary steps to protect and safeguard the right to access such personal information data and ensure that other entities with access to personal information data comply with the privacy policies and procedures of other entities. Furthermore, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and privacy measures. Additionally, policies and measures should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including considerations of specific jurisdictions. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.

[0113] Regardless of the foregoing, this disclosure also envisions examples of users selectively blocking the use or access to personal information data. That is, this disclosure contemplates providing hardware and / or software components to prevent or block access to such personal information data. For example, the inventive technology can be configured to allow a user to opt-in or opt-out during or at any time after registering for the service. In another example, a user can choose not to enable the recording of personal information data in a specific application (e.g., a first application and / or a second application). In addition to providing "opt-in" and "opt-out" options, this disclosure also contemplates providing notifications related to access to or use of personal information. For example, a user can be notified that their personal information data will be accessed after collection is initiated, and then reminded again just before one or more devices access the personal information data.

[0114] Furthermore, the intent of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by restricting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0115] For purposes of explanation, the foregoing description has been described by reference to specific examples. However, the illustrative description above is not intended to be exhaustive or to limit this disclosure to the precise form disclosed. Many modifications and variations are possible based on the teachings above. The examples were chosen and described in order to best elucidate the principles of this disclosure and its practical application, thereby enabling others skilled in the art to best utilize this disclosure with various modifications suitable for the particular intended use, as well as the various described examples.

Claims

1. A method comprising: At an electronic device that communicates with one or more cameras and one or more input devices: When a representation of a content item is displayed in a first visual state via one or more displays, movement of the electronic device is detected via the one or more input devices; In response to detecting the movement of the electronic device: Based on determining that one or more criteria are met, including criteria that are met when the electronic device detects that the movement of the electronic device is greater than a movement threshold, the representation of the content item is changed from the first visual state to a second visual state different from the first visual state; When the representation of the content item is in a second visual state, the cessation of the movement of the electronic device is detected via the one or more input devices; as well as In response to the detection of the cessation of the movement of the electronic device: The representation of the content item is changed from the second visual state to a third visual state that is different from the second visual state.

2. The method of claim 1, wherein changing the representation of the content item from the first visual state to the second visual state comprises updating the representation of the content item from being displayed at a first size to being displayed at a second size different from the first size.

3. The method according to claim 2, wherein the second dimension is smaller than the first dimension.

4. The method of claim 2, wherein updating the representation of the content item from being displayed at a first size to being displayed at a second size includes scaling the representation of the content item from the first size to the second size.

5. The method of claim 1, wherein changing the representation of the content item from the first visual state to the second visual state includes cropping the representation of the content item.

6. The method of claim 1, wherein the movement threshold is an angular movement threshold satisfied when the electronic device detects that the movement of the electronic device exceeds a predetermined angular rotation.

7. The method of claim 6, wherein the movement threshold is an angular movement speed threshold that is satisfied when the electronic device detects that the movement speed or rate of the electronic device exceeds a predetermined angular velocity or a predetermined angular rate.

8. The method of claim 7, wherein the movement threshold is an angular movement acceleration threshold that is satisfied when the electronic device detects that the movement acceleration of the electronic device exceeds a predetermined angular acceleration.

9. The method of claim 1, wherein the one or more criteria further include criteria satisfied when the representation of the content item is content of a predetermined type.

10. The method of claim 1, wherein the one or more criteria further include criteria satisfied when the representation of the content item includes a predefined visual cue.

11. The method of claim 10, wherein the one or more criteria further include anti-clipping / trimming criteria satisfied when the predefined visual cue is located in a predefined area of ​​the content item.

12. The method of claim 11, wherein determining that one or more criteria are met includes determining that the anti-clipping / trimming criteria are met, and changing the representation of the content item from the first visual state to the second visual state to maintain the display of one or more corners and boundaries of the content item.

13. The method of claim 1, wherein the one or more criteria further include a criterion satisfied when the representation of the content item is greater than a size threshold.

14. The method of claim 1, wherein the transition of the representation of the content item from the first visual state to the second visual state is stopped based on determining that one or more critical angle movement thresholds of the electronic device have been met.

15. The method of claim 1, wherein detecting the cessation of the movement of the electronic device via the one or more input devices includes detecting a predetermined time period during which the movement of the electronic device is less than a second threshold.

16. The method according to claim 1, further comprising: When the representation of the content item is in the second visual state, the representation of the content item is changed from the second visual state to the first visual state when a threshold period of no user interaction with the displayed user interface is detected by the one or more input devices.

17. The method according to claim 1, further comprising: In response to detecting the movement of the electronic device: If it is determined that one or more of the criteria are not met, the transition of the representation of the content item from the first visual state to the second visual state is abandoned.

18. An electronic device comprising: One or more processors; Memory; and One or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method according to any one of claims 1 to 17.

19. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 17.

20. An electronic device, comprising: One or more processors; Memory; and Components for performing the method according to any one of claims 1 to 17.

Images