System and method for motion-based user interface

Abstract

The invention relates to a motion-based user interface system and method. Some examples of the present disclosure relate to systems and methods for moving a virtual object in a three-dimensional environment according to detected movement of an electronic device. In some examples, the electronic device detects movement according to a first movement pattern or a second movement pattern described more detailed herein. In some examples, in response to detecting the first movement pattern, the electronic device applies a first correction factor to movement of a virtual object in the environment. In some examples, in response to detecting the first movement pattern, the electronic device applies a first correction factor to movement of a virtual object in the environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 583,571, filed on September 18, 2023, and U.S. Patent Application No. 18 / 824,823, filed on September 4, 2024, the contents of both applications are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to systems and methods for displaying user interface elements in a three-dimensional environment, and more particularly to adjusting the motion of user interface elements in a three-dimensional environment. Background Art

[0004] Some computer graphics environments provide two-dimensional environments and / or three-dimensional environments in which at least some of the objects presented for viewing by a user are virtual and generated by a computer. In some examples, some of the objects presented in the computer graphics environment are representations of real objects in the physical environment of the computer. In some examples, in response to detecting movement of the computer, the electronic device moves representations of real objects in the physical environment of the computer to simulate movement in the computer graphics environment based on movement in the physical environment. Summary of the invention

[0005] Some examples of the present disclosure relate to systems and methods for modifying the movement of virtual objects displayed in a three-dimensional environment according to a movement mode of an electronic device. In some examples, in response to detecting the movement of an electronic device, the electronic device updates the positioning of one or more elements in the three-dimensional environment to maintain the display of these elements when the user's viewpoint moves in the three-dimensional environment. For example, the electronic device simulates a virtual object following the user. In some examples, when the user walks, jogs, runs and / or sprints while using the electronic device, in addition to moving in the direction of travel of the user, the electronic device may also move vertically and / or horizontally. In some examples, in response to detecting a movement mode including up and down movement and / or left and right movement, the electronic device may adjust the movement of the virtual object using a correction factor corresponding to the detected movement mode. For example, in response to detecting the up and down movement characteristics of a first movement mode (e.g., walking), the electronic device applies a first correction factor to the movement of the virtual object to reduce the virtual object from appearing in the form of bouncing when the electronic device moves. As another example, in response to detecting the up and down movement and / or left and right movement characteristics of a second movement mode (e.g., running), the electronic device applies a second correction factor to the movement of the virtual object to reduce the virtual object from appearing in the form of bouncing when the electronic device moves.

[0006] A comprehensive description of these examples is provided in the drawings and detailed description, and it should be understood that this summary does not limit the scope of the present disclosure in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] For a better understanding of the various examples described herein, reference should be made to the following detailed description and the following drawings. Throughout the drawings, like reference numerals generally refer to corresponding parts.

[0008] Figure 1 An electronic device presenting an extended reality environment according to some examples of the present disclosure is shown.

[0009] Figure 2 A block diagram illustrating an example architecture of a device according to some examples of the present disclosure.

[0010] FIG. 3A to FIG. 3C An example is illustrated in which an electronic device according to some examples of the present disclosure updates the positioning of a virtual object in a three-dimensional environment according to the movement of the electronic device.

[0011] FIG. 4A to FIG. 4B An example in which an electronic device applies a first correction factor to a movement of a virtual object when the electronic device moves in a first movement mode according to an example of the present disclosure is illustrated.

[0012] FIG. 5A to FIG. 5B An example in which the electronic device applies a first correction factor to a movement of a virtual object when the electronic device moves in a second movement mode according to an example of the present disclosure is illustrated.

[0013] FIG. 6A to FIG. 6B An example of an electronic device modifying movement of a virtual object while the electronic device is moving according to the location of a gaze of a user of the electronic device according to some examples of the present disclosure is illustrated.

[0014] 7A to 7C An example is illustrated in which an electronic device according to some examples of the present disclosure modifies the content of a virtual object according to a movement pattern of the electronic device.

[0015] FIG. 8A to FIG. 8C An example is illustrated in which an electronic device according to some examples of the present disclosure modifies a size of a virtual object depending on a movement mode of the electronic device.

[0016] Fig. 9 An example method of modifying the movement of a virtual object displayed in a three-dimensional environment according to a movement mode of an electronic device according to some examples of the present disclosure is illustrated. DETAILED DESCRIPTION

[0017] Some examples of the present disclosure relate to systems and methods for modifying the movement of virtual objects displayed in a three-dimensional environment according to a movement mode of an electronic device. In some examples, in response to detecting the movement of an electronic device, the electronic device updates the positioning of one or more elements in the three-dimensional environment to maintain the display of these elements when the user's viewpoint moves in the three-dimensional environment. For example, the electronic device simulates a virtual object following the user. In some examples, when the user walks, jogs, runs and / or sprints while using the electronic device, in addition to moving in the direction of travel of the user, the electronic device may also move vertically and / or horizontally. In some examples, in response to detecting a movement mode including up and down movement and / or left and right movement, the electronic device may adjust the movement of the virtual object using a correction factor corresponding to the detected movement mode. For example, in response to detecting the up and down movement characteristics of a first movement mode (e.g., walking), the electronic device applies a first correction factor to the movement of the virtual object to reduce the virtual object from appearing in the form of bouncing when the electronic device moves. As another example, in response to detecting the up and down movement and / or left and right movement characteristics of a second movement mode (e.g., running), the electronic device applies a second correction factor to the movement of the virtual object to reduce the virtual object from appearing in the form of bouncing when the electronic device moves.

[0018] In some examples, a three-dimensional object is displayed in a computer-generated three-dimensional environment at a particular orientation that controls one or more behaviors of the three-dimensional object (e.g., when the three-dimensional object moves within the three-dimensional environment). In some examples, the orientation in which the three-dimensional object is displayed in the three-dimensional environment is selected by a user of the electronic device or automatically selected by the electronic device. For example, when initiating presentation of the three-dimensional object in the three-dimensional environment, the user may select a particular orientation of the three-dimensional object, or the electronic device may automatically select the orientation of the three-dimensional object (e.g., based on the type of the three-dimensional object).

[0019] In some examples, a three-dimensional object may be displayed in a three-dimensional environment in a world-locked orientation, a body-locked orientation, a tilt-locked orientation, or a head-locked orientation, as described below. As used herein, an object displayed in a three-dimensional environment in a body-locked orientation has a distance and orientation offset relative to a portion of the user's body (e.g., the user's torso). Alternatively, in some examples, the body-locked object has a fixed distance from the user, and the orientation of the content does not reference any part of the user's body (e.g., may be displayed in the same basic direction relative to the user regardless of head and / or body movement). Additionally or alternatively, in some examples, the body-locked object may be configured to always maintain gravity or horizon (e.g., perpendicular to gravity) alignment, so that changes in the head and / or body in the rolling direction will not cause the body-locked object to move within the three-dimensional environment. Instead, translational movement in either configuration will cause the body-locked object to reposition within the three-dimensional environment to maintain the distance offset.

[0020] As used herein, an object displayed in a head-locked orientation in a three-dimensional environment has a distance and orientation offset relative to the user's head. In some examples, the head-locked object moves within the three-dimensional environment when the user's head moves (when the user's viewpoint changes).

[0021] As used herein, an object displayed in a world-locked orientation in a three-dimensional environment has no distance or orientation offset relative to the user.

[0022] As used herein, an object displayed in a three-dimensional environment with a tilt-locked orientation (referred to herein as a tilt-locked object) has a distance offset relative to a user, such as a portion of a user's body (e.g., a user's torso) or a user's head. In some examples, the tilt-locked object is displayed with a fixed orientation relative to the three-dimensional environment. In some examples, the tilt-locked object moves according to a polar (e.g., spherical) coordinate system centered on a pole passing through the user (e.g., the user's head). For example, the tilt-locked object is moved in the three-dimensional environment based on the movement of the user's head in a spherical space around the user's head (e.g., centered on the user's head). Thus, if the user tilts his head relative to gravity (e.g., up or down in a pitch direction), the tilt-locked object will follow the head tilt and move radially along the sphere, so that the tilt-locked object is repositioned within the three-dimensional environment to the same distance offset relative to the user as before the head tilt, while optionally maintaining the same orientation relative to the three-dimensional environment. In some examples, if the user moves his head in a rolling direction (e.g., clockwise or counterclockwise) relative to gravity, the tilt-locked object is not repositioned within the three-dimensional environment.

[0023] Figure 1 An electronic device 101 is illustrated that presents an extended reality (XR) environment (e.g., a computer-generated environment that optionally includes representations of physical and / or virtual objects) according to some examples of the present disclosure. In some examples, such as Figure 1 As shown, the electronic device 101 is a head mounted display or other head mounted device configured to be worn on the head of the user of the electronic device 101. Figure 2 An example of an electronic device 101 is described by using an architectural block diagram of FIG. Figure 1 As shown, electronic device 101 and table 106 are located in a physical environment. The physical environment may include physical features, such as physical surfaces (e.g., floors, walls) or physical objects (e.g., tables, lights, etc.). In some examples, electronic device 101 may be configured to detect and / or capture an image of the physical environment including table 106 (shown in the field of view of electronic device 101).

[0024] In some examples, such as Figure 1As shown, the electronic device 101 includes one or more internal image sensors 114a (eg, as described below with reference to FIG. 1 ) oriented toward the user's face. Figure 2 101 and 102. In some examples, the internal image sensor 114a is used for eye tracking (e.g., detecting the gaze of the user). The internal image sensor 114a is optionally arranged on the left and right portions of the display 120 to enable eye tracking of the user's left and right eyes. In some examples, the electronic device 101 also includes external image sensors 114b and 114c facing outward from the user to detect and / or capture the physical environment of the electronic device 101 and / or the movement of the user's hands or other body parts.

[0025] In some examples, the display 120 has a field of view visible to the user (e.g., it may or may not correspond to the field of view of the external image sensors 114b and 114c). Because the display 120 is optionally part of the head-mounted device, the field of view of the display 120 is optionally the same as or similar to the field of view of the user's eyes. In other examples, the field of view of the display 120 may be smaller than the field of view of the user's eyes. In some examples, the electronic device 101 may be an optical see-through device, in which the display 120 is a transparent or translucent display through which parts of the physical environment can be directly viewed. In some examples, the display 120 may be included in a transparent lens and may overlap with all or only a portion of the transparent lens. In other examples, the electronic device may be a video pass-through device, in which the display 120 is an opaque display configured to display an image of the physical environment captured by the external image sensors 114b and 114c.

[0026] In some examples, in response to a trigger, the electronic device 101 may be configured to display in the XR environment Figure 1 , which does not exist in the physical environment but is displayed in the XR environment on top of the real-world table 106 (or its representation). Optionally, in response to detecting the plane of the table 106 in the physical environment 100, the virtual object 104 can be displayed on the surface of the table 106 in the XR environment displayed via the display 120 of the electronic device 101.

[0027] It should be understood that virtual object 104 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or other three-dimensional virtual objects) in a three-dimensional XR environment. For example, a virtual object may represent an application or user interface displayed in an XR environment. In some examples, a virtual object may represent content corresponding to an application and / or displayed via a user interface in an XR environment. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g., air gestures, such as air pinch gestures, air tap gestures, and / or air touch gestures), so that a user virtually touches, taps, moves, rotates, or otherwise interacts with virtual object 104.

[0028] In some examples, displaying an object in a three-dimensional environment may include interaction with one or more user interface objects in the three-dimensional environment. For example, initiation of display of an object in a three-dimensional environment may include interaction with one or more virtual options / indicators displayed in the three-dimensional environment. In some examples, when display of an object is initiated in a three-dimensional environment, an electronic device may track the user's gaze as an input for identifying one or more virtual options / indicators as targets for selection. For example, gaze may be used to identify one or more virtual options / indicators as targets for selection using another selection input. In some examples, a virtual option / indicator may be selected using hand tracking input detected via an input device that communicates with the electronic device. In some examples, an object displayed in a three-dimensional environment may be moved and / or reoriented in the three-dimensional environment based on movement input detected via an input device.

[0029] In the following discussion, an electronic device that communicates with a display generation component and one or more input devices is described. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. In addition, as described above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed between two or more devices. Therefore, as used in the present disclosure, information displayed on or by an electronic device is optionally used to describe information output by the electronic device to be displayed on an independent display device (touch-sensitive or non-touch-sensitive). Similarly, as used in the present disclosure, input received on an electronic device (e.g., touch input received on a touch-sensitive surface of an electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on an independent input device, and the electronic device receives input information from the independent input device.

[0030] The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television channel browsing application, and / or a digital video player application.

[0031] Figure 2 A block diagram of an example architecture of an electronic device 201 according to some examples of the present disclosure is illustrated. In some examples, the electronic device 201 includes one or more electronic devices. For example, the electronic device 201 can be a portable device, an auxiliary device that communicates with another device, a head-mounted display, etc. In some examples, the electronic device 201 corresponds to the above reference Figure 1 An electronic device 101 is described.

[0032] like Figure 2 As illustrated, the electronic device 201 optionally includes various sensors, such as one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206 (optionally corresponding to Figure 1 114a and / or external image sensors 114b and 114c in the embodiment of the present invention), one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, one or more body tracking sensors (e.g., torso and / or head tracking sensors), one or more display generation components 214 (optionally corresponding to Figure 1 The display 120 in the electronic device 201, one or more speakers 216, one or more processors 218, one or more memories 220 and / or communication circuits 222. One or more communication buses 208 are optionally used for communication between the above components of the electronic device 201.

[0033] The communication circuit 222 optionally includes circuits for communicating with electronic devices, networks such as the Internet, intranets, wired networks and / or wireless networks, cellular networks, and wireless local area networks (LANs). The communication circuit 222 optionally includes circuits for using near field communication (NFC) and / or short-range communication such as A circuit for communicating.

[0034] Processor 218 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 220 is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions, which are configured to be executed by processor 218 to perform the techniques, processes, and / or methods described below. In some examples, memory 220 may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in combination with an instruction execution system, device, and apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include, but is not limited to, a magnetic storage device, an optical storage device, and / or a semiconductor storage device. Examples of such storage devices include magnetic disks, optical disks based on compact disc (CD), digital versatile disc (DVD) or Blu-ray technology, and persistent solid-state memory (such as flash memory, solid-state drives, etc.).

[0035] In some examples, display generation component 214 includes a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generation component 214 includes multiple displays. In some examples, display generation component 214 may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, a transparent or translucent display, etc. In some examples, electronic device 201 includes touch-sensitive surfaces 209 for receiving user inputs (such as tap inputs and swipe inputs or other gestures). In some examples, display generation component 214 and touch-sensitive surface 209 form a touch-sensitive display (e.g., a touch screen integrated with electronic device 201 or a touch screen that communicates with electronic device 201 outside electronic device 201).

[0036] The electronic device 201 optionally includes an image sensor 206. The image sensor 206 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real-world environment. The image sensor 206 also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensor 206 also optionally includes one or more cameras configured to capture the movement of physical objects in the real-world environment. The image sensor 206 also optionally includes one or more depth sensors configured to detect the distance of the physical object from the electronic device 201. In some examples, information from the one or more depth sensors may allow the device to identify an object in the real-world environment and distinguish it from other objects in the real-world environment. In some examples, the one or more depth sensors may allow the device to determine the texture and / or topography of an object in the real-world environment.

[0037] In some examples, the electronic device 201 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around the electronic device 201. In some examples, the image sensor 206 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information of physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some examples, the electronic device 201 uses the image sensor 206 to detect the positioning and orientation of the electronic device 201 and / or the display generation component 214 in the real-world environment. For example, the electronic device 201 uses the image sensor 206 to track the positioning and orientation of the display generation component 214 relative to one or more fixed objects in the real-world environment.

[0038] In some examples, the electronic device 201 includes a microphone 213 or other audio sensor. The electronic device 201 optionally uses the microphone 213 to detect sounds from the user and / or the user's real-world environment. In some examples, the microphone 213 includes a microphone array (multiple microphones) that optionally operate in conjunction to identify ambient noise or locate a sound source in the space of the real-world environment.

[0039] The electronic device 201 includes a location sensor 204 for detecting the location of the electronic device 201 and / or the display generating component 214. For example, the location sensor 204 may include a global positioning system (GPS) receiver that receives data from one or more satellites and allows the electronic device 201 to determine the absolute location of the device in the physical world.

[0040] The electronic device 201 includes an orientation sensor 210 for detecting the orientation and / or movement of the electronic device 201 and / or the display generation component 214. For example, the electronic device 201 uses the orientation sensor 210 to track changes in the position and / or orientation of the electronic device 201 and / or the display generation component 214, such as relative to physical objects in the real world environment. The orientation sensor 210 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0041] In some examples, the electronic device 201 includes a hand tracking sensor 202 and / or an eye tracking sensor 212 and / or other body tracking sensors, such as a leg tracking sensor, a torso tracking sensor, and / or a head tracking sensor. The hand tracking sensor 202 is configured to track the position / location of one or more parts of the user's hand, and / or the movement of one or more parts of the user's hand relative to the extended reality environment, relative to the display generation component 214, and / or relative to another defined coordinate system. The eye tracking sensor 212 is configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or extended reality environment and / or relative to the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented together with the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented separately from the display generation component 214.

[0042] In some examples, the hand tracking sensor 202 (and / or other body tracking sensors, such as leg tracking sensors, torso tracking sensors, and / or head tracking sensors) may use an image sensor 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that captures three-dimensional information from the real world including one or more hands (e.g., hands of a human user). In some examples, the hands may be resolved with sufficient resolution to distinguish fingers and their corresponding positions. In some examples, one or more image sensors 206 are positioned relative to the user to define a field of view and an interaction space of the image sensor 206 in which finger / hand positions, orientations, and / or movements captured by the image sensor are used as input (e.g., to distinguish from the user's resting hands or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0043] In some examples, eye tracking sensor 212 includes at least one eye tracking camera (e.g., an infrared (IR) camera) and / or an illumination source (e.g., an IR light source such as an LED) that emits light toward the user's eyes. The eye tracking camera may be pointed toward the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by corresponding eye tracking cameras and illumination sources, and focus / gaze may be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by one or more corresponding eye tracking cameras / illumination sources.

[0044] The electronic device 201 is not limited to Figure 2 Rather than components and configurations of the electronic device 201, it may include fewer components, other components, or additional components in a variety of configurations. In some examples, the electronic device 201 may be implemented between two electronic devices (e.g., as a system). In some such examples, each (or more) electronic device may each include one or more of the same components discussed above, such as various sensors, one or more display generating components, one or more speakers, one or more processors, one or more memories, and / or communication circuits. One or more people using the electronic device 201 are optionally referred to herein as one or more users of the device.

[0045] Attention is now directed to interaction with one or more virtual objects displayed in a three-dimensional environment presented at an electronic device, e.g., corresponding to electronic device 201. In some examples, the electronic device adjusts the motion of the one or more virtual objects according to the detected movement pattern of the electronic device.

[0046] FIG. 3A to FIG. 3CAn example of the electronic device 101 updating the location of the virtual object 302 in the three-dimensional environment 300 according to the movement of the electronic device 101 according to some examples of the present disclosure is illustrated. Figure 3A In the embodiment of the present invention, for example, the electronic device 101 presents a three-dimensional environment 300, which includes representations of real objects in the environment of the electronic device 101 and virtual objects 302. In some examples, the three-dimensional environment 300 is a computer-generated environment, such as an extended reality environment. For example, the three-dimensional environment 300 may include a representation 310 of a real floor in the physical environment of the electronic device 101, representations 312a and 312b of real walls, and a representation 314 of a real window. The electronic device 101 optionally presents these representations 310, 312a, 312b, and 314 as optical perspective and / or video pass-through, in which parts of the physical environment of the electronic device 101 are visible through transparent or semi-transparent parts of the display, and in which the electronic device 101 uses the display to display a representation of the physical environment captured by the electronic device 101. In some examples, the virtual object 302 is a user interface element, a content item, or another virtual object displayed by the electronic device 101 using the display.

[0047] In some examples, electronic device 101 presents three-dimensional environment 300 from the viewpoint of a user of electronic device 101. In some examples, the viewpoint of the user of electronic device 101 is a location in three-dimensional environment 300 that corresponds to the location of electronic device 101 in the physical environment of electronic device 101. Figure 3A In the embodiment, the electronic device 101 presents a three-dimensional environment 300 including a view of a representation 314 of a real window, which view would correspond to a user's view of the real window in the physical environment at the location and orientation of the electronic device 101, and the electronic device 101 presents the three-dimensional environment 300 as shown in FIG. Figure 3A A three-dimensional environment 300 is shown.

[0048] In some examples, in response to detecting movement of electronic device 101, electronic device 101 updates three-dimensional environment 300 according to the movement of electronic device 101 to simulate movement through three-dimensional environment 300 corresponding to movement of electronic device 101 in the physical environment. Figure 3A As shown, electronic device 101 optionally detects movement of electronic device 101 to the right. In some examples, in response to detecting Figure 3A The illustrated movement of the electronic device 101, the electronic device 101 updates the Figure 3B The three-dimensional environment 300 is shown. Figure 3B As shown, for example, in response to detecting Figure 3A, the electronic device 101 updates the positions of the representations 310, 312a, and 314 of various real objects displayed relative to the display area of ​​the display. In some examples, the display of the representations 310, 312a, and 314 of the real objects is updated in a manner corresponding to the movement of the electronic device 101 in the physical environment to simulate the corresponding movement in the three-dimensional environment 300. For example, the electronic device 101 moves the representations 310, 312a, and 314 within the display area with minimal delay after detecting the movement of the electronic device 101. It should be understood that in an optical see-through implementation of the electronic device 101, the movement of the representations 310, 312a, and 314 within the transparent or semi-transparent display will occur naturally without the electronic device 101 having to actively update its position within the display.

[0049] In some examples, the virtual object 302 may be a head-locked object, and the electronic device 101 may update the position of the virtual object 302 in the three-dimensional environment 300 based on the change in the posture of the user's head so that it remains in the same position of the display of the electronic device 101. In other examples, the virtual object 302 may be a head-locked object, and the electronic device 101 may update the position of the virtual object 302 in the three-dimensional environment 300 based on the change in the posture of the user's head with a damping effect to simulate the virtual object 302 following the user with a delay. In some examples, the electronic device 101 simulates the movement delay of the virtual object 302 by applying damping to the position, velocity, and / or acceleration of the virtual object 302 in the three-dimensional environment relative to the position, velocity, and / or acceleration of the electronic device 101 in the physical environment. At a first time, such as, for example Figure 3A As shown, virtual object 302 may have a corresponding position relative to three-dimensional environment 300 and a corresponding position relative to the user's viewpoint. At a second time after the first time, such as Figure 3B As shown, the virtual object 302 may have different corresponding positions relative to the three-dimensional environment 300 and different corresponding positions relative to the user's viewpoint, rather than Figure 3A To illustrate this concept, Figure 3B The present invention includes indications of positions 304 and 306, which will now be described. For example, position 304 is where the positioning of the virtual object 302 relative to the three-dimensional environment 300 is responsive to movement of the electronic device 101 (e.g., no movement, infinite damping, or no interpolation relative to movement of the user's head) without being moved. Figure 3A and Figure 3B As another example, position 306 is when the positioning of the virtual object 302 relative to the user's viewpoint responds to movement of the electronic device 101 (e.g., undamped or maximally interpolated relative to movement of the user's head) without changing between positions 306 and 308. Figure 3A and Figure 3B In some examples, the electronic device 101 will display the position of the virtual object 302 when the virtual object 302 changes between Figure 3B , the electronic device 101 displays the virtual object 302 between positions 304 and 306 because the electronic device 101 moves the virtual object 302 with a damping amount greater than zero in accordance with the movement of the electronic device 101. For example, the electronic device 101 displays the virtual object 302 at a linearly interpolated position between position 304 (e.g., corresponding to no movement, infinite damping, or minimum interpolation) and position 306 (e.g., corresponding to no damping or maximum interpolation). In some examples, applying more interpolation may correspond to less damping, and applying less interpolation may correspond to more damping. For purposes of discussion, Figure 3B Indications of positions 304 and 306 are included, but in some examples, electronic device 101 does not display indications of positions 304 and 306 .

[0050] In some examples, when the electronic device 101 stops moving for a delay period corresponding to the movement of the virtual object 302, the electronic device 101 Figure 3A The position of the virtual object 302 relative to the user's viewpoint is displayed at the same position as the position of the virtual object 302 relative to the user's viewpoint in FIG. Figure 3C In the 3D environment 300, the user's viewpoint is at the same Figure 3B The same position as the user's viewpoint in FIG. 1 and the position of the virtual object 302 relative to the user's viewpoint is Figure 3A For illustration purposes, Figure 3C The virtual object 302 is also included relative to Figure 3A An indication of the position 304 of the three-dimensional environment 300 and the virtual object 302 relative to Figure 3B In some examples, the electronic device 101 does not display the indications of the positions 304 and 308; the indications of the positions 304 and 308 are provided for the purpose of explanation to illustrate the Figure 3A The virtual object 302 is displayed at the position 304 shown in FIG. Figure 3B The virtual object 302 is displayed at the position 308 shown in FIG. Figure 3C , as shown in Figure 3B A delay in displaying the virtual object 302 at position 306 in FIG.

[0051] Moving the virtual object 302 with a delay in the three-dimensional environment 300 can improve the user experience by damping the movement of the virtual object 302 in response to slight shifts in the position of the electronic device 101 and preventing the feeling that the virtual object 302 is rigidly attached to the electronic device (which can be distracting or uncomfortable for the user). In addition, in some cases, moving the virtual object 302 with a delay can enable the electronic device 101 to smooth the movement of the virtual object 302 while still moving the virtual object 302 according to the speed and / or direction of the movement of the electronic device 101, and maintaining the display of the virtual object 302 within the field of view of the display in the three-dimensional environment 300. In some examples, the amount of damping of the movement of the virtual object 302 can depend on one or more parameters of the interpolation applied by the electronic device 101 to the position of the virtual object 302 over time according to the movement of the electronic device 101. Example parameters of the interpolation include the relative weight between the positions 304 and 306 and / or the time period delay for the electronic device 101 to apply the interpolation.

[0052] When people walk or run, their heads can move in directions and / or dimensions other than the direction of travel. Therefore, in an example where the electronic device 101 is a head mounted display or includes a head mounted display, the electronic device 101 can also move in directions and / or dimensions other than the direction of travel during use. For example, when walking, in addition to moving in the direction of travel, the user's head can also move up and down according to the user's footsteps. As another example, when running, in addition to moving in the direction of travel, the user's head can also move up and down and move left and right according to the user's footsteps. When a person is walking or running, the human brain can ignore the user's environment relative to the user's head in directions other than the direction of travel, such as up and down movement / or left and right movement according to the user's footsteps. However, in some cases, for virtual objects displayed by electronic devices that simulate those elements that follow the user, the human brain cannot easily ignore such movement. For example, a virtual object displayed without delay in up and down movement and / or left and right movement may appear to the user as moving around and / or bouncing, which may distract or make the user uncomfortable or make it difficult for the user to focus on the virtual object. As another example, a virtual object displayed with the same up-and-down and / or side-to-side motion as the movement of the user's head while running or walking may appear to bounce, move around, and / or float in a manner that is distracting or uncomfortable to the user. Therefore, in some examples, the electronic device 101 applies a correction factor to the movement of the virtual object to adjust the delay and / or damping in the movement of the virtual object according to the user's footsteps (e.g., up-and-down and / or side-to-side), as described below with reference to FIG. 4A to FIG. 9 described.

[0053] FIG. 4A to FIG. 4BAn example of the electronic device 101 applying a first correction factor to the motion of the virtual object 402 when the electronic device 101 moves in a first movement mode according to an example of the present disclosure is illustrated. In some examples, the first movement mode may correspond to a user walking, and the second movement mode described below may correspond to a user running. In some examples, the first movement mode may correspond to a user jogging or running, and the second movement mode described below may correspond to a user sprinting. Other movement modes are possible without departing from the scope of the present disclosure.

[0054] In some examples, the electronic device 101 uses one or more first criteria to identify the first movement mode. For example, the one or more first criteria include criteria that are satisfied when the electronic device 101 detects a threshold amount (e.g., distance, speed, and / or frequency) of periodic and / or repeated up and down motion corresponding to the user's footsteps when the electronic device 101 moves according to the first movement mode (e.g., walking, jogging, or running according to the first movement mode). As another example, the one or more first criteria include criteria that are satisfied when the electronic device 101 detects a periodic and / or repeated movement left and right that is less than a threshold amount (e.g., distance, speed, and / or frequency), which will correspond to the user's footsteps in a different movement mode (e.g., running or sprinting according to the second movement mode). As another example, the one or more first criteria include criteria that are satisfied when the electronic device 101 detects movement of the electronic device 101 in the direction of travel in which the user is moving (such as forward movement when the user is walking, running, or jogging in the first movement mode).

[0055] Figure 4A An example of an electronic device 101 presenting a three-dimensional environment 400 is illustrated, the three-dimensional environment including representations 410, 412, and 414 of real objects in the physical environment of the electronic device 101 and a virtual object 402. For example, the three-dimensional environment 400 includes a representation 410 of a tree, a representation 412 of a bench, and a representation 414 of a path. In some examples, the electronic device 101 presents the representations 410 to 414 of the real objects using the optical perspective or video pass-through techniques described herein. In some examples, the virtual object 402 is a user interface element, a content item, or another virtual object displayed by the electronic device 101 using a display. In some examples, the electronic device 101 displays the virtual object 402 as the user's viewpoint moves through the three-dimensional environment 400 to simulate the virtual object 402 following the user. As described herein, in some examples, when the virtual object 402 is moved in the three-dimensional environment 400, the electronic device 101 applies a first correction factor corresponding to a first movement mode of the electronic device 101. For example, as described herein, Figure 4AAs shown, electronic device 101 detects movement of electronic device 101 in the direction of travel and up and down movement of electronic device 101 corresponding to a first movement mode (e.g., walking, jogging, or running). In some examples, a first correction factor is applied to virtual object 402 in response to a motion delay of up and down movement of electronic device 101, but is optionally not applied to movement of electronic device 101 in the direction of travel.

[0056] Figure 4B An example is illustrated in which the electronic device 101 applies the first correction factor to the movement of the virtual object 402 according to the first movement mode of the electronic device 101. Figure 4B For example, the user has changed from Figure 4A The position of the electronic device 101 in the user walks along the path in the direction of travel, and while the user is walking, the electronic device 101 has moved vertically upward according to the user's steps. Figure 4B As shown, for example, the presentation of representations 410 to 414 is updated according to the updated position of the electronic device 101 in the physical environment. Figure 4B As shown in the example of , the electronic device 101 updates the positioning of the virtual object 402 in the direction of travel of the electronic device 101 to simulate the virtual object 402 following the user and maintain the display of the virtual object 402 when the electronic device 101 moves in the physical environment.

[0057] In some examples, when the electronic device 101 is moving in the first movement mode, the electronic device 101 applies the first correction factor to the up and down movement of the virtual object 402. Figure 4B The virtual object 402 is displayed at the position shown because the electronic device 101 moves the virtual object 402 up and down at a speed and / or distance greater than zero but less than the speed and / or distance of the user's viewpoint moving up and down relative to the physical environment. To illustrate this concept, Figure 4B Indications of positions 404 and 406 are included, which will now be described. For example, position 404 is the position at which electronic device 101 would display virtual object 402 if the height of the object was fixed in three-dimensional environment 400 (e.g., corresponding to no movement or corresponding to infinite damping) rather than moving by the first correction factor. For example, position 406 is the position at which electronic device 101 would display virtual object 402 if electronic device 101 moved the virtual object up and down with the same motion as the up and down movement of electronic device 101 (e.g., no damping) rather than moving with the first correction factor. In some examples, Figure 4B, electronic device 101 displays virtual object 402 between positions 404 and 406 because electronic device 101 applies the first correction factor to the up and down motion of virtual object 402. For example, electronic device 101 displays virtual object 402 at a position that is linearly interpolated between position 404 (e.g., corresponding to no movement or corresponding to infinite damping) and position 406 (e.g., corresponding to no damping). In some examples, electronic device 101 does not display indications of positions 404 and 406; these indications are included in the figure for purposes of illustration and explanation. Figure 4B In some examples, the electronic device 101 applies damping and / or linear interpolation to the movement (e.g., positioning, velocity, and / or acceleration) of the virtual object 402 in the direction of travel of the electronic device 101 that is different from the first correction factor applied to the up and down movement of the first movement mode. For example, the first correction factor includes applying more damping to the up and down movement than to the movement in the direction of travel.

[0058] FIG. 5A to FIG. 5B An example of the electronic device 101 applying the first correction factor to the motion of the virtual object 502 when the electronic device 101 moves in the second movement mode according to an example of the present disclosure is illustrated. In some examples, the second movement mode may correspond to the user running, while the first movement mode described above may correspond to the user walking. In some examples, the second movement mode may correspond to the user sprinting, while the first movement mode described above may correspond to the user jogging or running. Other movement modes are possible without departing from the scope of the present disclosure.

[0059] In some examples, the electronic device 101 uses one or more second criteria to identify the second movement mode. In some examples, the one or more second criteria may be different from the one or more first criteria. For example, the one or more second criteria include criteria that are satisfied when the electronic device 101 detects a threshold amount (e.g., distance, speed, and / or frequency) of periodic and / or repeated up and down motion corresponding to the user's footsteps and another threshold amount (e.g., distance, speed, and / or frequency) of periodic and / or repeated left and right motion when the electronic device 101 moves according to the second movement mode (e.g., running or sprinting according to the second movement mode). As another example, the one or more second criteria include criteria that are satisfied when the electronic device 101 detects movement of the electronic device 101 in the direction of travel in which the user is moving (such as forward movement when the user is running or sprinting in the second movement mode).

[0060] Figure 5AAn example of an electronic device 101 presenting a three-dimensional environment 500 is illustrated, the three-dimensional environment including representations 510, 512, and 514 of real objects in the physical environment of the electronic device 101 and a virtual object 502. For example, the three-dimensional environment 500 includes a representation 510 of a tree, a representation 512 of a bench, and a representation 514 of a path. In some examples, the electronic device 101 presents the representations 510 to 514 of the real objects using the optical perspective or video pass-through techniques described herein. In some examples, the virtual object 502 is a user interface element, a content item, or another virtual object displayed by the electronic device 101 using a display. In some examples, the electronic device 101 displays the virtual object 502 as the user's viewpoint moves through the three-dimensional environment 500 to simulate the virtual object 502 following the user. As described herein, in some examples, when the virtual object 502 is moved in the three-dimensional environment 500, the electronic device 101 applies a second correction factor corresponding to a second movement mode of the electronic device 101. For example, as described herein, Figure 5A As shown, the electronic device 101 detects the movement of the electronic device 101 in the direction of travel and the up-and-down movement and / or the left-and-right movement of the electronic device 101 corresponding to the second movement mode (e.g., running or sprinting). In some examples, the second correction factor is applied to the movement delay of the virtual object 502 in response to the up-and-down movement of the electronic device 101 and the left-and-right movement of the electronic device 101, but is optionally not applied to the movement of the electronic device 101 in the direction of travel. As described in more detail below, in some examples, applying the second correction factor to the movement of the virtual object 502 may include applying different amounts of damping to the up-and-down movement and to the left-and-right movement. For example, the amount of damping applied to the up-and-down movement may be greater than the amount of damping applied to the left-and-right movement.

[0061] Figure 5B An example is illustrated in which the electronic device 101 applies the second correction factor to the movement of the virtual object 502 according to the second movement mode of the electronic device 101. Figure 5B For example, the user has changed from Figure 5A The position of the electronic device 101 in the running direction is along the path, and when the user is running, the electronic device 101 has moved vertically upward to the left according to the user's footsteps. Figure 5B As shown, for example, the presentation of representations 510 to 514 may be updated based on the updated position of the electronic device 101 in the physical environment. Figure 5B As shown in the example of , the electronic device 101 updates the positioning of the virtual object 502 in the direction of travel of the electronic device 101 to simulate the virtual object 502 following the user and maintain the display of the virtual object 502 when the electronic device 101 moves in the physical environment.

[0062] In some examples, when the electronic device 101 is moving in the second movement mode, the electronic device 101 applies the second correction factor to the up-and-down movement and / or the left-and-right movement of the virtual object 502. Figure 5B The virtual object 502 is displayed at the position shown because the electronic device 101 moves the virtual object 502 up and down and / or left and right at a speed and / or distance greater than zero but less than the speed and / or distance of the user's viewpoint moving up and down and / or left and right relative to the physical environment. To illustrate this concept, Figure 5B Indications of positions 504 and 506 are included, which will now be described. For example, position 504 is the position at which electronic device 101 would display virtual object 502 if the height and / or lateral positioning of the object were fixed in three-dimensional environment 500 (e.g., corresponding to no movement or corresponding to infinite damping) rather than moving by a second correction factor. For example, position 506 is the position at which electronic device 101 would display virtual object 502 if electronic device 101 moved the virtual object up and down and / or left and right with the same motion as the up and down movement and / or left and right movement of electronic device 101 (e.g., no damping) rather than moving by a second correction factor. In some examples, Figure 5B , the electronic device 101 displays the virtual object 502 between positions 504 and 506 because the electronic device 101 applies the second correction factor to the up-and-down motion and the left-and-right motion of the virtual object 502. For example, the electronic device 101 displays the virtual object 502 at a position that is linearly interpolated between position 504 (e.g., corresponding to no movement or corresponding to infinite damping) and position 506 (e.g., corresponding to no damping). Figure 5B As shown in , applying the second correction factor may include applying more damping to the up-and-down motion of the virtual object 502 than to the left-and-right motion of the virtual object 502. For example, Figure 5B The position of virtual object 502 in has a vertical position that is closer to the vertical position of position 506 (representing a fixed position or infinite damping) than the vertical position of position 504 (representing no damping). As another example, Figure 5B The position of virtual object 502 in FIG. 5 has a horizontal position that is closer to the horizontal position of position 504 (representing no damping) than the horizontal position of position 506 (representing a fixed position or infinite damping). In some examples, electronic device 101 does not display indications of positions 504 and 506; for purposes of illustration and explanation, these indications are included. Figure 5BIn some examples, the electronic device 101 applies damping and / or linear interpolation to the movement (e.g., positioning, velocity, and / or acceleration) of the virtual object 502 in the direction of travel of the electronic device 101 that is different from the first correction factor applied to the up-and-down movement and / or the left-and-right movement of the second movement mode. For example, the second correction factor includes applying more damping to the up-and-down movement and / or the left-and-right movement than to the movement in the direction of travel.

[0063] In some examples, the second correction factor may be different from the first correction factor. For example, the second correction factor may be applied to movement in more directions / dimensions than the first correction factor. For example, the second correction factor may be applied to up and down movement and left and right movement, while the first correction factor may be applied to up and down movement but not left and right movement. Additionally or alternatively, in some examples, the amount of damping and / or linear interpolation of the first correction factor may be different from the second correction factor. For example, the amount of damping in the up and down directions may be different between the first correction factor and the second correction factor. In some examples, in response to detecting the first movement mode or the second movement mode based on one or more first criteria or second criteria, the electronic device 101 may make additional or alternative modifications to the virtual object, as will be referenced below. 7A to 8C In some examples, these modifications may differ depending on whether the electronic device 101 detects a first movement pattern based on one or more first criteria or detects a second movement pattern based on one or more second criteria.

[0064] FIG. 6A to FIG. 6B An example is illustrated of the electronic device 101 modifying the movement of a virtual object when the electronic device 101 is moving according to the position of the gaze of the user of the electronic device 101 according to some examples of the present disclosure. In some examples, the electronic device 101 may adjust the amount of correction applied to the movement of the virtual object depending on whether the user's gaze is directed toward the virtual object, such as applying a first correction factor or a second correction factor when the user's gaze is directed toward the virtual object (e.g., for a threshold time of 0.1 seconds, 0.3 seconds, 0.5 seconds, 1 second, 2 seconds, or 3 seconds, or for any amount of time), and reducing or abandoning the first correction factor or the second correction factor when the user's gaze is away from the virtual object (e.g., for a threshold time or for any amount of time).

[0065] exist Fig. 6A , the electronic device 101 may present a three-dimensional environment 600 including a representation 610 of a real tree, a representation 612 of a real bench, and a representation 614 of a real path and a virtual object 602. For example, in Fig. 6A In some examples, the electronic device 101 is used when the user moves according to the second movement mode (e.g., running or sprinting). FIG. 5A to FIG. 5BThe second movement mode is detected according to one or more second criteria described in more detail. For example, when the user is moving according to the second movement mode, the electronic device 101 moves up and down and / or left and right according to the user's steps.

[0066] In some examples, while the user is moving according to the second movement mode, the electronic device 101 presents representations 610 to 614 of real objects in the three-dimensional environment 600, including movement of the representations 610 to 614 according to the movement of the electronic device 101. As described above, for example, because the user perceives themselves as moving in the environment, the user may not perceive the representations 610 to 614 of the real objects as moving.

[0067] like Fig. 6A As shown, for example, the user's gaze 603a is not directed at the virtual object 602. Instead, for example, the user's gaze 603a is directed at a tree in the three-dimensional environment. In some examples, when the user's gaze 603a turns away from the virtual object 602, the electronic device 101 stops applying the second correction factor to the movement of the virtual object 602 or reduces the second correction factor. Fig. 6A As shown, the amount of up and down movement and left and right movement of the virtual object 602 in the three-dimensional environment 600 according to the footsteps of the user is greater than when the user's gaze is directed to the virtual object 602. In some examples, reducing the application of the second correction factor includes moving the virtual object 602 up and down and / or left and right with a delay and / or damping smaller than the delay and / or damping of the second correction factor. In some examples, reducing the application of the second correction factor includes moving the virtual object 602 according to the movement of the representations 610 to 614 of the real objects in the three-dimensional environment. In some examples, reducing the application of the second correction factor includes moving the virtual object 602 up and down and / or left and right with the same delay and / or damping as the delay and / or damping of the electronic device 101 moving the virtual object 602 in the direction of travel of the electronic device 101. In some examples, while the user is not looking at the virtual object, the electronic device 101 does not apply the second correction factor to the movement of the virtual object 602. In some examples, the degree to which the electronic device 101 reduces the second correction factor varies between applying the second factor to the up and down movement of the virtual object and applying the second correction factor to the left and right movement of the virtual object 602. For example, electronic device 101 may reduce the second correction factor applied to up and down motion more than to left and right motion. As another example, electronic device 101 may reduce the second correction factor applied to left and right motion more than to left and right motion.

[0068] exist Figure 6B In the example, the user's gaze 603b is directed toward the virtual object 602. In some examples, Figure 6BIn the electronic device 101, the electronic device 101 is similar to the above reference Fig. 6A The representations 610 to 614 of the real objects are presented in the same manner as described above, thereby presenting the up-down movement and left-right movement of the representations 610 to 614 according to the up-down movement and left-right movement of the electronic device 101 caused by the user's footsteps. Figure 6B As shown, when the gaze 603b is directed to the virtual object 602, the electronic device 101 applies the second correction factor to the up and down movement and the left and right movement of the virtual object 602, thereby reducing the movement of the virtual object 602 in these directions. Therefore, in some examples, the electronic device 101 selectively applies the second correction factor to the movement of the virtual object 602 in response to detecting that the user's gaze 603b is directed to the virtual object 602, and reduces or omits the second correction factor in response to detecting that the user's gaze is away from the virtual object 602.

[0069] Although the description has been made with reference to the application of the second movement mode and the second correction factor Fig. 6A and Figure 6B , but this description is not intended to be limiting. In some examples, the electronic device 101 applies these techniques to the first correction factor in accordance with detecting movement of the electronic device 101 according to the first movement mode. For example, while detecting movement of the electronic device 101 according to the first movement mode, in response to detecting that the user's gaze is directed toward the virtual object, the electronic device 101 applies the first correction factor to the movement of the virtual object 602. For example, the electronic device 101 damps the up and down movement of the virtual object 602 according to the user's footsteps. In some examples, while detecting movement of the electronic device 101 according to the first movement mode, in response to detecting that the user's gaze is away from the virtual object 602, the electronic device 101 reduces the first correction factor or omits applying the first correction factor to the movement of the virtual object 602 in a manner similar to the selectively reducing application of the second correction factor described above.

[0070] 7A to 7C An example is illustrated in which the electronic device 101 according to some examples of the present disclosure modifies the content of the virtual object 702 according to the movement mode of the electronic device 101. In some examples, the virtual object 702 may include one or more user interface elements and / or content items, and in response to detecting the movement of the electronic device 101 according to a second or different movement mode, the electronic device 101 may reduce the number of user interface elements included in the virtual object 702.

[0071] Fig. 7A An example of an electronic device 101 displaying a virtual object 702 in a three-dimensional environment 700 is illustrated. For example, the three-dimensional environment 700 includes a representation 710 of a real tree, a representation 712 of a real bench, and a representation 714 of a real path. Fig. 7AIn some examples, the electronic device 101 may display the virtual object 702 with user interface elements 703, 704, 706, and 708 while the electronic device 101 is not moving or is moving less than a threshold amount (e.g., distance, duration, and / or speed). In some examples, the electronic device 101 may display the virtual object 702 with user interface elements 703, 704, 706, and 708 while the electronic device 101 is not moving or is moving less than a threshold amount. Additionally or alternatively, in some examples, the electronic device 101 may display the virtual object 702 with user interface elements 703 to 708 while the electronic device 101 is moving in a manner other than the first movement mode or the second movement mode. For example, movement other than the first movement mode or the second movement mode may include movement corresponding to a user turning or rotating their head, torso, and / or body or a user (e.g., as a passenger) moving in a vehicle. In some examples, the virtual object 702 may be a user interface of an application of the electronic device 101, and the user interface elements 703 to 708 may include selectable options, visual indications, content, and other interactive and / or non-interactive elements.

[0072] Figure 7B An example of an electronic device 101 displaying a virtual object 702 in a three-dimensional environment 700 is illustrated. Figure 7B In some examples, electronic device 101 detects that electronic device 101 is moving according to the first movement mode. For example, electronic device 101 detects that electronic device 101 is moving up and down according to the footsteps of the user (e.g., walking, jogging, or running) in the first movement mode. In some examples, when electronic device 101 detects movement in the first movement mode, electronic device 101 maintains display of virtual object 702 including user interface elements 703 to 708 described in more detail above. In some examples, in Figure 7B In the embodiment, when the movement of the first electronic device 101 is detected in the first movement mode, the electronic device 101 updates the positioning of the virtual object 702 according to the movement of the electronic device 101, including applying the first correction factor as described herein.

[0073] Figure 7C An example of an electronic device 101 displaying a virtual object 702 in a three-dimensional environment 700 is illustrated. Figure 7C In some examples, when electronic device 101 detects movement in the second movement mode, electronic device 101 reduces the number of user interface elements included in virtual object 702. For example, in Figure 7C In the example, virtual object 702 includes user interface element 708 but does not include FIG. 7A to FIG. 7B In some examples, when the electronic device 101 does not detect the second movement mode, FIG. 7A to FIG. 7B Compared to the size of the user interface element 708 in the second movement mode, the electronic device 101 increases in response to detecting movement in the second movement mode. Figure 7C In some examples, when the electronic device 101 does not detect the second movement mode, FIG. 7A to FIG. 7B Compared to the positioning of the user interface element 708 in the second movement mode, the electronic device 101 modifies the position of the user interface element 708 in the second movement mode in response to detecting the movement in the second movement mode. Figure 7C In some examples, in Figure 7C In the embodiment, when the movement of the first electronic device 101 is detected in the second movement mode, the electronic device 101 updates the positioning of the virtual object 702 according to the movement of the electronic device 101, including applying the second correction factor as described herein.

[0074] FIG. 8A to FIG. 8C An example is illustrated in which the electronic device 101 according to some examples of the present disclosure modifies the size of the virtual object 802 according to the movement mode of the electronic device 101. Fig. 8A In some examples, the electronic device 101 may present the three-dimensional environment 800 while the electronic device 101 detects that the electronic device 101 is not moving or is moving less than a threshold amount (e.g., speed, distance, and / or duration). In some examples, the three-dimensional environment 800 includes a representation 810 of a real tree, a representation 812 of a real bench, and a representation 814 of a real path. Fig. 8A As shown, while the electronic device 101 does not move or moves less than a threshold amount, the electronic device 101 displays the virtual object 802 at the first size. In some examples, in response to detecting a movement of the first electronic device 101 that does not correspond to the first movement mode or the second movement mode, such as the above examples, the electronic device 101 displays the virtual object 802 at the first size.

[0075] exist Figure 8B In some examples, while detecting movement corresponding to the first movement mode, the electronic device 101 may display the virtual object 802 with a first correction factor as described herein. Additionally or alternatively, in some examples, while detecting movement corresponding to the first movement mode, when the electronic device 101 does not detect movement corresponding to the first movement mode or the second movement mode, the electronic device 101 may detect movement corresponding to the first movement mode or the second movement mode. Fig. 8AIn some examples, increasing the size of virtual object 802 includes increasing the size of one or more user interface elements included in virtual object 802. Additionally or alternatively, in some examples, increasing the size of virtual object 802 includes modifying the padding between one or more user interface elements included in virtual object 802. In some examples, when electronic device 101 detects the first movement mode, electronic device 101 displays virtual object 802 at the same size as when electronic device 101 does not detect the first movement mode or the second movement mode, such as in Fig. 8A middle.

[0076] exist Figure 8C In some examples, while detecting movement corresponding to the second movement mode, the electronic device 101 may display the virtual object 802 with a second correction factor as described herein. Additionally or alternatively, in some examples, while detecting movement corresponding to the second movement mode, when the electronic device 101 detects movement corresponding to the first movement mode, the electronic device 101 may detect movement corresponding to the second movement mode. Figure 8B In some examples, increasing the size of virtual object 802 includes increasing the size of one or more user interface elements included in virtual object 802. Additionally or alternatively, in some examples, increasing the size of virtual object 802 modifies the padding between one or more user interface elements included in virtual object 802.

[0077] Additionally or alternatively, in some examples, the electronic device 101 limits the interactivity of the virtual object according to the detected movement mode. For example, one or more virtual objects may be interactive; in response to receiving input for the virtual object, the electronic device 101 may perform an associated action. In some examples, in response to detecting the input while the electronic device does not detect movement in the first movement mode or the second movement mode, the electronic device 101 performs an action associated with the virtual object. In some examples, in response to detecting the input for the virtual object while detecting the first movement mode (e.g., corresponding to walking, jogging, or running) or the second movement mode (e.g., corresponding to running or sprinting), the electronic device abandons performing the associated action. In some examples, the electronic device 101 performs the action in response to receiving the input while detecting the first movement mode, but abandons the action in response to receiving the input while detecting the second movement mode.

[0078] Fig. 9 An example method 900 of modifying the movement of a virtual object displayed in a three-dimensional environment according to a movement mode of the electronic device 101 according to some examples of the present disclosure is illustrated. In some examples, the electronic device 101 modifies the movement of a virtual object displayed in a three-dimensional environment according to the movement mode of the electronic device 101 described above. FIG. 3A to FIG. 8C One or more of the described techniques perform method 900 .

[0079] In some examples, method 900 may include presenting (902) a computer-generated environment (904) including a representation of a physical environment of the electronic device from a viewpoint of a user of the electronic device using a display. Figure 3A , the electronic device 101 presents a three-dimensional environment 300 including representations 310 to 314 of real objects in the physical environment of the electronic device 101 .

[0080] In some examples, method 900 may include presenting (902) a user interface element at a corresponding position relative to a user's viewpoint using a display. Figure 3A , the electronic device 101 presents a virtual object 302.

[0081] In some examples, method 900 may include detecting (908) movement of electronic device 101 relative to the physical environment while presenting the computer-generated environment. For example, detecting movement of electronic device 101 may include at least FIG. 4A to FIG. 4B As another example, detecting the movement of the electronic device 101 may include at least FIG. 5A to FIG. 5B The movement of the electronic device is shown to be detected according to the second movement mode.

[0082] In some examples, method 900 may include, in response to detecting movement of electronic device 101 (910), updating a display of a user interface element relative to a physical environment based on the movement of the selection device (912), including moving the user interface element relative to a user's viewpoint using a first correction factor based on determining that the movement of electronic device 101 satisfies one or more first criteria (914). For example, in FIG. 4A to FIG. 4B In the embodiment of the present invention, the electronic device 101 detects movement according to a first movement pattern based on one or more first criteria and, in response, moves the virtual object 402 according to a first correction factor.

[0083] In some examples, method 900 may include, in response to detecting movement of electronic device 101 (910), updating a display of a user interface element relative to a physical environment based on the movement of the selection device (912), including moving the user interface element relative to a user's viewpoint using a second correction factor based on determining that the movement of electronic device 101 satisfies one or more second criteria different from the one or more first criteria (916). For example, in FIG. 5A to FIG. 5B In the embodiment of the present invention, the electronic device 101 detects movement according to the second movement pattern based on one or more second criteria and, in response, moves the virtual object 502 according to the second correction factor.

[0084] It should be understood that method 900 is an example, and more, fewer, or different operations may be performed in the same or different order. Additionally, the operations in the above method 900 are optionally performed by running an information processing device (such as a general-purpose processor (e.g., as described in relation to Figure 2 One or more functional modules in the dedicated chip and / or by Figure 2 other components to achieve this.

[0085] Therefore, according to the above, some examples of the present disclosure relate to a method, which includes: at an electronic device that communicates with a display and one or more input devices of the electronic device, using the display to present: a view of a three-dimensional environment based on a physical environment of the electronic device from a viewpoint of a user of the electronic device, and user interface elements at corresponding positions relative to the viewpoint of the user; while presenting the view of the three-dimensional environment, detecting movement of the electronic device relative to the physical environment; and in response to detecting the movement of the electronic device: updating the presentation of the user interface element according to the movement of the electronic device includes: moving the user interface element relative to the three-dimensional environment according to the movement of the electronic device; moving the user interface element relative to the viewpoint of the user using a first correction factor according to the movement of the electronic device based on determining that one or more first criteria are satisfied, including a first criterion satisfied based on the movement of the electronic device; and moving the user interface element relative to the viewpoint of the user using a second correction factor different from the first correction factor according to the movement of the electronic device based on determining that one or more second criteria are satisfied, including a second criterion satisfied based on the movement of the electronic device. Additionally or alternatively, in some examples, the one or more first criteria and the one or more second criteria include criteria satisfied when the gaze of the user of the electronic device is directed toward the user interface element, and the method further includes: in response to detecting the movement of the electronic device, based on determining that the gaze of the user is not directed toward the user interface element, moving the user interface element relative to the viewpoint of the user with a third correction factor that is less than the first correction factor and less than the second correction factor. Additionally or alternatively, in some examples, moving the user interface element relative to the physical environment according to the movement of the electronic device includes: moving the user interface element in the first direction with an acceleration that is less than an acceleration of at least a portion of the movement of the electronic device in the first direction. Additionally or alternatively, in some examples, moving the user interface element relative to the physical environment according to the movement of the electronic device includes: moving the user interface element in the first direction with a speed equal to a speed of at least a portion of the movement of the electronic device in the first direction.Additionally or alternatively, in some examples, when the movement of the electronic device includes periodic movement of first and second amounts, respectively, relative to the physical environment in a first dimension and movement relative to the physical environment in a second dimension, the first criterion and the second criterion are met, and moving the user interface element relative to the user's viewpoint using the first correction factor according to the movement of the electronic device includes: corresponding to the periodic movement of the electronic device in the first dimension, moving the user interface element relative to the user's viewpoint in the first dimension with an amplitude greater than the amplitude of moving the user interface element relative to the user's viewpoint in the second dimension; and moving the user interface element relative to the user's viewpoint using the second correction factor includes: corresponding to the periodic movement of the electronic device in the first dimension, moving the user interface element relative to the user's viewpoint in the first dimension with an amplitude greater than the amplitude of moving the user interface element relative to the user's viewpoint in the second dimension. Additionally or alternatively, in some examples, when the movement of the electronic device includes periodic movement in the corresponding dimension with an amplitude greater than a predefined threshold, the first criterion is satisfied, and moving the user interface element relative to the user's viewpoint with the first correction factor according to the movement of the electronic device includes: moving the user interface element relative to the user's viewpoint in the corresponding dimension according to the periodic movement of the electronic device in the corresponding dimension; when the movement of the electronic device does not include the periodic movement in the corresponding dimension with an amplitude greater than the predetermined threshold, the second criterion is satisfied, and moving the user interface element relative to the user's viewpoint with the second correction factor according to the movement of the electronic device does not include: moving the user interface element relative to the user's viewpoint in the corresponding dimension according to the periodic movement of the electronic device in the corresponding dimension. Additionally or alternatively, in some examples, updating the presentation of the user interface element according to the movement of the electronic device also includes: presenting a first amount of information to the user interface element based on determining that the one or more first criteria are satisfied, and presenting a second amount of information greater than the first amount of information to the user interface element based on determining that the one or more second criteria are satisfied. Additionally or alternatively, in some examples, updating the display of the user interface element based on the movement of the electronic device also includes: presenting the user interface element at a first size based on determining that the one or more first criteria are satisfied, and presenting the user interface element at a second size smaller than the first size based on determining that the one or more second criteria are satisfied.Additionally or alternatively, in some examples, updating the display of the user interface element based on the movement of the electronic device also includes: based on determining that a portion of the movement of the electronic device relative to the physical environment is caused by movement of a vehicle that moves the electronic device: moving the user interface element relative to the physical environment based on the portion of the movement of the electronic device relative to the physical environment caused by the movement of the vehicle; based on determining that the one or more first criteria are satisfied, moving the user interface element relative to the user's viewpoint using the first correction factor regardless of the portion of the movement of the electronic device relative to the physical environment caused by the movement of the vehicle; and based on determining that the one or more second criteria are satisfied, moving the user interface element relative to the user's viewpoint using the second correction factor regardless of the portion of the movement of the electronic device relative to the physical environment caused by the movement of the vehicle. Additionally or alternatively, in some examples, moving the user interface element relative to the user's viewpoint using the first correction factor according to the movement of the electronic device includes: moving the user interface element relative to the user's viewpoint using a first interpolated amount relative to the movement of the electronic device in the first dimension, and moving the user interface element relative to the user's viewpoint using a second interpolated amount different from the first amount relative to the movement of the electronic device in the second dimension different from the first dimension; and moving the user interface element relative to the user's viewpoint using the second correction factor according to the movement of the electronic device includes: moving the user interface element relative to the user's viewpoint in the first dimension using a third interpolated amount relative to the movement of the electronic device in the first dimension, and moving the user interface element relative to the user's viewpoint using a fourth interpolated amount different from the third amount relative to the movement of the electronic device in the second dimension. Additionally or alternatively, in some examples, the first dimension is the direction of gravity, the first amount is less than the second amount, and the third amount is less than the fourth amount. Additionally or alternatively, some examples include, while presenting the user interface element, receiving input directed to the user interface element via the one or more input devices; and in response to receiving the input directed to the user interface element: performing an operation according to the user interface element based on determining that one or more third criteria are satisfied; and abandoning performing the operation according to the user interface element based on determining that the one or more third criteria are not satisfied. Additionally or alternatively, in some examples, the one or more third criteria are satisfied when the electronic device does not detect movement exceeding a threshold amount. Additionally or alternatively, in some examples, the one or more third criteria are satisfied based on satisfying the one or more first criteria, and are satisfied based on the one or more second criteria without satisfying the one or more third criteria.Additionally or alternatively, in some examples, the one or more third criteria are not satisfied based on satisfying the one or more first criteria or satisfying the one or more second criteria.

[0086] Some examples relate to an electronic device comprising: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing one or more of the methods described above. Some examples relate to a non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform one or more of the methods described above. Some examples relate to an electronic device comprising: one or more processors; a memory; and components for performing one or more of the methods described herein. Some examples relate to an information processing device for use in an electronic device, the information processing device including components for performing one or more of the methods described above.

[0087] For the purpose of explanation, the foregoing description is described by reference to specific examples. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible based on the above teachings. The examples are selected and described in order to best illustrate the principles of the invention and its practical application, so as to enable others skilled in the art to best use the invention and the various described examples with various modifications suitable for the specific purpose contemplated.

Claims

1. A method, comprising: At an electronic device in communication with a display and one or more input devices of the electronic device: Use the display to render: a view of a three-dimensional environment based on the physical environment of the electronic device from the viewpoint of a user of the electronic device, and user interface elements at respective positions relative to the viewpoint of the user; detecting movement of the electronic device relative to the physical environment while presenting the view of the three-dimensional environment; as well as In response to detecting the movement of the electronic device: Updating the presentation of the user interface element based on the movement of the electronic device includes: moving the user interface element relative to the three-dimensional environment based on the movement of the electronic device; Based on determining that one or more first criteria are satisfied, including a first criterion satisfied based on the movement of the electronic device, moving the user interface element relative to the viewpoint of the user using a first correction factor based on the movement of the electronic device; and Based on determining that one or more second criteria are satisfied, including a second criterion satisfied based on the movement of the electronic device, the user interface element is moved relative to the viewpoint of the user using a second correction factor different from the first correction factor based on the movement of the electronic device.

2. The method according to claim 1, wherein: The one or more first criteria and the one or more second criteria include criteria that are satisfied when the gaze of the user of the electronic device is directed toward the user interface element, and The method also includes: in response to detecting the movement of the electronic device, based on determining that the user's gaze is not directed at the user interface element, moving the user interface element relative to the user's viewpoint using a third correction factor that is smaller than the first correction factor and smaller than the second correction factor.

3. The method of claim 1 , wherein moving the user interface element relative to the physical environment based on the movement of the electronic device comprises: The user interface element is moved in the first direction with an acceleration that is less than an acceleration of at least a portion of the movement of the electronic device in the first direction.

4. The method of claim 1 , wherein moving the user interface element relative to the physical environment based on the movement of the electronic device comprises: The user interface element is moved in the first direction at a speed equal to a speed of at least a portion of the movement of the electronic device in the first direction.

5. The method according to claim 1, wherein: The first criterion and the second criterion are satisfied when the movement of the electronic device comprises periodic movement in a first dimension relative to the physical environment by a first amount and a second amount, respectively, and movement in a second dimension relative to the physical environment, Moving the user interface element relative to the viewpoint of the user with the first correction factor in accordance with the movement of the electronic device comprises: corresponding to the periodic movement of the electronic device in the first dimension, moving the user interface element relative to the viewpoint of the user in the first dimension by a magnitude greater than a magnitude by which the user interface element is moved relative to the viewpoint of the user in the second dimension, and Moving the user interface element relative to the user's viewpoint using the second correction factor includes: corresponding to the periodic movement of the electronic device in the first dimension, moving the user interface element relative to the user's viewpoint in the first dimension with an amplitude greater than the amplitude of moving the user interface element relative to the user's viewpoint in the second dimension.

6. The method according to claim 1, wherein: The first criterion is satisfied when the movement of the electronic device comprises a periodic movement in the corresponding dimension with an amplitude greater than a predefined threshold, Moving the user interface element relative to the viewpoint of the user using the first correction factor according to the movement of the electronic device comprises: moving the user interface element relative to the viewpoint of the user in the corresponding dimension according to the periodic movement of the electronic device in the corresponding dimension, the second criterion is satisfied when the movement of the electronic device does not include the periodic movement in the corresponding dimension with the amplitude greater than the predetermined threshold, Moving the user interface element relative to the viewpoint of the user using the second correction factor according to the movement of the electronic device does not include: moving the user interface element relative to the viewpoint of the user in the corresponding dimension according to the periodic movement of the electronic device in the corresponding dimension.

7. The method of claim 1 , wherein updating the presentation of the user interface element based on the movement of the electronic device further comprises: Based on determining that the one or more first criteria are satisfied, presenting a first amount of information to the user interface element, and Based on determining that the one or more second criteria are satisfied, a second amount of information greater than the first amount of information is presented to the user interface element.

8. The method of claim 1, wherein updating the display of the user interface element according to the movement of the electronic device further comprises: Based on determining that the one or more first criteria are satisfied, presenting the user interface element at a first size, and Based on determining that the one or more second criteria are satisfied, the user interface element is presented at a second size that is smaller than the first size.

9. The method of claim 1 , wherein updating the display of the user interface element based on the movement of the electronic device further comprises: Based on determining that a portion of the movement of the electronic device relative to the physical environment is caused by movement of a vehicle moving the electronic device: moving the user interface element relative to the physical environment in accordance with the portion of the movement of the electronic device relative to the physical environment caused by the movement of the vehicle; based on determining that the one or more first criteria are satisfied, moving the user interface element relative to the viewpoint of the user by the first correction factor independent of the portion of the movement of the electronic device relative to the physical environment caused by the movement of the vehicle; as well as Based on determining that the one or more second criteria are met, the user interface element is moved relative to the viewpoint of the user using the second correction factor, regardless of the portion of the movement of the electronic device relative to the physical environment caused by the movement of the vehicle.

10. The method of claim 1, wherein: Moving the user interface element relative to the viewpoint of the user using the first correction factor according to the movement of the electronic device includes: moving the user interface element in a first dimension relative to the viewpoint of the user using a first interpolation amount relative to movement of the electronic device in the first dimension, and relative to movement of the electronic device in a second dimension different from the first dimension, moving the user interface element relative to the viewpoint of the user in the second dimension by a second interpolated amount different from the first amount, and Moving the user interface element relative to the viewpoint of the user using the second correction factor according to the movement of the electronic device includes: moving the user interface element in the first dimension relative to the viewpoint of the user using a third interpolant relative to movement of the electronic device in the first dimension, and The user interface element is moved in the second dimension relative to the viewpoint of the user by a fourth interpolated amount different from the third amount relative to movement of the electronic device in the second dimension. 11 . The method of claim 10 , wherein the first dimension is a direction of gravity, the first amount is smaller than the second amount, and the third amount is smaller than the fourth amount.

12. The method according to claim 1, further comprising: While presenting the user interface element, receiving input directed to the user interface element via the one or more input devices; as well as In response to receiving the input directed to the user interface element: performing an operation based on the user interface element based on determining that one or more third criteria are satisfied; as well as Based on determining that the one or more third criteria are not satisfied, performing the operation based on the user interface element is abandoned.

13. The method of claim 12, wherein the one or more third criteria are satisfied when the electronic device does not detect movement exceeding a threshold amount.

14. The method of claim 12, wherein the one or more third criteria are satisfied based on satisfying the one or more first criteria, and the one or more third criteria are not satisfied based on satisfying the one or more second criteria.

15. The method of claim 12, wherein the one or more third criteria are not satisfied based on satisfying the one or more first criteria or satisfying the one or more second criteria.

16. An electronic device, comprising: one or more processors; Memory; and One or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 15.

17. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 15.

Images