Controlling virtual objects

Abstract

The invention relates to controlling a virtual object. According to some embodiments, an exemplary process is described for controlling a representation of a virtual object based on a user context of a location in a computer generated reality (CGR) environment. According to other embodiments, an exemplary process is described for controlling simultaneous display of representations of one or more virtual objects within a CGR environment. According to other embodiments, an exemplary process is described for controlling a representation of a virtual object in a CGR environment based on characteristics of an input mechanism.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 048833, international application date August 31, 2020, date of entry into the Chinese national phase October 9, 2021, Chinese national application number 202080027747.X, and invention name “Controlling Virtual Objects”.

[0003] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 058,217, filed on July 29, 2020, entitled “CONTROLLING REPRESENTATIONS OF VIRTUAL OBJECTS IN A COMPUTER-GENERATED REALITY ENVIRONMENT” and U.S. Provisional Patent Application Serial No. 62 / 907,216, filed on September 27, 2019, entitled “CONTROLLING REPRESENTATIONS OF VIRTUAL OBJECTS BASED ON USE CONTEXTS OF LOCATIONS IN A COMPUTER-GENERATED REALITY ENVIRONMENT”. The contents of the foregoing applications are hereby incorporated by reference in their entirety. Technical Field

[0004] The present disclosure relates generally to computer generated reality environments, and more particularly to controlling the display of virtual objects in a computer generated reality environment. Background Art

[0005] A computer generated reality (CGR) environment is an environment in which at least some of the objects displayed for viewing by a user are generated by a computer. In some applications, a user may interact with virtual objects. A user may move virtual objects to different locations within a computer generated reality environment. However, there is a lack of robust functionality for controlling how virtual objects are represented at different locations within a computer generated reality environment. Furthermore, there is a lack of functionality for controlling the representation of virtual objects in a computer generated environment based on the characteristics of an input mechanism that a user may use to interact with the virtual objects.

[0006] Furthermore, virtual objects may be represented as two-dimensional objects and / or three-dimensional objects within a computer-generated reality environment. However, there is a lack of robust functionality for controlling the simultaneous display of representations of virtual objects. Summary of the invention

[0007] According to some embodiments, a method includes: displaying a first representation of a virtual object at a first position within a CGR environment via a display of an electronic device, wherein the first position corresponds to a first usage context among multiple usage contexts; receiving a request to move the first representation within the CGR environment to a second position different from the first position; and in response to receiving the request: displaying a second representation of the virtual object at a second position based on the second usage context via the display of the electronic device based on a determination that the second position corresponds to a second usage context among multiple usage contexts, wherein the second representation is different from the first representation; and displaying a third representation of the virtual object at the second position based on the third usage context via the display of the electronic device based on a determination that the second position corresponds to a third usage context among multiple usage contexts, wherein the third representation is different from the first representation and the second representation.

[0008] According to some embodiments, a system includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: displaying a first representation of a virtual object at a first position in a CGR environment via the display, wherein the first position corresponds to a first usage context in a plurality of usage contexts; receiving a request to move the first representation in the CGR environment to a second position different from the first position; and in response to receiving the request: displaying a second representation of the virtual object at a second position based on the second usage context via the display based on a determination that the second position corresponds to a second usage context in a plurality of usage contexts, wherein the second representation is different from the first representation; and displaying a third representation of the virtual object at a second position based on the third usage context via the display based on a determination that the second position corresponds to a third usage context in a plurality of usage contexts, wherein the third representation is different from the first representation and the second representation.

[0009] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a system having a display, the one or more programs including instructions for the following operations: displaying a first representation of a virtual object at a first position within a CGR environment via the display, wherein the first position corresponds to a first usage context among multiple usage contexts; receiving a request to move the first representation within the CGR environment to a second position different from the first position; and in response to receiving the request: displaying a second representation of the virtual object at a second position based on the second usage context via the display based on a determination that the second position corresponds to a second usage context among multiple usage contexts, wherein the second representation is different from the first representation; and displaying a third representation of the virtual object at the second position based on the third usage context via the display based on a determination that the second position corresponds to a third usage context among multiple usage contexts, wherein the third representation is different from the first representation and the second representation.

[0010] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a system having a display, the one or more programs including instructions for the following operations: displaying a first representation of a virtual object at a first position within a CGR environment via the display, wherein the first position corresponds to a first usage context among multiple usage contexts; receiving a request to move the first representation within the CGR environment to a second position different from the first position; and in response to receiving the request: displaying a second representation of the virtual object at a second position based on the second usage context via the display based on a determination that the second position corresponds to a second usage context among multiple usage contexts, wherein the second representation is different from the first representation; and displaying a third representation of the virtual object at the second position based on the third usage context via the display based on a determination that the second position corresponds to a third usage context among multiple usage contexts, wherein the third representation is different from the first representation and the second representation.

[0011] According to some embodiments, a system includes: a display; a device for displaying a first representation of a virtual object at a first position within a CGR environment via the display, wherein the first position corresponds to a first usage context among multiple usage contexts; a device for receiving a request to move the first representation within the CGR environment to a second position different from the first position; and a device for performing the following operations in response to receiving the request: displaying a second representation of the virtual object at a second position based on the second usage context via the display based on a determination that the second position corresponds to a second usage context among multiple usage contexts, wherein the second representation is different from the first representation; and displaying a third representation of the virtual object at the second position based on the third usage context via the display based on a determination that the second position corresponds to a third usage context among multiple usage contexts, wherein the third representation is different from the first representation and the second representation.

[0012] According to some embodiments, a method includes: displaying a two-dimensional (2D) representation of a virtual object at a first position within a CGR environment via a display of an electronic device; receiving a request to simultaneously display a three-dimensional (3D) representation of the virtual object and the 2D representation; and in response to the request, simultaneously displaying the 2D representation at the first position and the 3D representation at a second position of the CGR environment via the display of a wearable electronic device, wherein the second position is different from the first position.

[0013] According to some embodiments, a system includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: displaying a 2D representation of a virtual object at a first location of a CGR environment via the display; receiving a request to display a 3D representation of the virtual object simultaneously with the 2D representation; and in response to the request, displaying the 2D representation at the first location of the CGR environment and the 3D representation at a second location via the display simultaneously, wherein the second location is different from the first location.

[0014] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a system including a display, the one or more programs including instructions for: displaying a 2D representation of a virtual object at a first location in a CGR environment via the display; receiving a request to simultaneously display a 3D representation and a 2D representation of the virtual object; and in response to the request, simultaneously displaying the 2D representation at the first location and the 3D representation at a second location of the CGR environment via the display, wherein the second location is different from the first location.

[0015] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a system including a display, the one or more programs including instructions for: displaying a 2D representation of a virtual object at a first location in a CGR environment via the display; receiving a request to simultaneously display a 3D representation and a 2D representation of the virtual object; and in response to the request, simultaneously displaying the 2D representation at the first location and the 3D representation at a second location of the CGR environment via the display, wherein the second location is different from the first location.

[0016] According to some embodiments, a system includes: a display; a device for displaying a 2D representation of a virtual object at a first position within a CGR environment via the display; a device for receiving a request to simultaneously display a 3D representation and a 2D representation of the virtual object; and a device for simultaneously displaying the 2D representation at the first position and the 3D representation at a second position of the CGR environment via the display in response to the request, wherein the second position is different from the first position.

[0017] According to some embodiments, a method includes: displaying a first representation of a virtual object in a CGR environment via a display of a wearable electronic device; in response to detected movement of an input mechanism: based on determining that a current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying a second representation of the virtual object in the CGR environment via the display of the wearable electronic device, wherein the second representation is different from the first representation; and based on determining that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, keeping the first representation displayed without displaying the second representation.

[0018] According to some embodiments, a system includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: displaying a first representation of a virtual object in a CGR environment via the display; and in response to the detected movement of an input mechanism: displaying a second representation of the virtual object in the CGR environment via the display based on determining that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, wherein the second representation is different from the first representation; and maintaining the display of the first representation without displaying the second representation based on determining that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object.

[0019] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a system including a display, the one or more programs including instructions for: displaying a first representation of a virtual object within a CGR environment via the display; and in response to detected movement of an input mechanism: based on determining that a current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying a second representation of the virtual object within the CGR environment via the display, wherein the second representation is different from the first representation; and based on determining that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, keeping the first representation displayed without displaying the second representation.

[0020] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a system including a display, the one or more programs including instructions for the following operations: displaying a first representation of a virtual object within a CGR environment via the display; and in response to detected movement of an input mechanism: displaying a second representation of the virtual object within the CGR environment via the display based on determining that a current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, wherein the second representation is different from the first representation; and keeping the first representation displayed without displaying the second representation based on determining that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object.

[0021] According to some embodiments, a system includes: a display; a device for displaying a first representation of a virtual object within a CGR environment via the display; and in response to detected movement of an input mechanism: a device for displaying a second representation of the virtual object within the CGR environment via the display based on determining that a current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, wherein the second representation is different from the first representation; and a device for keeping the first representation displayed instead of displaying the second representation based on determining that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object.

[0022] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In the following description, reference is made to the accompanying drawings that form a part of the present disclosure and illustrate multiple examples of the present disclosure. It should be understood that other examples may also be utilized, and structural and operational changes may be made without departing from the scope of the present disclosure. The same reference numerals are used in different drawings to represent similar or identical items.

[0024] Figure 1A to Figure 1B Depicted are exemplary systems for use in various computer-generated reality techniques.

[0025] FIG. 2A to FIG. 2E Depicted are exemplary techniques for controlling representation of virtual objects based on usage context at different locations within a CGR environment in accordance with aspects of the present disclosure.

[0026] FIG. 3A to FIG. 3CAnother example of a technique for controlling representation of a virtual object based on usage context at different locations within a CGR environment in accordance with aspects of the present disclosure is depicted.

[0027] FIG. 4A to FIG. 4C Yet another example of a technique for controlling representation of a virtual object based on usage context at different locations within a CGR environment in accordance with aspects of the present disclosure is depicted.

[0028] Figure 5 is a flow chart illustrating a method performed to implement aspects of the present disclosure.

[0029] FIG. 6A to FIG. 6C Depicted are exemplary techniques for controlling simultaneous display of representations of virtual objects within a CGR environment in accordance with aspects of the present disclosure.

[0030] Figure 7 is a flow chart illustrating a method performed to implement aspects of the present disclosure.

[0031] FIG. 8A to FIG. 8B Depicted are exemplary techniques for controlling representations of virtual objects within a CGR environment based on characteristics of an input mechanism in accordance with aspects of the present disclosure.

[0032] FIG. 9A to FIG. 9B Another example of a technique for controlling the representation of a virtual object within a CGR environment based on characteristics of an input mechanism in accordance with aspects of the present disclosure is depicted.

[0033] FIG. 10A to FIG. 10B Yet another example of a technique for controlling the representation of a virtual object within a CGR environment based on characteristics of an input mechanism in accordance with aspects of the present disclosure is depicted.

[0034] FIG. 11A to FIG. 11B Yet another example of a technique for controlling representation of a virtual object within a CGR environment based on characteristics of an input mechanism in accordance with aspects of the present disclosure is depicted.

[0035] FIG. 12A to FIG. 12B Another example of a technique for controlling the representation of a virtual object within a CGR environment based on characteristics of an input mechanism in accordance with aspects of the present disclosure is depicted.

[0036] Fig.13 is a flow chart illustrating a method performed to implement aspects of the present disclosure. DETAILED DESCRIPTION

[0037] Various examples of electronic systems related to various CGR techniques and techniques for using such systems are described.

[0038] The physical environment (or real environment) refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical items (or physical objects or real objects), such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0039] In contrast, a CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0040] A person may sense and / or interact with CGR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects, which create a (3D) or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with audio objects.

[0041] Examples of CGR include virtual reality and mixed reality.

[0042] A virtual reality (VR) environment (virtual environment) refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0043] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, an MR environment is anything between a fully physical environment at one end and a VR environment at the other end, but not including both ends.

[0044] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0045] Examples of MR include augmented reality and augmented virtuality.

[0046] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0047] An AR environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not true version of the original captured image. For another example, a representation of a physical environment may be transformed by graphically eliminating or blurring a portion thereof.

[0048] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0049] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one example, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0050] Figure 1A and Figure 1B An exemplary system 100 is depicted for use in various CGR techniques.

[0051] In some examples, such as Figure 1A As shown, system 100 includes device 100a. Device 100a includes various components, such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components are optionally in communication via a communication bus 150 of device 100a.

[0052] In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop computer), and other elements of system 100 are implemented in a head mounted display (HMD) device designed to be worn by a user, wherein the HMD device communicates with the base station device. In some examples, device 100a is implemented in either the base station device or the HMD device.

[0053] like Figure 1B As shown, in some examples, system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection. The first device 100b (e.g., a base station device) includes a processor 102, an RF circuit 104, and a memory 106. These components optionally communicate through a communication bus 150 of device 100b. The second device 100c (e.g., an HMD) includes various components, such as a processor 102, an RF circuit 104, a memory 106, an image sensor 108, an orientation sensor 110, a microphone 112, a position sensor 116, a speaker 118, a display 120, and a touch-sensitive surface 122. These components optionally communicate through a communication bus 150 of device 100c.

[0054] In some examples, system 100 is a mobile device. In some examples, system 100 is a HDM device. In some examples, device 100 is a wearable HUD device.

[0055] System 100 includes processor 102 and memory 106. Processor 102 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 102 to perform the following techniques.

[0056] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, an intranet), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 104 optionally includes circuitry for communicating using near field communications and / or short range communications (such as ) circuit for communication.

[0057] System 100 includes a display 120. In some examples, display 120 includes a first display (e.g., a left eye display panel) and a second display (e.g., a right eye display panel), each display being used to display an image to a corresponding eye of a user. Corresponding images are displayed simultaneously on the first display and the second display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby generating a parallax effect that provides the user with a stereoscopic effect of the object on the display. In some examples, display 120 includes a single display. For each eye of the user, the corresponding image is simultaneously displayed on a first area and a second area of ​​the single display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby generating a parallax effect that provides the user with a stereoscopic effect of the object on the single display.

[0058] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user inputs, such as tap inputs and swipe inputs. In some examples, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0059] System 100 includes an image sensor 108. Image sensor 108 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real environment. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the real environment. Image sensor 108 also optionally includes one or more event cameras configured to capture the movement of physical objects in the real environment. Image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of physical objects from system 100. In some examples, system 100 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical environment around system 100. In some examples, image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in the real environment from two different perspectives. In some examples, the system 100 uses the image sensor 108 to receive user input, such as gestures. In some examples, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and / or the display 120 in the real environment. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed objects in the real environment.

[0060] In some examples, system 100 includes microphone 112. System 100 uses microphone 112 to detect sounds from the user and / or the user's real environment. In some examples, microphone 112 includes a microphone array (including multiple microphones), which are optionally operated in series to identify ambient noise or locate a sound source in the space of the real environment.

[0061] System 100 includes an orientation sensor 110 for detecting orientation and / or movement of system 100 and / or display 120. For example, system 100 uses orientation sensor 110 to track changes in position and / or orientation of system 100 and / or display 120, such as with respect to physical objects in a real environment. Orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0062] Aspects of the present disclosure are directed to systems and techniques that provide functionality for controlling representations of virtual objects within a CGR environment.

[0063] In particular, aspects of the present disclosure relate to systems and techniques that provide functionality for controlling the representation of virtual objects based on a usage context associated with the location of the virtual objects within a CGR environment. The systems and techniques described herein allow the representation of virtual objects to be adapted to a specific usage context associated with a location within a CGR environment.

[0064] FIG. 2A to FIG. 2E Exemplary techniques for controlling the representation of virtual objects of a CGR environment based on a usage context associated with the location of the virtual objects within the CGR environment are shown in accordance with aspects of the present disclosure. Specifically, Figure 2A A user 202 and an electronic device 200 are shown. In some embodiments, the electronic device 200 may be a wearable electronic device (eg, an HMD). Examples of wearable electronic devices are described herein, such as with respect to the above reference Figure 1A and Figure 1B An electronic device 100a is described.

[0065] like Figure 2A As shown, user 202 wears electronic device 200, which is configured to enable user 202 to perceive CGR environment 290. As described above, CGR environment 290 may include physical objects or representations thereof and virtual objects, wherein virtual objects are superimposed on physical objects (e.g., in AR implementations) or physical objects are superimposed on virtual objects (e.g., in AV implementations) to present a coherent CGR environment to user 202. In some embodiments, CGR environment 290 may be a fully virtual environment (e.g., in VR implementations), wherein each object within CGR environment 290 is a virtual object. Whether fully virtual or partially virtual, in Figure 2A In the example shown, the virtual object 210 can be a representation of a presentation application (e.g., an application configured to facilitate multimedia presentation) and can be presented to the user 202 within the CGR environment 290.

[0066] In an embodiment, the virtual object 210 may be located anywhere within the CGR environment 290. FIG. 2A to FIG. 2E In the specific example shown: CGR environment 290 may include at least positions 220, 222, 224, 226, and 228. It should be understood that these positions are described for illustrative purposes and are not intended to be limiting in any way. That is, any other position within CGR environment 290 may be applicable to the features and functions described herein.

[0067] In various aspects, the location 220 may correspond to a location on a representation of an electronic device within the CGR environment 290. For example, the location 220 may correspond to a location on a display 240 (e.g., a display, screen, surface or housing of an electronic device). The display 240 may be, for example, a display of a computer, laptop, tablet, phone, monitor, projector display, etc. The display 240 may be an actual physical device (e.g., a physical object) or may be a virtual representation of a display (e.g., a virtual object) within the CGR environment 290.

[0068] The location 222 may correspond to a location on a vertical plane of the CGR environment 290 (e.g., a primary vertical plane, such as a structure that is a vertical plane, a wall, a surface corresponding to a wall-like structure, such as the side of a building, a bedroom wall, a fence, a vertical or secondary vertical plane, etc.). Figure 2A In the particular example shown, location 222 corresponds to a location on a wall of CGR environment 290 .

[0069] Position 224 and / or position 228 may correspond to a position on a horizontal plane of CGR environment 290 (e.g., a primary horizontal plane, such as a horizontal plane, a tabletop, a desk, a countertop, a shelf, a floor, an elevated horizontal plane, such as a horizontal plane above another horizontal plane within the CGR environment, a non-elevated horizontal plane, etc.). Figure 2A In the particular example shown, locations 224 and 228 correspond to locations on a desktop 242, which may be a physical object or a virtual object.

[0070] Location 226 may correspond to a location on a horizontal plane of CGR environment 290, but be of a different type than locations 224 and / or 228. For example, location 226 may be a location on a predominantly horizontal plane, such as a structure that is a horizontal plane, a floor, a sidewalk, grass, a lawn, a surface on which one or more people stand, a non-elevated horizontal plane (such as a horizontal plane below another horizontal plane within the CGR), and the like. Figure 2A In the particular example shown, location 226 corresponds to a location on the floor of CGR environment 290 .

[0071] like Figure 2AAs shown, the virtual object 210 may be displayed (e.g., by the electronic device 200) at the location 220. In some embodiments, the location (e.g., location 220) within the CGR environment 290 may be associated with or otherwise correspond to at least one of a plurality of usage contexts. In an embodiment, the usage context may be related to the type of surface (e.g., desk, wall, computer screen, floor, etc.) or the type of material of the surface (e.g., sand, grass, concrete, carpet, etc.) on which the virtual object will be placed, and / or may be related to the manner in which the virtual object will be used (e.g., manipulated, interacted with) or displayed (e.g., presented) in the CGR environment.

[0072] In various aspects, the position 220 may be associated with a first usage context. For example, as described above, the position 220 may be a position on the display 240. The display 240 may be a representation of an electronic device. In this case, the first usage context associated with the position 220 may be the type of surface or object of the position 220 as an electronic device. Therefore, in this case, when it is determined that the position 220 is a position on the representation of an electronic device, the first usage context may be satisfied. In other embodiments, the first usage context associated with the position 220 may be the manner in which the virtual object 210 will be used when in the position 220. For example, it may be determined that at the position 220 (which is an electronic device), the virtual object 220 will be used as an application for multimedia presentation on the display 240. In this case, it may be determined that the virtual object will be represented as a two-dimensional (2D) window based on the manner in which the virtual object will be used.

[0073] Note that, as used herein, a representation of a virtual object may include the content, size, functionality, user interface object, form, shape, design, graphical presentation, etc. of the virtual object within a CGR environment. For example, a virtual object may be represented as a 2D object (e.g., an application icon, an application window, an image, a user interface of an application, etc.). In other examples, a virtual object may be represented as a 3D object within a CGR environment. In some embodiments, a first representation of a virtual object may be a 3D object including specific content, and a different second representation of the virtual object may be a 3D object including content different from the specific content in the first representation.

[0074] In some embodiments, representations of virtual objects within a CGR environment may include audio features. For example, one representation may include a particular sound, noise, spoken word, etc., and a second representation may include a different sound, noise, spoken word, etc. In some cases, representations of virtual objects may also include sound levels, wherein one representation of a virtual object may include one sound level, and a different representation may include a higher or lower sound level.

[0075] As described above, when the virtual object 210 is at least partially located at the position 220, whether by moving or dragging to the position 220 or by being displayed at the position 220, the virtual object 210 is displayed as a 2D window on the display 240 (for example, by the electronic device 200) based on determining that the position 220 is associated with a usage context, and the usage context is satisfied by determining that the position 220 is on the display 240, which is an electronic device.

[0076] In some embodiments, the virtual object 210 may be configured so that the user 202 can interact with the virtual object 210. As described above, the interaction with the virtual object 220 may be via an input sensor configured to detect user input interacting with the virtual object of the CGR environment 290. In some embodiments, the input sensor may include a mouse, a stylus, a touch-sensitive surface, an image sensor (e.g., to perform hand tracking), etc., which may be configured to allow the user 202 to grab, move, drag, click, select, and / or otherwise select the virtual object 210. Thus, in an embodiment, a request to move the virtual object 210 to a location within the CGR environment 290 may be received.

[0077] exist Figure 2A In the example shown, the request to move the virtual object 210 from the location 220 to another location within the CGR environment 290 may include the user 202 grabbing or otherwise selecting the virtual object 210 for moving from the location 220, and may cause the virtual object 210 to leave the location 220. In some embodiments, once the virtual object 210 is removed from a location (e.g., location 220), the current representation of the virtual object 210 may change. For example, once the virtual object 210 is removed from the location 220, the current representation of the virtual object 210 as a 2D window of a multimedia presentation application may change to another representation. In some specific implementations, the current representation of the virtual object 210 may change to some transition representation, which may not be associated with a specific usage context, but may be a default representation indicating that the virtual object 210 is transitioning from one location to another. In other implementations, the current representation of the virtual object 210 may not change when the virtual object 210 is removed from the location, but rather the current representation of the virtual object 210 may remain unchanged until the virtual object is positioned at another location determined to be associated with a usage context in which a different representation of the virtual object 210 may be determined to be displayed. In this case, the current representation of the virtual object 210 may be maintained during the transition of the virtual object 210 from the current location to the new location.

[0078] Figure 2BAn example of a virtual object 210 displayed at a position 224 (e.g., by an electronic device 200) is shown. In this example, in response to a request to move a virtual object to the position 224, at least one usage context corresponding to the position 224 may be determined. For example, the position 224 may correspond to a position on a desktop 242. In this case, it may be determined that the position 224 is associated with a usage context satisfied by a position type (e.g., surface type, air) of the position 224, and the position 224 is a position on the desktop 242 (e.g., a position on a horizontal plane). In an alternative or additional embodiment, the position 224 on the desktop 242 may be determined as, for example, a position at which the user 202 may use the virtual object 210 to annotate a multimedia presentation. In either case, whether because the position 224 is a position on the desktop or because the position 224 is a position at which the virtual object may be used to annotate a multimedia presentation, the virtual object 210 may be represented as a 3D object (e.g., a notepad, a notebook, a book, or any other 3D representation) that is configured to facilitate the user to annotate and / or take notes on the multimedia presentation.

[0079] Although not shown, the virtual object 210 may be moved from the position 224 on the desktop 242 to the position 228, which is also on the desktop 242. In an embodiment, the representation of the virtual object 210 (e.g., a 3D virtual notepad) may remain the same at the position 228 as in the position 224, because the two positions may be associated with the same context of use. Alternatively, although both the positions 224 and 228 are on the desktop 242 (e.g., the same type of surface), the representation of the virtual object when at the position 228 may be different from the representation when at the position 224. For example, the representation of the virtual object when at the position 228 may have a different size (e.g., smaller or larger) or a different orientation than the representation when at the position 224, because the position 228 may be determined to be unable to accommodate the size and / or orientation of the representation of the virtual object 210 when at the position 224. In some embodiments, different positions within the same type of surface (e.g., different positions on the desktop 242, on the wall 222, etc.) may be configured for different contexts of use. For example, a particular location on the desktop 242 may be configured to have a usage context in which a representation of the virtual object 210 may be used in a particular language, and another location on the desktop 242 may be configured to have a usage context in which a representation of the virtual object 210 may be used in a different language.

[0080] Figure 2CAn example of a virtual object 210 displayed at a location 222 is shown. For example, a request to move the virtual object 210 to the location 222 may be received. The request may include a request to move the virtual object 210 from any other location within the CGR environment 290 (e.g., location 220, location 224, location 226, etc.). In response to the request, the virtual object 210 may be moved to the location 222, and a representation of the virtual object 210 to be displayed at the location 222 may be determined. In this example, in response to the request to move the virtual object to the location 222, at least one usage context corresponding to the location 222 may be determined. For example, the location 222 may correspond to a location on a vertical plane (e.g., a wall) of the CGR environment 290. In this case, it may be determined that the location 222 is associated with a usage context satisfied by a location type (e.g., a surface type) of the location 222, which is a location on a wall of the CGR environment 290. In an alternative or additional embodiment, a location 222 on a wall of the CGR environment 290 may be determined as a location where the virtual object 210 may be used to present a multimedia presentation. In either case, whether because the location 222 is a location on a wall or because the location 222 is a location where the virtual object may be used to present a multimedia presentation, the virtual object 210 may be represented (e.g., displayed by the electronic device 200) as a large window object configured to facilitate the presentation of the multimedia presentation. For example, the large window object may be a 3D representation of a 2D window or a large monitor that is displayed as being fixed against a wall. In some embodiments, the size of the large window object against the wall may be determined based on the distance of the wall against which the large window object is displayed relative to the location of the user 202 within the CGR environment 290.

[0081] In some embodiments, the content (e.g., information and / or information arrangement) of the representation of the virtual object 210 at the location 222 may be different from the content in the representation of the virtual object 210 at other locations. For example, when at the location 224, the 3D notepad used as the representation of the virtual object 210 may include information arranged in a particular arrangement within the 3D notepad. While at the location 222, the large window display against the wall used as the representation of the virtual object 210 may include different information that may be arranged in a different arrangement within the large window display.

[0082] Figure 2DAn example of a virtual object 210 displayed at position 226 (e.g., by electronic device 200) is shown. For example, a request to move virtual object 210 to position 226 may be received. The request may include a request to move virtual object 210 from any other location within CGR environment 290 (e.g., location 220, location 222, location 224, location 228, etc.). In response to the request, virtual object 210 may be moved to position 226 and a representation of virtual object 210 to be displayed at position 226 (e.g., by electronic device 200) may be determined. In this example, in response to the request to move the virtual object to position 226, at least one usage context corresponding to position 226 may be determined. For example, position 226 may correspond to a location on a horizontal plane (e.g., a floor) of CGR environment 290. Note that in this example, position 226 corresponds to a different type of location on a horizontal plane than the horizontal plane corresponding to position 224, which is a location on desktop 242. In this case, the location 226 may be determined to be associated with a usage context that is satisfied by the location type (e.g., surface type) of the location 226, which is a location on the floor of the CGR environment 290. In alternative or additional embodiments, the location 226 on the floor of the CGR environment 290 may be determined to be a location where the virtual object 210 may be used to present a multimedia presentation at least partially immersively (e.g., from a first-person view mode). In either case, whether because the location 226 is a location on a wall or because the location 226 is a location where the virtual object may be used to present a multimedia presentation at least partially immersively, the virtual object 210 may be represented as a 3D podium placed on or near the location 226, the podium being configured to facilitate presentation of the multimedia presentation from the podium by the user 202. In some embodiments, the representation of the virtual object 210 at the location 226 may include content 212 related to the multimedia presentation (e.g., a notepad, annotations, presentation content, etc.), and may be presented on top of the podium where the user 202 may perceive the content 212.

[0083] Figure 2E An example of a virtual object 210 displayed in a fully immersive mode (e.g., by an electronic device 200) is shown. In some embodiments, a particular location may be associated with a fully immersive use context. For example, a location (such as location 226 on the floor of a CGR environment 290) may be associated with a use context in which the presentation will be presented as a fully immersive experience. In response to a request to move the virtual object 210 to location 226, the virtual object may be moved to location 226, and a fully immersive representation of the virtual object 210 may be displayed. In this case, displaying the virtual object 210 as a fully immersive representation may include displaying the entire CGR environment 290 as a virtual auditorium configured to present a multimedia application.

[0084] In some embodiments, a representation of a virtual object 210 associated with a particular usage context may be displayed without having to move the virtual object to a particular location. Figure 2A In some embodiments, the enablement representation 214 may be presented within the CGR environment 290. The enablement representation 214 may be a virtual object (e.g., a button, an enablement representation, a user interface element, an interactive element, etc.) configured to allow a user (e.g., user 202) to interact. The enablement representation 214 may correspond to at least one usage context. In some embodiments, the enablement representation 214 may also be associated with the virtual object 214 (e.g., associated with a specific application of the virtual object 214 such as multimedia presentation, a calculator, weather, etc.). When the user 202 selects the enablement representation 214 of the virtual object 210, the usage context corresponding to the enablement representation 214 may be considered to be satisfied, and the associated representation (e.g., a representation of the virtual object 210 associated with the usage context) may be displayed. For example, in the case where the enablement representation 214 corresponds to a usage context associated with a location 224 (e.g., a desktop), as Figure 2B As shown, a representation of virtual object 210 (such as a 3D notepad) can be displayed by electronic device 200. In some cases, the representation of virtual object 210 can be displayed at a location associated with the usage context (e.g., it is not necessary to move virtual object 210 from its current location to a location corresponding to the usage context associated with affordance 214), or can be displayed at any location where virtual object 210 is currently being displayed. In some embodiments, displaying a representation of virtual object 210 at a location associated with a usage context corresponding to affordance 214 may include moving virtual object 210 from its current location to a location associated with the usage context. In these cases, the movement of virtual object 210 to the location associated with the usage context can be displayed in an animated manner.

[0085] As another example, where affordance 214 corresponds to a usage context associated with a fully immersive usage context, Figure 2E As shown, a representation of a virtual object 210 as a fully immersive experience may be displayed by the electronic device 200 in response to the user 202 selecting the affordance 214 .

[0086] In some embodiments, the enablement representation 214 may include multiple enablement representations, each of which corresponds to a specific usage context. In these embodiments, each of the multiple enablement representations can be an optional enablement representation, which, when selected, can cause the corresponding usage context to be considered satisfied, and can cause the associated representation (e.g., a representation of the virtual object 210 associated with the satisfied usage context) to be displayed according to the foregoing content.

[0087] It should be noted that although the present disclosure describes an embodiment in which a virtual object is displayed at a single location within a CGR environment at one time, this is done for illustrative purposes and should not be construed as limiting in any way. In fact, in some embodiments, separate and in some cases different representations of the same virtual object may be displayed simultaneously at more than one location within the CGR environment. In an embodiment, the separate representations at different locations may all be different (e.g., may include different information or may have different shapes and / or forms, as described above), or some representations at different locations may be the same, while other representations at other locations may be different. In some embodiments, a change to the configuration of a virtual object (e.g., a change to an application associated with the virtual object) may trigger a change to all representations at all locations, or may trigger a change to some representations at some locations rather than to all representations at all locations. In some cases, a change to a representation at one location within a CGR environment (e.g., a change caused in response to a user interaction and / or caused by a change in an associated application) may trigger at least one change to at least one representation of a virtual object at another location within the CGR environment.

[0088] FIG. 3A to FIG. 3C An example of functionality for controlling the representation of a virtual object based on a usage context associated with a location within a CGR environment in accordance with aspects of the present disclosure is shown. Specifically, Figure 3A 2 shows a user 202 wearing an electronic device 200, which can be configured to allow the user 202 to view a CGR environment 290. In some embodiments, the electronic device 200 can be similar to the one described above with reference to FIG. Figure 1A and Figure 1B An electronic device 100a is described.

[0089] The CGR environment 290 includes a display 340, which may be a physical display or a virtual representation of a display. In any case, a representation of the virtual object 310 may be displayed at a location 320 (e.g., by the electronic device 200), which is a location on the display 340. Figure 3AIn the example shown, the virtual object 310 may be a calculator application. In this case, it may be determined that the position 320 corresponds to at least one usage context (e.g., the type of location, surface, material, etc., and / or the type of use of the virtual object at the location). For example, the position 320 may be determined as a location on an electronic device (e.g., a physical device or a computer-generated simulation of a physical device) of the CGR environment 290. In this case, it may be determined that the position 320 is associated with a usage context satisfied by the location type (e.g., surface type) of the position 320, and the position 320 is a location on the electronic device. Based on determining that the position 320 is a location on the electronic device, the virtual object 310 may be displayed as a 2D window or desktop applet of a calculator application on the display 340 (e.g., by the electronic device 200). Therefore, it should be understood that the representation of the virtual object 310 at the position 320 is based on the usage context corresponding to the position 320.

[0090] Figure 3B The user 202 is shown interacting with the virtual object 310 at a location 320. The user 202's interaction with the virtual object 310 at the location 320 may include a request to move the virtual object 310 to another location (eg, location 324). Figure 3C It is shown that in response to the request to move the virtual object 310, the virtual object 310 has been moved to the position 324. In this example, at least one usage context associated with the position 324 can be determined. For example, the position 324 is a position on the desktop 342. In this case, it can be determined that the position 324 is associated with a usage context satisfied by the position type (e.g., surface type) of the position 324, and the position 324 is a position on the desktop 342 (e.g., a position on a horizontal plane). In an alternative or additional embodiment, the position 324 on the desktop 342 can be determined as a position at which the virtual object 310 (e.g., a calculator application) can be used, for example, by the user 202 to manipulate the calculator application in a manner such as input into the calculator application in a real-world physical calculator by using the user 202's hand or a virtual representation thereof. In either case, whether because location 324 is a location on a desktop or because location 324 is a location where the virtual object can be used to enter into a calculator using the user's hand or a virtual representation thereof, virtual object 310 may be represented as a 3D object (e.g., a 3D representation of a physical calculator) that is configured to facilitate the user entering entries into a calculator application.

[0091] FIG. 4A to FIG. 4C Another example of a representation of a virtual object of a CGR environment based on a usage context associated with a location of the virtual object within the CGR environment is shown in accordance with aspects of the present disclosure. Specifically, Figure 4A2 shows a user 202 wearing an electronic device 200, which is configured to allow the user 202 to view a CGR environment 290. As described above, in some embodiments, the electronic device 200 may be similar to the device described above with reference to FIG. Figure 1A and Figure 1B An electronic device 100a is described.

[0092] The CGR environment 290 includes a display 440. As described above, the display 440 can be a physical display or a virtual representation of a display. A representation of the virtual object 410 can be displayed by the electronic device 200 at a location 420, which is a location on the display 440. Figure 4A In the example shown, virtual object 410 can be an application for presenting an interactive and / or animated robot. It should be understood that the description of the animated robot herein is for illustrative purposes only and should not be construed as limiting in any way. In fact, the technology herein is applicable to any application that can be represented as a virtual object within a CGR environment. In this example, position 420 can be determined as a position on a representation of an electronic device (e.g., a representation of a display of a physical computer). Based on determining that position 420 is a position on a representation of an electronic device, virtual object 410 can be displayed (e.g., by electronic device 200) as a 2D window or desktop applet on display 440.

[0093] Figure 4B It is shown that the virtual object 410 has been moved to the position 424. In various aspects, the virtual object 410 may be moved to the position 424 in response to a user's request (e.g., the user interacts with the virtual object 410 to drag or otherwise cause the virtual object 410 to be moved to the position 424). In this example, at least one usage context associated with the position 424 may be determined. For example, the position 424 is a position on the desktop 442. In this case, it may be determined that the position 424 is associated with a usage context satisfied by the position type (e.g., surface type) of the position 424, and the position 424 is a position on the desktop 442 (e.g., a position on a horizontal plane). Based on the usage context corresponding to the position 424, the virtual object 410 may be represented (e.g., displayed by the electronic device 200) as a 3D object (e.g., a 3D representation of an animated robot). In an embodiment, the representation of the virtual object 410 at the position 424 may include different functions than the representation of the virtual object at the position 420. For example, an animated 3D robot on the desktop 442 may be configured to move around the desktop 442 on more than one axis. In addition or alternatively, the animated 3D robot on the table 442 can rotate around its own axis. In addition or alternatively, the animated 3D robot on the table 442 can be configured to have a larger size than when in the position 420.

[0094] Figure 4C It is shown that the virtual object 410 has been moved to the position 426. In various aspects, the virtual object 410 may be moved to the position 426 in response to a user's request (e.g., the user interacts with the virtual object 410 to drag or otherwise cause the virtual object 410 to be moved to the position 426). In this example, at least one usage context associated with the position 426 may be determined. For example, the position 426 is a position on the floor of the CGR environment 290. Note that in this example, the position 426 corresponds to a position on a horizontal plane that is a different type of horizontal plane than that corresponding to the position 424, which is a position on the desktop 442. In this case, it may be determined that the position 426 is associated with a usage context that is satisfied by the position type (e.g., surface type) of the position 426, which is a position on the floor of the CGR environment 290. Based on the usage context corresponding to the position 424, the virtual object 410 may be represented (e.g., displayed by the electronic device 200) as a 3D object (e.g., a 3D representation of an animated robot) on the floor of the CGR environment 290. In an embodiment, the representation of the virtual object at position 426 may be different than the representation of the virtual object at position 424. For example, the animated 3D robot on the floor of the CGR environment 290 may be larger than the animated 3D robot at position 424 on the desktop 442. Furthermore, the animated 3D robot on the floor of the CGR environment 290 may be configured to move at a faster rate than the animated 3D robot at position 424 on the desktop 442.

[0095] In some embodiments, some locations within the CGR environment 290 may not be associated with a usage context for a particular application, or may be prohibited locations relative to a virtual object associated with a particular application. For example, location 422 may be a location on a vertical plane (e.g., a wall) of the CGR environment 290. In this example, location 422 may not have an associated usage context. If the user 202 attempts to move the virtual object 210 to location 422, the movement may not be allowed because, for example, a 3D robot may not be able to navigate on a vertical surface. Alternatively, a default representation of the virtual object (e.g., a 2D image or a 2D application window) may be displayed.

[0096] Figure 5 1 is a flow chart illustrating a method 500 for controlling a representation of a virtual object of a CGR environment based on a usage context associated with a location of the virtual object within the CGR environment. In some embodiments, the method 500 may be performed by the system 100 or a portion of the system 100. In some embodiments, the method 500 may be performed by one or more external systems and / or devices. In some embodiments, the method 500 may be performed by the system 100 (or a portion of the system 100) in conjunction with one or more external systems and / or devices.

[0097] At block 502, the system displays a first representation of a virtual object at a first location within a CGR environment via a display of an electronic device (e.g., a wearable electronic device, an HMD device, etc.). For example, the first representation of the virtual object may be displayed via a first display (e.g., a left-eye display panel) or a second display (e.g., a second-eye display panel) of the electronic device.

[0098] In an embodiment, the first position may correspond to a first use context of a plurality of use contexts. In an embodiment, the plurality of use contexts may include a use context related to a type of surface (e.g., desk, wall, computer screen, floor, etc.) and / or a type of material (e.g., sand, grass, concrete, carpet, etc.) on which the virtual object will be placed, and / or a use context corresponding to how the virtual object will be used (e.g., manipulated, interacted with) or displayed (e.g., presented) in the first position of the CGR environment. In some embodiments, the system may be part of an electronic device, or the electronic device may be part of the system.

[0099] In some embodiments, when the representation of the virtual object is displayed at a first position, the representation of the virtual object may be displayed on a first type of surface (e.g., a desktop, a wall, a computer screen, a floor, etc.), and the representation of the virtual object may be displayed based on the first position (e.g., the type of surface corresponding to the first position). In some embodiments, one or more of the multiple usage contexts may be predefined. For example, one or more of the multiple usage contexts may be predefined based on a specific application corresponding to the virtual object. In some embodiments, the first application may have a first number of predefined usage contexts, and the second application may have a second number of predefined usage contexts that are different from the first number of predefined usage contexts. In some embodiments, the second application may have a different usage context from the first application, and vice versa.

[0100] At block 504, the system receives a request to move a first representation within the CGR environment to a second location different from the first location. In some embodiments, the request may be received or detected by the system based on detecting movement of the first representation from the first location to the second location. In some embodiments, one or more user inputs may be detected, and in response to detecting the user inputs, the system may receive a request to move the representation to the second location. In some embodiments, the request to move the first representation from the first location to the second location may be received based on one or more determinations of an external application, wherein the request to move the first representation from the first location to the second location is received based on the one or more determinations.

[0101] At box 506, in response to receiving the request and based on determining that the second location corresponds to a second usage context in a plurality of usage contexts (e.g., the second usage context is different from the first usage context), the system displays a second representation of the virtual object based on the second usage context and / or based on one or more applications associated with the virtual object via a display of the electronic device at the second location, near the second location, and / or on a surface corresponding to the second location. In an embodiment, the second representation may be different from the first representation. For example, the second representation may have a different size, shape, user interface object, function, audio characteristics, surface material, etc., and / or may be configured to have different and / or additional operations than the first representation.

[0102] In some embodiments, a second usage context among multiple usage contexts may include a usage context that is satisfied when it is determined that the second position corresponds to a position (e.g., a display, screen, surface or housing of an electronic device) on an electronic device (e.g., a computer, laptop, tablet, phone, display, projector display). In some embodiments, based on determining that the second position corresponds to a second usage context among multiple usage contexts, as part of displaying a second representation of the virtual object based on the second usage context, the system displays a 2D representation of the virtual object on the electronic device within the CGR environment. In some embodiments, the second representation of the virtual object may be a 2D representation on the electronic device. In some embodiments, the second representation may be moved (e.g., dragged off the display of the electronic device) to a position in the virtual environment that corresponds to a physical surface in the physical environment. In some embodiments, a 2D application may be manipulated as a 3D application on the electronic device. In some embodiments, a second usage context among multiple usage contexts may include a usage context that is satisfied when it is determined that the second position corresponds to a position on an electronic device (e.g., a computer, laptop, tablet, phone, display, projector display). In these embodiments, based on determining that the second location corresponds to a second usage context in the plurality of usage contexts, displaying a second representation of the virtual object based on the second usage context may include displaying a 3D representation on the electronic device within the CGR environment. In some embodiments, the representation may vary depending on the type of electronic device (e.g., display (e.g., monitor), tablet, personal computer, laptop).

[0103] In some embodiments, a second usage context among multiple usage contexts may include a usage context that is satisfied when it is determined that the second position corresponds to a position on a vertical plane (e.g., a wall, a surface corresponding to a wall-like structure, a side of a building, a bedroom wall, a fence, etc.). In some embodiments, based on determining that the second position corresponds to a second usage context among multiple usage contexts, as part of displaying a second representation of the virtual object based on the second usage context, the system displays a 2D representation on a vertical plane (e.g., on a wall) within the CGR environment. In some embodiments, the second representation of the virtual object may be a 2D representation on an electronic device. In some embodiments, the 2D representation displayed on a vertical plane (e.g., on a wall) within the CGR environment may be larger, may have more visual content, and may include one or more additional (or different) user interface objects compared to the 2D representation displayed on the electronic device. In some embodiments, the representation may vary depending on the type of vertical plane (e.g., the side of a building, a bedroom wall, a fence) and / or one or more characteristics of the vertical plane (e.g., virtual or physical), such as size, shape (e.g., circular, rectangular), material (e.g., brick, wood, metal), texture (e.g., rough, worn), color, opacity, etc.

[0104] In some embodiments, the size of the second representation may be based on the distance between the display of the electronic device and the vertical plane within the CGR environment. In some embodiments, the 2D representation may be smaller when the vertical plane is closer to the display of the electronic device, and larger when the vertical plane is farther away from the display of the electronic device. In some embodiments, after the 2D representation is initially displayed, the size of the 2D representation may be maintained as the user moves farther away from or closer to the 2D representation. In some embodiments, after the 2D representation is initially displayed, the size of the 2D representation may be changed as the user moves farther away from or closer to the 2D representation. In some embodiments, the size of the 2D representation may be based on whether the distance is in a certain category (e.g., a category of distance (e.g., long distance, close distance, average distance), where each category of distance corresponds to a different size representation (e.g., extra large, small, medium).

[0105] In some embodiments, a second usage context among the plurality of usage contexts includes a usage context that is satisfied when it is determined that the second position corresponds to a position on a horizontal plane within the CGR environment (e.g., a desktop, a table, a countertop, a shelf, a floor, an elevated horizontal plane, a horizontal plane above another horizontal plane, a non-elevated horizontal plane, etc.). In some embodiments, based on determining that the second position corresponds to a second usage context among the plurality of usage contexts, as part of displaying a second representation of a virtual object based on the second usage context, the system may display a 3D representation on a horizontal plane within the CGR environment. In some embodiments, the second representation of the virtual object may be a 3D representation on a horizontal plane. In some embodiments, the representation may vary depending on the type of the horizontal plane (e.g., desktop, table, countertop, shelf) and / or one or more features (e.g., virtual or physical) of the horizontal plane, such as size, shape (e.g., circular, rectangular), material (e.g., brick, wood, metal), texture (e.g., rough, worn), color, opacity, etc.

[0106] In some embodiments, based on determining that the horizontal plane is a first type of horizontal plane, the 3D representation may be a representation of a first size. In some embodiments, based on determining that the horizontal plane is a second type of horizontal plane, the 3D representation may be a representation of a second size that is different from (e.g., larger than) the first size. In embodiments, the first type of horizontal plane and the second type of horizontal plane may be selected from horizontal planes that may include the following types: a major horizontal plane, a structure of horizontal planes, a floor, a crosswalk, grass, a lawn, a surface on which one or more people stand, a non-elevated horizontal plane, a horizontal plane below another horizontal plane within a CGR environment, and the like.

[0107] In some embodiments, a 3D representation displayed on a first type of horizontal plane (e.g., a desktop, table, countertop, shelf) within a CGR environment may be larger, may have more visual content, and may include one or more additional (or different) user interface objects than a 3D representation displayed on a second type of horizontal plane (e.g., a floor, a sidewalk, grass, a lawn, a surface on which one or more people are standing).

[0108] In some embodiments, a second usage context among the plurality of usage contexts may include a usage context that is satisfied when a maximized view criterion is satisfied. For example, when a user interface element (e.g., a button, an affordance, and / or any other interactive element) is selected based on a second position (e.g., a position where a virtual object is moved to or dropped), a position on a body part of a user of the device corresponding to the satisfied maximized criterion (e.g., a position on a hand), a gesture, etc., based on a room in which the application may be running, the maximized view criterion may be satisfied. In these embodiments, as part of displaying a second representation of a virtual object based on a second usage context, the system displays multiple representations of the virtual object on multiple planes within the CGR environment. In some embodiments, displaying multiple representations of virtual objects on multiple planes within the CGR environment may include changing one or more aspects of the physical environment and / or the CGR environment to create a fully or partially immersive experience. For example, a room (e.g., physical or virtual) within a CGR environment may be transformed into a virtual auditorium when the application is a presentation application; transformed into a virtual sports venue (e.g., a football stadium) when the application is a sports viewing application (e.g., a fantasy sports application, a live sports application); and transformed into a virtual store when shopping on a shopping application, etc. In some embodiments, the maximized view may be displayed via a companion application (e.g., a fantasy sports application, a live sports application, a shopping application, a presentation application, etc.). In some embodiments, the companion application may correspond to the virtual object and / or may be a companion application to an application corresponding to the virtual object.

[0109] In some embodiments, a selectable virtual object corresponding to the maximized view affordance representation may be displayed (e.g., a selectable virtual object currently displayed with a representation of a virtual object (such as a first representation)). In some embodiments, the maximized view criteria may include criteria that are met when the selectable virtual object corresponding to the maximized view affordance representation is selected (e.g., a tap or swipe on the virtual object).

[0110] In some embodiments, the second position may be determined to correspond to a second usage context in the plurality of usage contexts. In some embodiments, the first representation may include first visual content (e.g., a representation of text, a button, audio / video, a user interface element, etc.). In some embodiments, the second representation may not include the first visual content.

[0111] In some embodiments, the second position may be determined to correspond to a second usage context in a plurality of usage contexts. In some embodiments, the first representation may include third visual content displayed at a third size. In some embodiments, the second representation may include third visual content displayed at a fourth size that is different from the third size (e.g., a larger or smaller representation of text, buttons, audio / video, user interface elements, etc.).

[0112] In some embodiments, it can be determined that the second position corresponds to a second usage context in a plurality of usage contexts. In some embodiments, the first representation may include a first optional object (e.g., one or more optional user interface elements). In some embodiments, the second representation may not include the first optional object.

[0113] In some embodiments, it can be determined that the second position corresponds to a second usage context in a plurality of usage contexts. In some embodiments, the first representation is a fourth size. In some embodiments, the second representation is a fifth size that is different (e.g., larger or smaller) from the fourth size.

[0114] In some embodiments, as part of displaying a second representation of a virtual object based on a second usage context, when the first representation is at a predetermined distance from the second location (e.g., a distance near the second location when the first representation reaches the second location), the system can transition the display of the first representation to the display of the second representation. In some embodiments, when the first representation moves from the first location, the display of the first representation is maintained until the first representation reaches the second location or is within a specific distance from the second location.

[0115] In some embodiments, based on determining that the second position corresponds to a fourth usage context among multiple usage contexts, wherein when the second position corresponds to a prohibited position (e.g., a position prohibited by the application to which the virtual object corresponds and / or one or more other applications and / or the system), the fourth usage context is satisfied, and the system abandons displaying the representation of the virtual object within the CGR environment based on the fourth usage context. In some embodiments, even when the second position corresponds to a position that satisfies the usage context (e.g., the second usage context) but prohibits display of a different usage context, the first representation may continue to remain displayed because display of a representation different from the first representation is prohibited and / or display of a representation corresponding to a usage context (e.g., the second usage context) that would satisfy but prohibit display of a different representation is prohibited.

[0116] In some embodiments, based on determining that the second location corresponds to a fourth usage context of the multiple usage contexts, the system may display within the CGR environment (e.g., a message or symbol displayed to note that the representation corresponding to the fourth usage context cannot be displayed or is prohibited) an indication that the second location is a prohibited location (e.g., a location prohibited by the application corresponding to the virtual object and / or one or more other applications and / or the system).

[0117] At box 508, in response to receiving the request and based on determining that the second location corresponds to a third usage context among multiple usage contexts (e.g., the third usage context is different from the first usage context and the second usage context), the system may display a third representation of the virtual object based on the third usage context (and / or based on one or more applications associated with the virtual object) at the second location (e.g., on a surface corresponding to the second location) via a display of the electronic device, wherein the third representation is different from the first representation and the second representation.

[0118] Aspects of the present disclosure relate to systems and techniques that provide functionality for controlling simultaneous display of representations of virtual objects within a CGR environment. In an embodiment, controlling simultaneous display of representations of virtual objects may include displaying a first representation on a first surface (e.g., a physical or virtual surface) of the CGR environment, and displaying a second representation on a second surface of the CGR environment that is different from the first surface. In an embodiment, a control may be provided for requesting that the second representation of the virtual object be displayed simultaneously with the first representation of the virtual object.

[0119] FIG. 6A to FIG. 6C Exemplary techniques for controlling simultaneous display of representations of virtual objects within a CGR environment in accordance with aspects of the present disclosure are shown. Specifically, Fig. 6A 2 shows a user 202 wearing an electronic device 200, which is configured to allow the user 202 to view a CGR environment 290. As described above, in some embodiments, the electronic device 200 may be similar to the device described above with reference to FIG. Figure 1A and Figure 1B An electronic device 100a is described.

[0120] like Fig. 6A As shown, the CGR environment 290 includes a display 640. As described above, the display 640 can be a physical display or a virtual representation of a display. The first representation 620 of the virtual object 610 can be displayed by the electronic device 200 at the first surface of the CGR environment. For example, the first representation 620 of the virtual object 610 can be displayed on the display 640. Fig. 6AIn the example shown, the first representation 620 is a 2D representation displayed on the display 640. In an embodiment, the first representation 620 can be displayed on any surface (e.g., physical or virtual) within the CGR environment 290. The first representation 620 may include various graphical elements associated with the virtual object. For example, as shown, the virtual object 610 is associated with a calculator application and includes various graphical elements associated with the calculator application. It should be understood that the use of the calculator application to illustrate the virtual object 610 is for illustrative purposes and is not intended to be limiting in any way. Therefore, the virtual object 610 can be associated with any other type of application (e.g., calendar, multimedia application, presentation, etc.).

[0121] In some embodiments, a control for requesting display of a second representation of virtual object 610 may be provided. A user (e.g., user 202) may request simultaneous display, and the request may be received by device 200. The request to display the second representation of virtual object 610 may include a request to display the second representation of virtual object 610 simultaneously with first representation 620. The control for requesting simultaneous display may include any technology for providing a selection (e.g., by user 202). For example, in some embodiments, the control for requesting simultaneous display may include an affordance 611 presented within CGR environment 290. In some embodiments, affordance 611 may be set within first representation 620, or may be set outside first representation 620. In some embodiments, affordance 611 may be a virtual object (e.g., a button, affordance, user interface element, interactive element, etc.) displayed within CGR environment 290 and configured to allow a user (e.g., user 202) to interact. In other embodiments, affordance 611 may be a graphical element (e.g., rather than a virtual element) displayed on a physical display.

[0122] In an embodiment, the control for requesting simultaneous display may include a gesture that may include moving or dragging the virtual object 610 out of the display 640. For example, the user 202 may perform a gesture (e.g., using an appendage, an input sensor, etc.) in which the virtual object 610 may be dragged or moved out of the display 640. The drag gesture may be determined as a request to display the second representation of the virtual object 610 simultaneously with the first representation 620. In some embodiments, the user 202 may drag the virtual object 610 out of the display 640 and may continue to drag the virtual object to a location within the CGR environment 290 where the second representation of the virtual object 610 is to be displayed.

[0123] In some embodiments, in response to receiving a request to simultaneously display representations of virtual object 610, a second representation of virtual object 610 may be displayed within CGR environment 290. In an embodiment, the request to simultaneously display representations of virtual object 610 may result in an animation in which the second representation of virtual object 610 appears (e.g., pops up) with first representation 620. This may be achieved by displaying a second representation of virtual object 610 in CGR environment 290. Figure 6B Shown in.

[0124] Figure 6C 610 is shown displayed within the CGR environment 290 in response to receiving a request to simultaneously display a representation of the virtual object 610. In an embodiment, the second representation 621 can be displayed on any surface (e.g., physical or virtual) within the CGR environment 290. In an embodiment, the second representation 621 can be separate or distinct from the first representation 620. For example, Figure 6C As shown, the first representation 620 can be a 2D representation of the virtual object 610 displayed on the display 640, and the second representation 621 can be a 3D representation of the virtual object 610 displayed outside the display 640 on a second and different surface of the CGR environment 290. In some embodiments, a 2D representation of an object (e.g., an object within a particular application or a particular type of application (e.g., a calculator application or a key fob presentation application, a presentation application, a media or entertainment application, a productivity application) can be displayed simultaneously with the 3D representation of the object. In some embodiments, the 3D representation can be displayed with or without the 3D representation of the particular application or a particular type of application.

[0125] In some embodiments, the first representation 620 and the second representation 621 may provide different or the same functions, although they are associated with the same virtual object. For example, the first representation 620 and the second representation 621 may share a common set of UI elements. In this example, the first representation 620 may be a 2D representation of an application (e.g., a calculator) that includes a set of specific UI elements for user interaction with the application. The second representation 621 may be a 3D representation of an application (e.g., a calculator) that includes the same set of specific UI elements for user interaction as the first representation 620. However, in some embodiments, the first representation 620 and the second representation 621 may have different sets of UI elements. For example, the first representation 620 may include a set of specific UI elements, while the second representation 621 may include a different set of UI elements. In an embodiment, a set of UI elements in different sets of UI elements may include at least one UI element that is not included in another set of UI elements. In other embodiments, different sets of UI elements have no common UI elements. It should be understood that by providing different functions, while displaying the representation of a virtual object provides an improved system, because the system can be configured to adjust the representation of a virtual object (e.g., a 2D representation or a 3D representation) having functions that depend on the type of representation.

[0126] In some embodiments, one representation of a virtual object may be a virtual representation, while another representation of the virtual object may not be a virtual representation. For example, the display 640 may be a physical display, and the first representation 620 may be a graphical representation of a virtual object 610 displayed on the physical display 640. In this case, the first representation 620 may not be a virtual representation because the first representation 620 is actually displayed in the real world on the physical display and is perceived by the user 202 via a transparent or semi-transparent display of the electronic device 200. In this example, the second representation 621 may be a virtual representation of the virtual object 610 because the second representation 621 is not actually displayed in the real world on the physical display, but is displayed on the display of the electronic device 200 and superimposed on the real world physical display. In this way, the user may be provided with the ability to request the display of a 3D representation of a virtual object by interacting with a control provided in a 2D representation of the same virtual object. In some embodiments, both the first representation 620 and the second representation 621 may be virtual representations.

[0127] In an embodiment, the modification of one representation of a virtual object may selectively result in the modification of another representation of the virtual object. For example, when the first representation 620 and the second representation 621 are displayed simultaneously, a request to modify the first representation 620 may be received. In an embodiment, a request to modify the first representation 620 may be received (e.g., from the user 202), such as modifying the size, UI element, shape, theme, etc. In an embodiment, the request to modify the first representation 620 (e.g., user input) may result in a corresponding modification to the second representation 621 (e.g., size, UI element, shape, theme, etc.). In various aspects, the first representation 620 and the second representation 621 may be modified according to the modification request. In some embodiments, whenever a modification to the first representation 621 is requested, the second representation 621 is modified accordingly. In other embodiments, the first request to modify the first representation 621 may result in a corresponding modification to the second representation 621. However, the second request to modify the first representation 621 does not result in a corresponding modification to the second representation 621. In this case, when the second request to modify the first representation 620 is received, the modification to the second representation 621 is abandoned. Note that although the foregoing discussion describes selectively modifying the second representation 621 based on a request to modify the first representation 620, this is for illustrative purposes and not limiting. Therefore, the same techniques may be used to selectively modify the first representation 620 based on a request to modify the second representation 621.

[0128] Figure 7 is a flow chart illustrating a method 700 for controlling simultaneous display of representations of virtual objects within a CGR environment. In some embodiments, the method 700 may be performed by the system 100 or a portion of the system 100. In some embodiments, the method 700 may be performed by one or more external systems and / or devices. In some embodiments, the method 700 may be performed by the system 100 (or a portion of the system 100) in conjunction with one or more external systems and / or devices.

[0129] At box 702, the system displays a 2D representation of a virtual object at a first surface (and / or location) of a CGR environment via a display of an electronic device (e.g., a wearable electronic device, an HMD device, etc.). For example, a first representation of a virtual object may be displayed on a representation of a display within a CGR environment via a first display (e.g., a left-eye display panel) or a second display (e.g., a second-eye display panel) of the electronic device. In some embodiments, the first surface may be a virtual surface within the CGR environment. For example, the first surface may be a virtual representation of a physical display. In other embodiments, the first surface may be a real-world physical surface of the CGR environment. For example, the first surface may be a surface of a physical display. The 2D representation of the virtual object may be a virtual representation (e.g., a virtual representation superimposed on the first surface via a semi-transparent display of an electronic device) or may be a real-world graphical representation (e.g., a real-world graphical representation displayed on a real-world physical display).

[0130] In some embodiments, the 2D representation of the virtual object may include a set of UI elements for user interaction with the virtual object. In an embodiment, the 2D representation of the virtual object may also include at least one control for requesting a second representation of the virtual object to be displayed simultaneously.

[0131] At block 704, the system receives a request to display the 3D representation of the virtual object simultaneously with the 2D representation. In an embodiment, the request for simultaneous display may include user input. The request may be input by the user using a control element (e.g., a button, an affordance, a user interface element, an interactive element, etc.) displayed with the 2D representation (e.g., within the 2D representation or outside the 2D representation). For example, the user may select the control element, and the selection may cause the system to receive the request for simultaneous display.

[0132] In some embodiments, the request to display the 2D representation and the 3D representation simultaneously may include a gesture to move or drag the 2D representation off the first surface. For example, the user 202 may grab, click, and / or otherwise select (e.g., using an appendage, an input device, an input sensor, etc.) a 2D representation displayed at the first surface, and may move or drag the 2D representation away from the first surface. In some aspects, the drag gesture may be determined as a request for simultaneous display.

[0133] In an embodiment, a request to display a 3D representation of a virtual object simultaneously with a 2D representation may cause an animation to be played, wherein the 3D representation is configured to come out (or pop up) from the 2D representation. In an embodiment, the animation may include sounds that may be played during the animation.

[0134] At block 706, in response to the request for simultaneous display, the system simultaneously displays a 2D representation at a first surface and a 3D representation at a second surface of the CGR environment via a display of the electronic device. In some embodiments, the second surface may be different from the first surface. In an embodiment, the second surface may be a virtual surface, or may be a real-world physical surface within the CGR environment. For example, the second surface may be a physical real-world surface of a desk, or may be a virtual representation of a surface of a physical desk.

[0135] In an embodiment, the second surface on which the 3D representation can be displayed can be determined by user input. For example, a user can drag a 2D representation from a first surface and continue to drag it to a second surface. In this way, the 3D representation can be displayed in any surface within the CGR environment where the drag gesture stops. In other specific implementations, for example, in the case where a control element in the 2D representation is used to request simultaneous display, the second surface can be predetermined. In some specific implementations, the user can indicate the surface on which the 3D representation is to be displayed before requesting simultaneous display. For example, the user can first indicate (e.g., via user input (e.g., using user input detected by an input sensor that may include a mouse, a stylus, a touch-sensitive surface, an image sensor (e.g., to perform hand tracking)) a ​​surface other than the first surface within the CGR environment. When requesting simultaneous display, the 3D representation can be displayed at the surface indicated by the user.

[0136] In some embodiments, the 3D representation of the virtual object may include a set of UI elements for user interaction. In an embodiment, the set of UI elements represented in 3D may be different from the set of UI elements represented in 2D. For example, one set of UI elements may include UI elements that are not included in another set of UI elements.

[0137] Aspects of the present disclosure relate to the following systems and techniques, which provide functionality for controlling the representation of virtual objects based on characteristics of an input mechanism. In an embodiment, the representation of a virtual object may be based on the characteristics of the input mechanism relative to the virtual object (e.g., the direction of movement of the input mechanism, the distance, the type of gesture, etc.). For example, in an embodiment, the representation of the virtual object may be modified or maintained based on whether the input mechanism associated with the virtual object is within a predetermined distance from the first representation of the virtual object. In other embodiments, for example, the representation of the virtual object may be modified or maintained based on determining whether the input mechanism associated with the virtual object moves toward or away from the first representation of the virtual object. In other embodiments, for example, the representation of the virtual object may be modified or maintained based on whether a gesture associated with the input mechanism is determined to indicate the possibility of the user interacting with the first representation of the virtual object. It should be understood that the functionality provided by the systems and techniques described herein provides an advantageous system in which the representation of a virtual object may be adapted to the characteristics of the input mechanism, thereby providing an improved user interface.

[0138] Fig. 8A and Figure 8B Exemplary techniques for controlling the representation of virtual objects within a CGR environment based on characteristics of an input mechanism in accordance with various aspects of the present disclosure are shown. Specifically, Fig. 8A 800 and a virtual object 810. In an embodiment, the CGR environment 890 can be presented to a user (e.g., user 202) wearing an electronic device (e.g., electronic device 200) configured to allow the user 202 to view the CGR environment 890. As described above, in some embodiments, the electronic device 200 can be similar to the above referenced Figure 1A and Figure 1B An electronic device 100a is described.

[0139] like Fig. 8A As shown, a first representation 810 of a virtual object may be displayed by the electronic device 200. In an embodiment, the first representation 810 may be a 3D representation of the virtual object, and the virtual object may be associated with a specific application. Fig. 8A As shown, the first representation 810 can be associated with a calculator application. It should be understood that the use of a specific application (e.g., a calculator application) to illustrate the first representation 810 and other representations of virtual objects is for illustrative purposes and is not intended to be limiting in any way. Therefore, the first representation 810 can be associated with any type of application (e.g., a calendar, a multimedia application, a presentation, etc.).

[0140] In an embodiment, the first representation 810 may be configured to facilitate indirect interaction between the user and the first representation 810. As used herein, indirect interaction may refer to user interaction with a representation of a virtual object without directly manipulating elements of the representation of the virtual object. A non-limiting example of indirect interaction may be a user perceiving information provided by a user interface (UI) element of the representation of the virtual object without the user directly manipulating the UI element. In contrast, as used herein, direct interaction may refer to user interaction with a representation of a virtual object, wherein UI elements of the representation of the virtual object representation may be directly manipulated by the user. For example, a user may press a button, may interact with an interactive element, may click on an optional item and / or an affordance, etc.

[0141] The first representation 810 may include UI elements 811 and 815. In an embodiment, UI element 815 may represent at least one UI element configured to provide (e.g., output) information associated with a virtual object represented by the first representation 810. For example, UI element 815 may be a display of the first representation 810. Therefore, UI element 815 may be configured for indirect interaction so that the user can perceive the output without directly manipulating UI element 815. UI element 811 may represent at least one UI element that may be configured to facilitate user interaction (e.g., direct interaction or indirect interaction). For example, UI element 811 may be a button, an affordance, a user interface element, an interactive element, etc. and / or any combination thereof. When UI element 811 is configured to facilitate direct interaction, the user may select, click, select and / or otherwise manipulate UI element 811. In some embodiments, UI element 811 may be configured to facilitate indirect interaction by displaying the UI element as a 3D element. In this case, the user may perceive UI element 811 as a 3D element.

[0142] In an embodiment, the input mechanism 800 may include a mechanism configured to facilitate interaction with a representation of a virtual object. For example, the input mechanism may include a mechanism for a user (e.g., user 202) to manipulate at least one element of the representation of a virtual object or perceive data provided by an element of the representation of a virtual object. In an embodiment, the input mechanism 800 may include a representation of a user's appendage (e.g., a finger, hand, leg, foot, etc.), a user's gaze (e.g., head gaze, eye gaze, etc.), an input device (e.g., a mouse, a stylus, etc.) (e.g., different from an electronic device, operably communicating with an electronic device, physically connected to an electronic device (e.g., a part of an electronic device)), etc. In an embodiment, the representation of the user's appendage may include a virtual representation of the appendage and / or may include data representing features of the appendage within a CGR environment (e.g., location, orientation, distance from a particular point, etc.). In various aspects, the input mechanism 800 may be detected using an input sensor (e.g., a touch-sensitive surface, an image sensor, etc.) configured to perform hand tracking, head gaze tracking, eye gaze tracking, finger tracking, etc. Fig. 8A As shown, the input mechanism may include an appendage (eg, a finger) of the user.

[0143] like Fig. 8A As shown and described above, the first representation 810 can be displayed within the CGR environment 890, and the first representation 810 can be configured to facilitate indirect interaction rather than direct interaction with the user (e.g., by providing UI elements 811 and 815 configured for indirect interaction). Fig. 8AAs shown, the input mechanism 800 may be located at a current position that is a distance 831 from the first representation 810. In some embodiments, a predetermined distance 830 from the first representation 810 may be provided, but in some implementations, the predetermined distance 830 may not be shown within the CGR environment 890. The predetermined distance 830 may be configured to operate as a threshold, such that when the current position of the input mechanism is not within the predetermined distance 830 from the first representation 810, the display of the first representation 810 may be maintained. For example, since the distance 831 may be determined to be greater than the predetermined distance 830, the current position of the input mechanism 800 may be determined to be not within the predetermined distance 830 from the first representation 810.

[0144] In an embodiment, whether the display of the first representation 810 can be modified or maintained can be based on the characteristics of the input mechanism 800. In some embodiments, the characteristics of the input mechanism 800 can include the direction of movement, the distance to the representation of the virtual object, the gesture type, etc. Based on determining that the current position of the input mechanism 800 is not within the predetermined distance 830 from the first representation 810, the display of the first representation 810 can be maintained without displaying another representation of the virtual object. In contrast, as described below, and as Figure 8B As shown in the illustrated example, based on determining that the current position of the input mechanism 800 is within a predetermined distance 830 from the first representation 810, the display of the first representation 810 can be modified and a second representation of the virtual object can be displayed. In various aspects, the second representation of the virtual object can be different from the first representation 810.

[0145] In some embodiments, determining whether the position of the input mechanism 800 is within the predetermined distance 830 from the first representation 810 may be performed in response to detecting movement of the input mechanism 800. In these cases, if movement of the input mechanism 800 is not detected, the determination of whether the position of the input mechanism 800 is within the predetermined distance 830 from the first representation 810 may not be performed. In some embodiments, the determination of whether the position of the input mechanism 800 is within the predetermined distance 830 from the first representation 810 may be performed when the detected movement is determined to be toward the first representation 810. In these cases, if the movement of the input mechanism 800 is determined to be away from the first representation 810, the determination of whether the position of the input mechanism 800 is within the predetermined distance 830 from the first representation 810 may not be performed even though movement of the input mechanism 800 may be detected.

[0146] In some implementations, in response to determining that the input mechanism 800 is not within a predetermined distance 830 from the location where the first representation 810 is to be displayed, the first representation 810 may be initially displayed within the CGR environment 890. For example, it may be determined that a representation of a virtual object is initially displayed at a first location within the CGR environment 890. In this example, the first representation of the virtual object may be configured for indirect interaction. Further, in this example, the CGR environment 890 may not include any representation of the virtual object at the first location, but in some cases, at least one other representation of the virtual object may be displayed at another location within the CGR environment 890. In response to determining that a representation of the virtual object is initially displayed at a first location within the CGR environment 890, it may be determined whether the current location of the input mechanism 800 is within the predetermined distance 830 from the first location. If it is determined that the current location of the input mechanism 800 is not within the predetermined distance 830 from the first location, the first representation (e.g., the first representation 810) may be displayed at the first location. In some embodiments, if the current position of the input mechanism 800 is determined to be within a predetermined distance 830 from the first position, a second representation configured for direct interaction (eg, second representation 820 described below) may be displayed at the first position.

[0147] like Figure 8B As shown, the input mechanism 800 can be moved from a previous position (e.g., Fig. 8A 830 to the first representation 810. The input mechanism 800 may be moved (e.g., in direction 833) to a current position having a distance 832 to the first representation 810. The movement from the previous position to the current position may be detected (e.g., using an input sensor as described above). In response to detecting the movement of the input mechanism 800 from the previous position to the current position, it may be determined whether the current position of the input mechanism 800 to the first representation 810 may be within a predetermined distance 830. For example, the distance 832 from the current position of the input mechanism 800 to the first representation 810 may be compared to the predetermined distance 830. Based on determining that the distance 832 is greater than the predetermined distance 830, it may be determined that the current position of the input mechanism 800 is not within the predetermined distance 830 from the first representation 810. Conversely, based on determining that the distance 832 is not greater than the predetermined distance 830, it may be determined that the current position of the input mechanism 800 is within the predetermined distance 830 from the first representation 810.

[0148] In an embodiment, based on determining that the current position of the input mechanism 800 is within the predetermined distance 830 from the first representation 810, the display of the first representation 810 can be modified. In an embodiment, modifying the display of the first representation 810 can include ceasing to display the first representation 810 and displaying a second representation 820, wherein the second representation 820 can be different from the first representation 810. In some embodiments, the second representation 820 can be displayed at the same location and / or on the same surface where the first representation 810 is displayed.

[0149] In an embodiment, the second representation 820 can be configured for direct interaction between a user (e.g., user 202) and the second representation 820 (e.g., elements of the second representation 820). Fig. 8A As shown, while the first representation 810 includes a UI element 811 configured for non-direct interaction (e.g., a UI element displayed as a protruding 3D UI element), the second representation 820 may include a UI element 821 configured for direct interaction. In this example, the UI element 821 may include at least one UI element displayed as a flat button or a 2D element, wherein the flat button may not protrude from the second representation 820. It should be understood that a flat 2D UI element (e.g., a 2D button) displayed on a physical table (e.g., on the same plane as the physical table) may be easier to provide physical feedback when a user manipulates the 2D element. For example, when a user manipulates a 2D element, the user receives feedback provided by the physical table on which the virtual 2D element 1 is displayed. In addition, displaying the second representation 820 configured for direct interaction may also encourage a user (e.g., user 202) to interact with the second representation 820.

[0150] In some embodiments, modifying the first representation 810 (which may include displaying the second representation 820) may include displaying the modification in an animated manner. For example, one of the differences between the first representation 810 and the second representation 820 may be that the UI element 811 of the first representation 810 is displayed as a protruding 3D UI element, and the UI element 821 of the second representation 820 is displayed as a flat 2D UI element. In this example, the modification of the first representation 810 may include displaying the UI element in an animated manner so that the protruding 3D UI element of the first representation 810 is presented as receding into the flat 2D UI element of the second representation 820. In an embodiment, the animation may also include a sound that may be played while the animation occurs.

[0151] In another embodiment, modifying the first representation of the virtual object may include moving the first representation to a position closer to the user (e.g., user 202). For example, based on a feature of the input mechanism 800 (e.g., the current position of the input mechanism 800 is within a predetermined distance (e.g., predetermined distance 830) from the current position of the first representation (e.g., first representation 810)), a second representation of the virtual object may be displayed. In an embodiment, the second representation of the virtual object may be the same as the first representation, but in a position closer to the user than the current position of the first representation. In some embodiments, for example, according to the above description, the second representation displayed at the new position may be a different representation of the first representation.

[0152] In other embodiments, determining the characteristics of the input mechanism based on which the modification or maintenance of the first representation 810 may be based may include determining whether the direction of movement of the input mechanism 800 is toward or away from the first representation 810. For example, Figure 8B As shown, the input mechanism 800 can move in direction 833, which is a direction toward the first representation 810. In this case, based on determining that the movement direction of the input mechanism 800 is toward the first representation 810, the display of the first representation 810 can be modified, and a second representation of the virtual object (e.g., a second representation 820 configured to facilitate direct interaction with the user) can be displayed. Conversely, based on determining that the movement direction of the input mechanism 800 is away from the first representation 810, the display of the first representation 810 can be maintained, and another representation of the virtual object (e.g., the second representation 820) is not displayed. In various aspects, the second representation of the virtual object can be different from the first representation 810.

[0153] In other embodiments, determining the characteristics of the input mechanism on which to modify or maintain the first representation 810 may include determining whether the input mechanism 800 has made a particular type of gesture. In various aspects, the particular type of gesture may be a gesture that may indicate the possibility for direct user interaction. For example, Figure 8B As shown, the input mechanism 800 can be a pointing hand. In an embodiment, the pointing hand can be considered to be the type of gesture indicating the possibility of user interaction. It should be understood that a user who wishes to use a finger to interact with a virtual object (such as a virtual object for user input represented by a UI element) can do so by making his or her hand form a pointing hand with the finger pointing outward. In this sense, the pointing hand can indicate that the user intends or expects to interact with the virtual object. Therefore, when it is determined that the input mechanism has made a gesture indicating the possibility of user interaction (e.g., a pointing hand, a grabbing hand, etc.), it can be determined that the current representation (e.g., the first representation 810) configured for indirect interaction is modified to a representation (e.g., the second representation 820) configured for direct interaction. In various aspects, the current representation configured for indirect interaction is modified to a representation configured for direct interaction according to the foregoing description.

[0154] In another example, the determination to keep displaying the first representation 810 configured for non-direct interaction may be based on a gesture that does not indicate the possibility of user interaction. For example, a gesture may be detected that may include a user (e.g., user 202) crossing his or her arms and / or leaning back. In this case, the gesture may be considered to be a gesture type that does not indicate the possibility of user interaction. Therefore, when it is determined that the user has crossed his or her arms and / or has leaned back, it may be determined to keep the current representation configured for non-direct interaction (e.g., the first representation 810) without displaying the representation configured for direct interaction (e.g., the second representation 820). In some embodiments, detecting a gesture that does not indicate the possibility of user interaction may result in a determination that the current representation configured for direct interaction (e.g., the second representation 820) is modified to be configured for non-direct interaction (e.g., the first representation 810).

[0155] It should be noted that although the foregoing examples and subsequent examples may focus on the description of the modification of the representation of a virtual object configured for indirect interaction to the representation of a virtual object configured for direct interaction, this is for illustrative purposes and is not intended to be limiting in any way. In some embodiments, the representation of a virtual object configured for direct interaction may be modified to a representation of a virtual object configured for indirect interaction based on the characteristics of the input mechanism. For example, in some specific implementations, based on the movement of the detected input mechanism, based on the characteristics of the input mechanism (for example, based on determining that the position of the input mechanism is not within a predetermined distance from the representation configured for direct interaction (for example, the first representation 810)), the display of the representation configured for direct interaction (for example, the above-mentioned first representation 810) may be modified to display the representation configured for indirect interaction (for example, the above-mentioned second representation 820). Therefore, the present disclosure provides a technology for selectively and dynamically configuring the representation of virtual object enhanced interaction (for example, direct or indirect) based on the characteristics of the input mechanism. Therefore, when the characteristics based on the input mechanism are more advantageous, the representation of the virtual object can be configured for direct or indirect interaction.

[0156] In addition, although the foregoing discussion describes the second representation 820 as being configured to interact directly with a flat 2D UI element, it should be understood that this is for illustrative purposes rather than as a limitation. It should be understood that the representation of a virtual object may be configured to interact directly by other methods (e.g., orientation, size, angle, shape, color, brightness, language, position, distance, direction, etc.). For example, in an embodiment, based on the characteristics of an input mechanism (e.g., based on determining that the current position of the input mechanism is within a predetermined distance from the first representation of a virtual object), the display of the first representation may be modified, and the modification may include displaying a second representation different from the first representation. In these embodiments, the second representation may include an orientation, size, angle, shape, color, brightness, language, position, distance, direction, etc. different from the first representation, wherein the modification may be configured to allow, encourage, enable and / or otherwise promote direct interaction with the second representation of a virtual object. Some embodiments in these embodiments will be described in more detail below.

[0157] Fig.9A and Fig. 9B Another example of a technique for controlling the representation of a virtual object within a CGR environment based on characteristics of an input mechanism in accordance with aspects of the present disclosure is shown. Fig.9A As shown, the first representation 910 of the virtual object can be displayed via the display of the electronic device 200. In an embodiment, the first representation 910 can be a 3D representation of the virtual object, and the virtual object can be associated with a specific application program (e.g., a calendar, a multimedia application, a presentation, etc.), as described above. Fig.9A In the example shown, the first representation 910 may be associated with a calculator application.

[0158] In an embodiment, the first representation 910 may be configured to facilitate indirect interaction with an associated virtual object. For example, the first representation 910 may include UI elements 911 and 915. In an embodiment, UI element 915 may represent at least one UI element configured to provide (e.g., output) information associated with a virtual object represented by the first representation 910. For example, UI element 915 may be a display (e.g., a virtual display) of the first representation 910. In this case, the first representation 910 may be configured to facilitate indirect interaction with the user by displaying it in an orientation that facilitates the user (e.g., user 202) to interact indirect with the UI element 915. For example, the first representation 910 may include an orientation display of angle 912. In an embodiment, angle 912 may be an angle configured to place the first representation 910 in an orientation that enables the user to see, hear, or otherwise perceive the UI element 915. In this way, angle 912 facilitates indirect interaction between the user and the UI element 915. In an embodiment, angle 912 may be measured relative to a surface (e.g., surface 916) on which the first representation 910 is displayed.

[0159] In an embodiment, the orientation at which the first representation 910 may be displayed may be determined based on the position of the user. For example, the user's gaze (e.g., head gaze and / or eye gaze) may be determined (e.g., by detecting the position of the user's head and / or eyes and then determining the user's gaze), and the determined user's gaze may then be used to determine the orientation at which the first representation 910 is to be displayed such that a UI element configured for indirect interaction (e.g., UI element 915) faces the user's gaze.

[0160] In an embodiment, the UI element 911 of the first representation 910 may be configured for indirect interaction. In this case, the UI element 911 may be displayed as a protruding button, or as a 3D element, where the flat button may not protrude from the first representation 910. In this way, the UI element 911 (e.g. Fig.9A shown) are not configured for direct interaction.

[0161] like Fig.9A As shown, and as described above, the first representation 910 can be configured to facilitate indirect interaction rather than direct interaction by the user (e.g., by providing a prominent 3D UI element 911 and by orienting the first representation 910 at an angle 912). Fig.9A As shown, the input mechanism 800 may be located at a current position that is a distance 931 from the first representation 910. In some embodiments, a predetermined distance 930 from the first representation 910 may be provided.

[0162] In an embodiment, based on determining that the current position of the input mechanism 800 is not within the predetermined distance 930 from the first representation 910, the display of the first representation 910 can be maintained. For example, the first representation 910 configured for non-direct interaction can continue to be displayed without displaying another representation of the virtual object and / or without changing the first representation 910. Conversely, as described below, based on determining that the current position of the input mechanism 800 is within the predetermined distance 930 from the first representation 910, the display of the first representation 910 can be modified and a second representation of the virtual object can be displayed. In various aspects, the second representation of the virtual object can be different from the first representation 910.

[0163] like Fig. 9B As shown, the input mechanism 800 can be moved from a previous position (e.g., Fig.9A 930 . The input mechanism 800 may be moved (e.g., in direction 933) to a current position having a distance 932 to the first representation 910. The movement from the previous position to the current position may be detected (e.g., using an input sensor as described above). In response to detecting the movement of the input mechanism 800 from the previous position to the current position, it may be determined whether the current position of the input mechanism 800 to the first representation 910 may be within a predetermined distance 930. For example, the distance 932 from the current position of the input mechanism 800 to the first representation 910 may be compared to the predetermined distance 930. Based on determining that the distance 932 is greater than the predetermined distance 930, it may be determined that the current position of the input mechanism 800 is not within the predetermined distance 930 from the first representation 910. Conversely, based on determining that the distance 932 is not greater than the predetermined distance 930, it may be determined that the current position of the input mechanism 800 is within the predetermined distance 930 from the first representation 910.

[0164] In an embodiment, based on determining that the current position of the input mechanism 800 is within the predetermined distance 930 from the first representation 910, the display of the first representation 910 can be modified. In an embodiment, modifying the display of the first representation 910 can include ceasing to display the first representation 910 and displaying a second representation 920, wherein the second representation 920 can be different from the first representation 910. In some embodiments, the second representation 920 can be displayed at the same location and / or on the same surface where the first representation 910 is displayed.

[0165] In an embodiment, the second representation 920 may be configured to facilitate direct interaction of a user (e.g., user 202) with an associated virtual object. For example, while the first representation 910 is displayed with an orientation having an angle 912, which facilitates the user to perceive (e.g., see, hear, etc.) information provided by the UI element 915 (e.g., indirect interaction), the second representation 920 may be displayed in an orientation that facilitates the user to directly interact (e.g., directly manipulate, select, click, drag, and / or otherwise select) with the UI element (e.g., UI element 921) of the second representation 920. For example, the second representation 920 may be displayed within the CGR environment 890 with an orientation that is longitudinal to the surface 916. Therefore, the second representation 920 may be displayed as being flat on the surface 916. It should be understood that a flat surface may be easier to interact with than an angled surface. Therefore, by modifying the representation of the virtual object from an angled orientation to a flat orientation, or vice versa, the representation of the virtual object is selectively adapted to enhance direct interaction based on the characteristics of the input mechanism. In some embodiments, second representation 920 may be displayed at an orientation with respect to surface 916 having a non-zero angle different than angle 912 .

[0166] Furthermore, although the first representation 910 includes a UI element 911 configured for indirect interaction, such as Fig.9A 910), but second representation 920 may include UI elements 921 configured for direct interaction, as previously described. For example, UI element 921 may include at least one UI element displayed as a flat 2D UI element displayed on a physical object, which facilitates physical feedback when a user manipulates the 2D UI element.

[0167] In some embodiments, modifying the first representation 910 (which may include displaying the second representation 920) may include animating the modification. For example, the modification of the first representation 910 may include animating a change in the orientation of the first representation 910, such that the first representation 910 is displayed as moving from a current orientation (e.g., angled at angle 912) to an orientation of the second representation 920 (e.g., flat on surface 916). Additionally or alternatively, the modification of the first representation 910 may include animating the UI elements such that a protruding 3D UI element of the first representation 910 is rendered as receding into a flat 2D UI element of the second representation 920. In an embodiment, the animation may also include a sound that may be played while the animation occurs.

[0168] Fig. 10A and Fig. 10BAnother example of a technique for controlling the representation of a virtual object within a CGR environment based on characteristics of an input mechanism in accordance with aspects of the present disclosure is shown. Specifically, Fig. 10A and Fig. 10B An example is shown in which a representation of a virtual object is modified based on characteristics of an input mechanism, and wherein the modification includes adding UI elements for user interaction and changing a size of the representation.

[0169] like Fig. 10A As shown, the first representation 1010 of the virtual object can be displayed via the display of the electronic device 200. In an embodiment, the first representation 1010 can be a 3D representation of the virtual object, and the virtual object can be associated with a specific application program (e.g., a calendar, a multimedia application, a presentation, etc.), as described above. Fig. 10A In the example shown, the first representation 1010 may be associated with a calculator application.

[0170] In an embodiment, the first representation 1010 may be configured to facilitate indirect interaction with an associated virtual object. For example, the first representation 910 may include a UI element 1012. The UI element 1012 may represent at least one UI element configured to provide (e.g., output) information associated with a virtual object represented by the first representation 1010. For example, the UI element 1012 may be a display (e.g., a virtual display) of the first representation 1010. In some embodiments, the first representation 1010 may have a size. In some embodiments, the first representation 1010 may not include any UI elements (e.g., buttons, affordances, user interface elements, interactive elements, etc.) configured for user input.

[0171] like Fig. 10A As shown and described above, the first representation 1010 can be displayed within the CGR environment 890, and the first representation 1010 can be configured to facilitate indirect interaction rather than direct interaction with the user. Fig. 10A As shown, the input mechanism 800 may be located at a current position that is a distance 1031 from the first representation 1010. In some embodiments, a predetermined distance 1030 from the first representation 1010 may be provided.

[0172] In an embodiment, based on determining that the current position of the input mechanism 800 is not within the predetermined distance 1030 from the first representation 1010, the display of the first representation 1010 can be maintained. For example, the first representation 1010 configured for non-direct interaction can continue to be displayed without displaying another representation of the virtual object and / or without changing the first representation 1010. Conversely, as described below, based on determining that the current position of the input mechanism 800 is within the predetermined distance 1030 from the first representation 1010, the display of the first representation 1010 can be modified and a second representation of the virtual object can be displayed. In various aspects, the second representation of the virtual object can be different from the first representation 1010.

[0173] like Fig. 10B As shown, the input mechanism 800 can be moved from a previous position (e.g., Fig.9A The input mechanism 800 may be moved to a current position having a distance 1030 to the first representation 1010 (as shown). The movement from the previous position to the current position may be detected (e.g., using an input sensor as described above). In response to detecting the movement of the input mechanism 800 from the previous position to the current position, it may be determined whether the current position of the input mechanism 800 to the first representation 1010 is within the predetermined distance 1030. Based on determining that the current position of the input mechanism 800 is within the predetermined distance 1030 from the first representation 1010, the display of the first representation 1010 may be modified. In an embodiment, modifying the display of the first representation 1010 may include ceasing to display the first representation 1010 and displaying a second representation 1020, wherein the second representation 1020 may be different from the first representation 1010. In some embodiments, the second representation 1020 may be displayed at the same location and / or on the same surface where the first representation 1020 is displayed.

[0174] In an embodiment, the second representation 1020 may be configured to facilitate direct interaction of a user (e.g., user 202) with an associated virtual object. For example, while the first representation 1010 may not include UI element 911 configured for user input, the second representation 1020 may include UI element 1021 configured for user interaction, as described above. For example, UI element 1021 may include at least one UI element displayed as a flat 2D UI element.

[0175] Additionally, the second representation 1020 can be displayed to have a size that is different from the size of the first representation 1010. For example, the second representation 1020 can be displayed at a size that is larger than the size of the first representation 1010. In some embodiments, the second representation 1020 can be displayed at a size that is smaller than the size of the first representation 1010.

[0176] As previously described, in some embodiments, modifying the first representation 1010 (which may include displaying the second representation 1020) may include animating the modification. For example, the modification of the first representation 1010 may include animating the size change of the first representation 1010, so that the first representation 1010 is displayed as growing or shrinking from a current size to the size of the second representation 1020, as appropriate. In addition or alternatively, the modification of the first representation 1010 may include animating the UI elements, so that a protruding 3D UI element of the first representation 910 is presented as receding into a flat 2D UI element of the second representation 920. In an embodiment, the animation may also include a sound that may be played while the animation occurs.

[0177] Fig.11A and Fig. 11B Another example of a technique for controlling the representation of a virtual object within a CGR environment based on characteristics of an input mechanism in accordance with aspects of the present disclosure is shown. Specifically, Fig.11A and Fig. 11B Examples are shown of modifying the representation of a virtual object based on characteristics of an input mechanism (eg, a user's gaze).

[0178] like Fig.11A As shown, the first representation 1110 of the virtual object can be displayed via the display of the electronic device 200. In an embodiment, the first representation 1110 can be a representation of the virtual object, and the virtual object can be associated with a specific application program (e.g., a calendar, a multimedia application, a presentation, etc.), as described above. Fig.11A In the example shown, the first representation 1110 may be associated with a calendar application.

[0179] In an embodiment, the first representation 1110 may have a size and may be displayed at position 1152. In an embodiment, the first representation 1110 may not be configured for user interaction, whether direct or indirect. For example, the size of the first representation 1110 may be a small size, and the small size may not enable the user to perceive any information from or interact with any UI element of the first representation 1110. In some embodiments, the first representation 1110 may not include any UI elements.

[0180] like Fig.11AAs shown, a gaze 1150 of a user 202 of the wearable electronic device 200 may be detected. In various aspects, the detected gaze 1150 may be a head gaze (e.g., a direction in which the user's head is facing), an eye gaze (e.g., a direction in which the user's eyes are looking), a combination thereof, or the like. The gaze 1150 of the user 202 may be determined to be focused, placed, or otherwise directed to a location 1151, which may be different from a location 1152 at which the first representation 1110 is displayed. In various aspects, based on determining that the gaze 1150 is directed to a location different from the location of the first representation 1110, the display of the first representation 1110 at the current location and with the size may be maintained without displaying another representation of the virtual object and / or without making any changes to the first representation 1110.

[0181] Fig. 11B It is shown that the gaze 1150 of the user 202 has changed to a direction different from the direction pointed to the location 1151. In an embodiment, the change in gaze can be detected (e.g., via an input sensor). In response to the detected change in the user's gaze, the direction of the new direction of the gaze can be determined. For example, it can be determined that the new direction of the gaze 1150 can point to the location 1152. The location 1152 can be the location where the first representation 1110 is displayed. In an embodiment, based on determining that the gaze 1150 is pointing to the same location as the location of the first representation 1110, the display of the first representation 1110 can be modified.

[0182] In some embodiments, determining to modify the display of the first representation 1110 based on determining that the gaze 1150 is directed at the same location as the location of the first representation 1110 may include determining that the gaze 1150 has remained directed at the same location as the location of the first representation 1110 for at least a predetermined period of time. When it is determined that the gaze 1150 has remained directed at the same location as the location of the first representation 1110 for a period of time that is less than the predetermined period of time (e.g., the direction of the gaze 1150 moves to a different direction before the predetermined period of time expires), the display of the first representation 1110 may not be modified, but may be maintained without displaying another representation of the virtual object and / or without making any changes to the first representation 1110. When it is determined that the gaze 1150 remains directed at the same location as the location of the first representation 1110 for at least the same period of time as the predetermined period of time (e.g., the direction of the gaze 1150 does not move to a different direction before the predetermined period of time expires), the display of the first representation 1110 may be modified.

[0183] In an embodiment, modifying the display of the first representation 1110 may include ceasing to display the first representation 1110 and displaying the second representation 1120, wherein the second representation 1120 may be different from the first representation 1110. In some embodiments, the second representation 1120 may be displayed at the same location and / or on the same surface where the first representation 1120 is displayed.

[0184] In an embodiment, the second representation 1120 may be different from the first representation 1110, and the second representation 1120 may be configured to facilitate interaction by a user (e.g., user 202). For example, the second representation 1120 may be configured to include a UI element 1112. The UI element 1112 may include at least one UI element configured for user interaction, such as a display. In some embodiments, the second representation 1120 may alternatively or additionally have a size different from the size of the first representation 1110. For example, the size of the second representation 1120 may be larger or smaller than the size of the first representation 1110. In an embodiment, the size of the second representation 1120 may be based on the distance between the position of the second representation 1120 (e.g., position 1152) and the position of the user's head and / or eyes (e.g., position 1153).

[0185] In some embodiments, second representation 1120 may be configured for indirect interaction, but may not be configured for direct interaction. For example, second representation 1120 may not include any UI elements (e.g., buttons, affordances, user interface elements, interactive elements, etc.) configured for direct interaction with a user. In this case, the above description with respect to Fig. 8A , Figure 8B , Fig.9A , Fig. 9B , Fig. 10A and Fig. 10B The described techniques may be used to selectively modify the second representation 1120 to a configuration for direct interaction based on a characteristic of an input mechanism according to the disclosure (e.g., a representation of an appendage, a mouse, a stylus, etc.). In this way, the representation of a virtual object may be selectively and dynamically modified from a non-interaction configuration to a non-direct interaction configuration based on a characteristic of the input mechanism (e.g., a user's gaze), and then may be further modified from a non-direct interaction configuration to a direct interaction configuration based on another characteristic of the input mechanism or based on a characteristic of another input mechanism (e.g., a representation of an appendage, an input device, etc.).

[0186] Fig. 12A and Fig. 12B Another example is shown in which the representation of a virtual object within a CGR environment is modified based on the user's gaze. Specifically, Fig. 12A 8 shows a user 202 wearing an electronic device 200 configured to allow the user 202 to view a CGR environment 890. Fig. 12AAs shown, the first representation 1210 of the virtual object can be displayed at a specific size at a location 1251 via the display of the electronic device 200. In various aspects, the location 1251 can be on a wall of the CGR environment 890. In an embodiment, the first representation 1210 can be a representation of the virtual object, and the virtual object can be associated with a specific application program (e.g., a calendar, a multimedia application, a presentation, etc.), as described above. Fig. 12A In the example shown, first representation 1210 may be associated with a calendar application. In an embodiment, first representation 1210 may not be configured for user interaction, whether direct or indirect. For example, the size of first representation 1210 may be small, and the small size may not enable the user to perceive any information from or interact with any UI element of first representation 1210.

[0187] like Fig. 12A As shown, it can be determined that the gaze 1250 of the user 202 is directed to a location 1252, which can be different from the location 1251 at which the first representation 1210 is displayed. In various aspects, based on determining that the gaze 1250 is directed to a location different from the location of the first representation 1210, the display of the first representation 1210 can be maintained without displaying another representation of the virtual object and / or without making any changes to the first representation 1210.

[0188] Fig. 12B It is shown that the gaze 1250 of the user 202 has changed to a direction different from the direction pointed to the location 1252. In an embodiment, the change in gaze can be detected (e.g., via an input sensor). In response to the detected change in the user's gaze, the direction of the new direction of the gaze can be determined. For example, it can be determined that the new direction of the gaze 1250 can point to the location 1251. The location 1251 is the location where the first representation 1210 is displayed. In an embodiment, based on determining that the gaze 1250 is pointing to the same location as the location of the first representation 1210, the display of the first representation 1210 can be modified. For example, the first representation 1210 can stop being displayed, and the second representation 1220 can be displayed, wherein the second representation 1220 can be different from the first representation 1210. In some embodiments, the second representation 1220 can be displayed at the same location and / or on the same surface where the first representation 1210 is displayed.

[0189] In an embodiment, the second representation 1220 may be configured to include a UI element 1221. The UI element 1221 may include at least one UI element configured for user interaction, such as a display. In some embodiments, the second representation 1220 may alternatively or in addition have a size different from the size of the first representation 1210. For example, the size of the second representation 1220 may be larger or smaller than the size of the first representation 1210. In an embodiment, the size of the second representation 1220 may be based on the distance between the position of the second representation 1220 (e.g., position 1251) and the position of the user's head and / or eyes. In some embodiments, the second representation 1120 may be configured for indirect interaction, but may not be configured for direct interaction. For example, the second representation 1120 may not include any UI element configured for direct interaction with a user (e.g., a button, an affordance, a user interface element, an interactive element, etc.).

[0190] In some embodiments, determining to modify the display of the first representation 1210 based on determining that the gaze 1250 is directed to the same location as the location of the first representation 1210 may include determining that the gaze 1250 has remained directed to the same location as the location of the first representation 1210 for at least a predetermined period of time, such as with reference to Fig.11A and Fig. 11B described.

[0191] As previously described, in an embodiment, modifying the first representation (which may include displaying the second representation) may include animating the modification. For example, the modification of the first representation may include animating a change in size of the first representation, such that the first representation is displayed as increasing or shrinking from a current size to the size of the second representation, as appropriate. Additionally or alternatively, the modification of the first representation may include animating a UI element of the first representation, such that the UI element is presented as receding into the first representation. In an embodiment, the animation may also include a sound that may be played while the animation occurs.

[0192] It should be noted that in embodiments, the specific implementation of the technology described herein may include any combination of the above features and functions. For example, the representation of a virtual object may be modified to have any one of different sizes, different UI elements, different types of UI elements (e.g., flat UI elements, protruding UI elements, etc.), different orientations, different positions, different shapes, different brightness, etc. and / or any combination thereof.

[0193] Fig.131 is a flow chart illustrating a method 1300 for controlling a representation of a virtual object within a CGR environment based on characteristics of an input mechanism. In some embodiments, the method 1300 may be performed by the system 100 or a portion of the system 100. In some embodiments, the method 1300 may be performed by one or more external systems and / or devices. In some embodiments, the method 1300 may be performed by the system 100 (or a portion of the system 100) in conjunction with one or more external systems and / or devices.

[0194] At block 1302, the system displays a first representation of a virtual object within a CGR environment via a display of an electronic device (e.g., a wearable electronic device, an HMD device, etc.). For example, the first representation of the virtual object may be displayed on a representation of a display within the CGR environment via a first display (e.g., a left-eye display panel) or a second display (e.g., a second-eye display panel) of the electronic device. In an embodiment, the first representation of the virtual object may be a virtual representation (e.g., a virtual representation superimposed on a first surface of the CGR environment via a semi-transparent display of the electronic device).

[0195] In an embodiment, the first representation of a virtual object may be configured to facilitate indirect interaction with the virtual object. For example, the first representation of a virtual object may include at least one UI element configured for indirect interaction, so that the user can perceive the interaction with the UI element without directly manipulating the UI element (e.g., a UI element configured for output).

[0196] In an embodiment, the first representation of the virtual object may include at least one UI element that can be configured to facilitate indirect interaction but is not configured for direct interaction (e.g., the UI element can be displayed as a protruding 3D UI element). For example, the UI element may include a button, an affordance, a user interface element, an interactive element, etc. and / or any combination thereof. When the UI element is configured to facilitate direct interaction, the user may select, click, select, and / or otherwise manipulate the UI element.

[0197] In an embodiment, the movement of an input mechanism may be detected. The input mechanism may include a mechanism configured to facilitate interaction with a virtual object. For example, the input mechanism may include at least one element for a user to manipulate the representation of a virtual object or a mechanism for perceiving data provided by a virtual object. In an embodiment, the input mechanism may include a representation of a user's appendage (e.g., a finger, hand, leg, foot, etc.), a user's gaze (e.g., head gaze, eye gaze, etc.), an input device (e.g., a mouse, a stylus, etc.), etc. In an embodiment, the representation of a user's appendage may include a virtual representation of the appendage and / or may include data representing the features of the appendage within a CGR environment (e.g., location, orientation, distance from a specific point, etc.). In various aspects, an input sensor (e.g., a touch-sensitive surface, an image sensor, etc.) configured to perform hand tracking, head gaze tracking, eye gaze tracking, finger tracking, etc. may be used to detect the movement of the input mechanism. For example, the input mechanism may move from a previous position to a current position.

[0198] In an embodiment, in response to detected movement of the input mechanism, it may be determined whether the current position of the input mechanism is within a predetermined distance from the first representation. However, when movement of the input mechanism is not detected, the determination of whether the current position of the input mechanism is within a predetermined distance from the first representation may not be performed. In some embodiments, when the detected movement is determined to be toward the first representation, the determination of whether the current position of the input mechanism is within a predetermined distance from the first representation may be performed. In these cases, if the movement of the input mechanism is determined to be away from the first representation, the determination of whether the current position of the input mechanism is within a predetermined distance from the first representation may not be performed even if the movement of the input mechanism can be detected.

[0199] At block 1304, based on determining that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, the system displays a second representation of the virtual object within the CGR environment via a display of the electronic device. In an embodiment, the second representation of the virtual object may be different from the first representation of the virtual object.

[0200] In an embodiment, in response to displaying a second representation of a virtual object, the first representation may be stopped from being displayed. In some embodiments, the second representation may be displayed at the same location and / or on the same surface where the first representation is displayed.

[0201] In an embodiment, the second representation may be configured to facilitate direct interaction between the user and the associated virtual object. For example, the second representation may include at least one UI element of the UI elements configured for direct interaction. In an embodiment, the UI element may include at least one UI element displayed as a flat 2D UI element displayed on a physical object. In an embodiment, the UI element may include any one and / or any combination of buttons, affordances, user interface elements, interactive elements, etc.

[0202] In some embodiments, the second representation may have a size that is different from the size of the first representation. For example, the size of the second representation may be larger than the size of the first representation. In embodiments, the second representation may include a portion of the first representation, and the portion of the first representation included in the second representation may be larger than the size of the same portion in the first representation.

[0203] In some embodiments, the second representation of the virtual object may be displayed at a location different from the current location of the first representation. In an embodiment, the location where the second representation of the virtual object may be displayed may be a location closer to the user than the current location of the first representation. In an embodiment, the second representation displayed at the new location may be the same representation as the first representation.

[0204] In some embodiments, the first representation may be a 3D representation of the virtual object, and the second representation may be a 2D representation of the virtual object. In embodiments, the second representation may include at least a portion of the virtual object that is not displayed in the first representation of the virtual object.

[0205] As described above, one aspect of the present technology is to collect and use data that can be obtained from various sources to provide dedicated resource management for low-power devices with additional displays (e.g., HMD devices with additional displays), thereby saving battery life for users and providing dedicated content to users of low-power devices. The present disclosure contemplates that in some instances, this collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0206] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to save the battery life of the user's low-power device. Thus, for example, using such personal information data, the system properly manages resources to save the battery life of the low-power device. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0207] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving the user's informed consent. In addition, such entities should consider taking any necessary steps to defend and safeguard access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0208] Regardless of the foregoing, the present disclosure also contemplates an example in which a user selectively blocks the use or access of personal information data. That is, the present disclosure contemplates providing hardware elements and / or software elements to prevent or block access to such personal information data. For example, with respect to managing resources for low-power devices, the present technology may be configured to allow a user to choose to "opt in" or "opt out" at any time during or after registration for a service to participate in the collection of personal information data. In another example, a user may choose not to provide eye tracking data, such as pupil location, pupil dilation, and / or blink rate, for specialized resource management. In yet another example, a user may choose to limit the length of time that eyeglass tracking data is maintained or to completely prohibit the development of a baseline eyeglass tracking file. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then the user may be reminded again just before the personal information data is accessed by the application.

[0209] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0210] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed examples, the present disclosure also contemplates that various examples may also be implemented without access to such personal information data. That is, various examples of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, resources of a low-power device may be managed, and preferences may be inferred based on non-personal information data or an absolute minimum of personal information (such as content requested by a device associated with a user, other non-personal information available to a system controlling the low-power device, or publicly available information), thereby selecting content (e.g., status updates and / or objects) and delivering them to the user.

Claims

1. A method comprising: displaying a first representation of the virtual object within a computer-generated reality (CGR) environment via a display of the wearable electronic device, wherein the first representation is displayed at the corresponding location; and In response to detected movement of the input mechanism: Based on determining that the current position of the input mechanism is within a predetermined distance from the first representation of the virtual object, displaying a second representation of the virtual object within the CGR environment via the display of the wearable electronic device, wherein the second representation is different from the first representation, and wherein at least a portion of the second representation is displayed at the corresponding position; and Based on determining that the current position of the input mechanism is not within the predetermined distance from the first representation of the virtual object, the first representation remains displayed without displaying the second representation.

2. The method according to claim 1, wherein: The detected movement of the input mechanism is detected at least in part by an image sensor.

3. The method according to any one of claims 1 to 2, wherein: The input mechanism includes at least one of an attachment of a user of the wearable electronic device, a gaze of the user, and input received on an input device different from the electronic device.

4. The method according to claim 3, wherein: The input mechanism is determined based on a position of the first representation of the virtual object within the CGR environment.

5. The method according to any one of claims 1 to 2, wherein: Displaying the second representation of the virtual object includes displaying at least one interactive user interface (Ul) element of the second representation.

6. The method according to claim 5, wherein: The at least one interactive UI element is configured for direct interaction.

7. The method according to claim 6, wherein: The at least one interactive UI element configured for direct interaction corresponds to at least one UI element of the first representation, the at least one UI element of the first representation being configured for indirect interaction.

8. The method according to claim 5, wherein: The at least one interactive UI element of the second representation corresponds to at least one UI element of the first representation, and wherein a size of the at least one interactive UI element of the second representation is different from a size of the corresponding at least one UI element of the first representation.

9. The method according to claim 5, wherein: The at least one interactive UI element of the second representation corresponds to at least one UI element of the first representation, wherein the at least one UI element of the first representation is displayed as a three-dimensional element in the first representation, and wherein the at least one interactive UI element of the second representation is displayed as a two-dimensional representation of the corresponding three-dimensional element.

10. The method according to claim 5, wherein: The at least one interactive UI element of the second representation corresponds to at least one UI element not displayed in the first representation.

11. The method according to any one of claims 1 to 2, wherein: The second representation of the virtual object has a size that is different from a size of the first representation of the virtual object.

12. The method according to any one of claims 1 to 2, wherein: Displaying the second representation of the virtual object includes displaying the second representation at a location different from the location where the first representation is displayed.

13. The method according to claim 12, wherein: The location at which the second representation is displayed is a location closer to a user of the wearable electronic device than the location at which the first representation is displayed.

14. The method according to any one of claims 1 to 2, wherein: The first representation is a three-dimensional 3D representation of the virtual object and the second representation is a two-dimensional 2D representation of the virtual object.

15. The method according to any one of claims 1 to 2, wherein: Displaying the second representation of the virtual object within the CGR environment includes animating at least one difference between the second representation and the first representation.

16. The method according to claim 15, wherein: Animating the at least one difference between the second representation and the first representation includes playing at least one sound associated with the at least one difference.

17. The method according to any one of claims 1 to 2, wherein: The input mechanism includes a body part of a user of the wearable electronic device and / or a gaze of the user.

18. The method according to any one of claims 1 to 2, wherein: The first representation of the virtual object is a three-dimensional object; and The second representation of the virtual object is a three-dimensional object.

19. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a wearable electronic device that communicates with a display, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 18.

20. A wearable electronic device configured to communicate with a display, the wearable electronic device comprising: one or more processors; as well as A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method according to any one of claims 1 to 18.

21. A wearable electronic device configured to communicate with a display, the wearable electronic device comprising: Device for carrying out the method according to any one of claims 1 to 18.

22. A computer program product comprising one or more programs configured to be executed by one or more processors of a wearable electronic device in communication with a display, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 18.

23. A method comprising: displaying, via a display of the wearable electronic device, a two-dimensional (2D) representation of the virtual object at a first location within the computer-generated reality (CGR) environment; receiving a request to display the three-dimensional (3D) representation of the virtual object simultaneously with the 2D representation; as well as In response to the request, the 2D representation at the first position and the 3D representation at the second position of the CGR environment are simultaneously displayed via the display of the wearable electronic device, wherein the second position is different from the first position, and wherein the 2D representation and the 3D representation include different numbers of interactive user interface (UI) elements.

24. The method according to claim 23, wherein: Receiving the request for simultaneous display includes receiving a request to drag the 2D representation of the virtual object from the first position to the second position of the CGR environment.

25. The method according to any one of claims 23 to 24, wherein: The request for concurrent display includes a selection of an interactive element to be displayed with the 2D representation of the virtual object.

26. The method according to any one of claims 23 to 24, wherein: Simultaneously displaying the 2D representation at the first location of the CGR environment and displaying the 3D representation at a second location of the CGR environment includes animating displaying the 3D representation as a pop-up of the 2D representation.

27. The method according to any one of claims 23 to 24, further comprising: In response to receiving a first request to modify the 2D representation: modifying the 3D representation according to a modification corresponding to the first request to modify the 2D representation; as well as In response to receiving a first request to modify the 3D representation: The 2D representation is modified according to a modification corresponding to the first request to modify the 3D representation.

28. The method according to claim 27, further comprising: In response to receiving a second request to modify the 2D representation: abandoning modifying the 3D representation according to a modification corresponding to the second request to modify the 2D representation; as well as In response to receiving a second request to modify the 3D representation: Modifying the 2D representation according to the modification corresponding to the second request to modify the 3D representation is abandoned.

29. The method according to any one of claims 23 to 24, wherein: The 2D representation of the virtual object includes at least one interactive user interface element.

30. The method of claim 29, wherein: The at least one interactive UI element of the 2D representation is not included in the 3D representation of the virtual object.

31. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a wearable electronic device that communicates with a display, the one or more programs comprising instructions for executing a method according to any one of claims 23 to 30.

32. A wearable electronic device configured to communicate with a display, the wearable electronic device comprising: one or more processors; as well as A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method according to any one of claims 23 to 30.

33. A wearable electronic device configured to communicate with a display, the wearable electronic device comprising: Device for carrying out the method according to any one of claims 23 to 30.

34. A computer program product comprising one or more programs configured to be executed by one or more processors of a wearable electronic device in communication with a display, the one or more programs including instructions for performing the method according to any one of claims 23 to 30.

Images