Controlling display

Abstract

The invention relates to controlling a display. According to some embodiments, an exemplary process is described for dynamically controlling the size of a display in a computer generated reality (CGR) environment based on movement of visual objects satisfying criteria. According to other embodiments, an exemplary process is described for controlling the display of a representation in a display within a CGR environment. According to yet other embodiments, an exemplary process is described for controlling display of UI elements based on gaze within a CGR environment. According to other embodiments, an exemplary process is described for controlling display of a virtual display window based on an articulation angle of a physical device within a CGR environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 048845, international application date August 31, 2020, date of entry into the Chinese national phase July 29, 2021, Chinese national application number 202080027017.X, and invention name “Controlling Display”.

[0003] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 058,238, filed on July 29, 2020, entitled “CONTROLLING DISPLAYS IN A COMPUTER-GENERATED REALITY ENVIRONMENT” and U.S. Provisional Patent Application Serial No. 62 / 906,465, filed on September 26, 2019, entitled “DYNAMICALLY CONTROLLING THE SIZE OF DISPLAYS IN A COMPUTER-GENERATED REALITY ENVIRONMENT”. The contents of the foregoing applications are hereby incorporated by reference in their entirety. Technical Field

[0004] The present disclosure relates generally to computer generated reality environments, and more particularly to dynamically controlling displays in computer generated reality environments. Background Art

[0005] A computer-generated reality environment is an environment in which at least some of the objects displayed for viewing by a user are generated by a computer. In some uses, a user may interact with a representation of a physical display within the computer-generated reality environment. However, there is a lack of robust functionality for dynamically controlling the size of the display within the computer-generated reality environment. Furthermore, there is a lack of functionality for controlling the display of representations within the computer-generated reality environment.

[0006] Furthermore, in current systems, there is a lack of functionality for controlling the display of user interface elements based on gaze within a computer-generated reality environment. Additionally, existing systems lack functionality for controlling the display of virtual display windows based on the articulation angle of a physical device within a computer-generated reality environment. Summary of the invention

[0007] According to some embodiments, a method includes: presenting a representation of a physical display within a computer generated reality (CGR) environment via a display of a wearable electronic device, wherein the representation of the physical display includes a first display area; displaying a visual object at a first location within the first display area of ​​the representation of the physical display via the display of the wearable electronic device; receiving a request to move the visual object to a second location within the CGR environment; and in response to receiving the request, displaying an extended display area of ​​the representation of the physical display via the display of the wearable electronic device when the second location meets a criterion, wherein the extended display area includes at least a portion of the first display area and a second display area, wherein the second display area is sized to accommodate the visual object at at least the second location.

[0008] According to some embodiments, a wearable electronic device includes a display, one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: presenting a representation of a physical display within a CGR environment via the display, wherein the representation of the physical display includes a first display area; displaying a visual object at a first location within the first display area of ​​the representation of the physical display via the display; receiving a request to move the visual object to a second location within the CGR environment; and in response to receiving the request, displaying an extended display area of ​​the representation of the physical display via the display when a criterion is met at the second location, wherein the extended display area includes at least a portion of the first display area and a second display area, wherein the size of the second display area is set to accommodate the visual object at at least the second location.

[0009] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for: presenting a representation of a physical display within a CGR environment via the display, wherein the representation of the physical display includes a first display area; displaying a visual object at a first location within the first display area of ​​the representation of the physical display via the display; receiving a request to move the visual object to a second location within the CGR environment; and in response to receiving the request, displaying an extended display area of ​​the representation of the physical display via the display when a criterion is met at the second location, wherein the extended display area includes at least a portion of the first display area and a second display area, wherein the size of the second display area is set to accommodate the visual object at at least the second location.

[0010] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for the following operations: presenting a representation of a physical display via the display within a CGR environment, wherein the representation of the physical display includes a first display area; displaying a visual object via the display at a first location within the first display area of ​​the representation of the physical display; receiving a request to move the visual object to a second location within the CGR environment; and in response to receiving the request, displaying an extended display area of ​​the representation of the physical display via the display when a criterion is met at the second location, wherein the extended display area includes at least a portion of the first display area and a second display area, wherein the size of the second display area is set to accommodate the visual object at at least the second location.

[0011] According to some embodiments, a wearable electronic device includes: a display; and a device for presenting a representation of a physical display via the display within a CGR environment, wherein the representation of the physical display includes a first display area; a device for displaying a visual object at a first location within the first display area of ​​the representation of the physical display via the display; a device for receiving a request to move the visual object to a second location within the CGR environment; and a device for displaying an extended display area of ​​the representation of the physical display via the display in response to receiving the request when a criterion is met at the second location, wherein the extended display area includes at least a portion of the first display area and a second display area, wherein the size of the second display area is set to accommodate the visual object at at least the second location.

[0012] According to some embodiments, a method includes: displaying a virtual display area representing a physical display within a CGR environment via a display of a wearable electronic device; displaying a first representation of an area for displaying system elements in a first area; and when the first representation of the area for displaying system elements is not selected: detecting an event in a second area of ​​the virtual display area, wherein the second area is outside the first area; and when it is determined that the event meets a movement criterion, displaying a second representation of the area for displaying system elements in the second area.

[0013] According to some embodiments, a wearable electronic device includes a display, one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: displaying a virtual display area representing a physical display within a CGR environment via the display of the wearable electronic device; displaying a first representation of an area for displaying system elements in a first area; and when the first representation of the area for displaying system elements is not selected: detecting an event in a second area of ​​the virtual display area, wherein the second area is outside the first area; and when it is determined that the event meets a movement criterion, displaying a second representation of the area for displaying system elements in the second area.

[0014] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for the following operations: displaying a virtual display area representing a physical display within a CGR environment via the display of the wearable electronic device; displaying a first representation of an area for displaying system elements in a first area; and when the first representation of the area for displaying system elements is not selected: detecting an event in a second area of ​​the virtual display area, wherein the second area is outside the first area; and when it is determined that the event meets a movement criterion, displaying a second representation of the area for displaying system elements in the second area.

[0015] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for the following operations: displaying a virtual display area representing a physical display within a CGR environment via the display of the wearable electronic device; displaying a first representation of an area for displaying system elements in a first area; and when the first representation of the area for displaying system elements is not selected: detecting an event in a second area of ​​the virtual display area, wherein the second area is outside the first area; and when it is determined that the event meets a movement criterion, displaying a second representation of the area for displaying system elements in the second area.

[0016] According to some embodiments, a wearable electronic device includes: a display and a device for displaying a virtual display area representing a physical display within a CGR environment via the display; a device for displaying a virtual display area representing a physical display within a CGR environment via the display of the wearable electronic device; a device for displaying a first representation of an area for displaying system elements in a first area; and when the first representation of the area for displaying system elements is not selected, a device for the following operations: detecting an event in a second area of ​​the virtual display area, wherein the second area is outside the first area; and when it is determined that the event meets a movement criterion, displaying a second representation of the area for displaying system elements in the second area.

[0017] According to some embodiments, a method includes: at a wearable electronic device configured to facilitate presentation of a CGR environment: obtaining data representing an articulation angle of a physical object; and in response to obtaining the data representing the articulation angle of the physical object: displaying a virtual display window relative to the physical device via a display of the wearable electronic device when it is determined that the articulation angle exceeds a threshold articulation angle.

[0018] According to some embodiments, a wearable electronic device includes: a display, one or more processors and a memory, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: obtaining data representing an articulation angle of a physical object; and in response to obtaining data representing the articulation angle of the physical object: displaying a virtual display window relative to the physical device via the display when it is determined that the articulation angle exceeds a threshold articulation angle.

[0019] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for: obtaining data representing an articulation angle of a physical object; and in response to obtaining data representing the articulation angle of the physical object: displaying a virtual display window relative to the physical device via the display when it is determined that the articulation angle exceeds a threshold articulation angle.

[0020] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for: obtaining data representing an articulation angle of a physical object; and in response to obtaining data representing the articulation angle of the physical object: displaying a virtual display window relative to the physical device via the display when it is determined that the articulation angle exceeds a threshold articulation angle.

[0021] According to some embodiments, a wearable electronic device includes a display and a device for the following operations: obtaining data representing an articulation angle of a physical object; and in response to obtaining the data representing the articulation angle of the physical object: displaying a virtual display window relative to the physical device via the display when it is determined that the articulation angle exceeds a threshold articulation angle.

[0022] According to some embodiments, a method includes: displaying a user interface (UI) element at a first location within a CGR environment via a display of a wearable electronic device; detecting a current gaze of a user within the CGR environment; determining a second location within the CGR environment based on the current gaze of the user, wherein the second location is different from the first location; receiving a request to move the UI element; in response to determining that the request meets movement criteria, displaying the UI element at a third location within the CGR environment, wherein the third location is different from the first location and the second location; and in response to displaying the UI element at the third location, moving the UI element to the second location according to the request.

[0023] According to some embodiments, a wearable electronic device includes a display, one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: displaying a UI element at a first location within a CGR environment via the display; detecting a current gaze of a user within the CGR environment; determining a second location within the CGR environment based on the current gaze of the user, wherein the second location is different from the first location; receiving a request to move the UI element; in response to determining that the request meets movement criteria, displaying the UI element at a third location within the CGR environment, wherein the third location is different from the first location and the second location; and in response to displaying the UI element at the third location, moving the UI element to the second location according to the request.

[0024] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for: displaying a UI element at a first location within a CGR environment via the display; detecting a current gaze of a user within the CGR environment; determining a second location within the CGR environment based on the user's current gaze, wherein the second location is different from the first location; receiving a request to move the UI element; in response to determining that the request meets a movement criterion, displaying the UI element at a third location within the CGR environment, wherein the third location is different from the first location and the second location; and in response to displaying the UI element at the third location, moving the UI element to the second location according to the request.

[0025] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a wearable electronic device including a display, the one or more programs including instructions for: displaying a UI element at a first location within a CGR environment via the display; detecting a current gaze of a user within the CGR environment; determining a second location within the CGR environment based on the user's current gaze, wherein the second location is different from the first location; receiving a request to move the UI element; in response to determining that the request meets movement criteria, displaying the UI element at a third location within the CGR environment, wherein the third location is different from the first location and the second location; and in response to displaying the UI element at the third location, moving the UI element to the second location according to the request.

[0026] According to some embodiments, a wearable electronic device includes a display and a device for the following operations: displaying a UI element at a first position within a CGR environment via the display; detecting a current gaze of a user within the CGR environment; determining a second position within the CGR environment based on the current gaze of the user, wherein the second position is different from the first position; receiving a request to move the UI element; in response to determining that the request meets movement criteria, displaying the UI element at a third position within the CGR environment, wherein the third position is different from the first position and the second position; and in response to displaying the UI element at the third position, moving the UI element to the second position according to the request.

[0027] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In the following description, reference is made to the accompanying drawings that form a part of the present disclosure and illustrate multiple examples of the present disclosure. It should be understood that other examples may also be utilized, and structural and operational changes may be made without departing from the scope of the present disclosure. The same reference numerals are used in different drawings to represent similar or identical items.

[0029] Figure 1A to Figure 1B Exemplary systems for use in various CGR techniques are depicted.

[0030] FIG. 2A to FIG. 2F Exemplary techniques for dynamically controlling the size of a display in a CGR environment are described in accordance with aspects of the present disclosure.

[0031] Figure 3An example of a shift operation of a display in a CGR environment according to aspects of the present disclosure is depicted.

[0032] Figure 4A and Figure 4B An example of a resizing operation according to aspects of the present disclosure is shown.

[0033] Figure 5 is a flow chart illustrating a method performed to implement aspects of the present disclosure.

[0034] Fig. 6A and Figure 6B Depicted are exemplary techniques for controlling display of representations of system bars in a display within a CGR environment in accordance with aspects of the present disclosure.

[0035] Fig. 7A and Figure 7B Another example of a technique for controlling display of a representation of system bars in a display within a CGR environment is depicted in accordance with aspects of the present disclosure.

[0036] Fig. 8A and Figure 8B Yet another example of a technique for controlling display of a representation of system bars in a display within a CGR environment is depicted in accordance with aspects of the present disclosure.

[0037] Fig. 9 is a flow chart illustrating a method performed to implement aspects of the present disclosure.

[0038] FIG. 10A to FIG. 10J Depicted are exemplary techniques for controlling the display of a virtual display window based on an articulation angle of a physical device within a CGR environment in accordance with aspects of the present disclosure.

[0039] Fig.11 is a flow chart illustrating a method performed to implement aspects of the present disclosure.

[0040] FIG. 12A to FIG. 12E Depicted are exemplary techniques for controlling display of UI elements based on gaze within a CGR environment in accordance with aspects of the present disclosure.

[0041] Fig.13 is a flow chart illustrating a method performed to implement aspects of the present disclosure. DETAILED DESCRIPTION

[0042] Various examples of electronic systems related to various CGR techniques and techniques for using such systems are described.

[0043] The physical environment (or real environment) refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical items (or physical objects or real objects), such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0044] In contrast, a CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0045] A person may sense and / or interact with CGR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object, which creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0046] Examples of CGR include virtual reality and mixed reality.

[0047] A virtual reality (VR) environment (virtual environment) refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0048] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, an MR environment is anything between a fully physical environment at one end and a VR environment at the other end, but not including both ends.

[0049] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0050] Examples of MR include augmented reality and augmented virtuality.

[0051] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0052] An AR environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not true version of the original captured image. For another example, a representation of a physical environment may be transformed by graphically eliminating or blurring a portion thereof.

[0053] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0054] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one example, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0055] Figure 1A and Figure 1B An exemplary system 100 is depicted for use in various CGR techniques.

[0056] In some examples, such as Figure 1A As shown, system 100 includes device 100a. Device 100a includes various components, such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components are optionally in communication via a communication bus 150 of device 100a.

[0057] In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop computer), and other elements of system 100 are implemented in a head mounted display (HMD) device designed to be worn by a user, wherein the HMD device communicates with the base station device. In some examples, device 100a is implemented in either the base station device or the HMD device.

[0058] like Figure 1B As shown, in some examples, system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection. The first device 100b (e.g., a base station device) includes a processor 102, an RF circuit 104, and a memory 106. These components optionally communicate through a communication bus 150 of device 100b. The second device 100c (e.g., an HMD) includes various components, such as a processor 102, an RF circuit 104, a memory 106, an image sensor 108, an orientation sensor 110, a microphone 112, a position sensor 116, a speaker 118, a display 120, and a touch-sensitive surface 122. These components optionally communicate through a communication bus 150 of device 100c.

[0059] In some examples, system 100 is a mobile device. In some examples, system 100 is a HDM device. In some examples, device 100 is a wearable HUD device.

[0060] System 100 includes processor 102 and memory 106. Processor 102 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 102 to perform the following techniques.

[0061] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, an intranet), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 104 optionally includes circuitry for communicating using near field communications and / or short range communications (such as ) circuit for communication.

[0062] System 100 includes a display 120. In some examples, display 120 includes a first display (e.g., a left eye display panel) and a second display (e.g., a right eye display panel), each display being used to display an image to a corresponding eye of a user. Corresponding images are displayed simultaneously on the first display and the second display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby generating a parallax effect that provides the user with a stereoscopic effect of the object on the display. In some examples, display 120 includes a single display. For each eye of the user, the corresponding image is simultaneously displayed on a first area and a second area of ​​the single display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby generating a parallax effect that provides the user with a stereoscopic effect of the object on the single display.

[0063] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user inputs, such as tap inputs and swipe inputs. In some examples, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0064] System 100 includes an image sensor 108. Image sensor 108 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real environment. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the real environment. Image sensor 108 also optionally includes one or more event cameras configured to capture the movement of physical objects in the real environment. Image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of physical objects from system 100. In some examples, system 100 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical environment around system 100. In some examples, image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in the real environment from two different perspectives. In some examples, the system 100 uses the image sensor 108 to receive user input, such as gestures. In some examples, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and / or the display 120 in the real environment. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed objects in the real environment.

[0065] In some examples, system 100 includes microphone 112. System 100 uses microphone 112 to detect sounds from the user and / or the user's real environment. In some examples, microphone 112 includes a microphone array (including multiple microphones), which are optionally operated in series to identify ambient noise or locate a sound source in the space of the real environment.

[0066] System 100 includes an orientation sensor 110 for detecting orientation and / or movement of system 100 and / or display 120. For example, system 100 uses orientation sensor 110 to track changes in position and / or orientation of system 100 and / or display 120, such as with respect to physical objects in a real environment. Orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0067] Various aspects of the present disclosure are directed to systems and techniques that provide functionality for controlling displays (eg, virtual displays) within a CGR environment.

[0068] Specifically, aspects of the present disclosure relate to the following systems and techniques, which provide functionality for dynamically controlling the size of a display in a CGR environment. In some aspects, controlling the size of a display may include controlling the resizing of the display. The systems and techniques described herein allow, for example, the representation of a display to be expanded and / or reduced in a CGR environment in response to the movement of a visual object satisfying a resizing criterion. Throughout the present disclosure, resizing may be used to collectively refer to the expansion and / or reduction of the representation of a display within a CGR. As shown in the embodiments described below, the expansion of the representation of a display may be performed virtually, wherein the expanded representation of the display may include at least one area or portion as a computer-generated display area. Also as shown in the embodiments described below, the reduction of the representation of a display may be performed virtually, wherein reducing the representation of a display may include stopping displaying at least one area or portion of a computer-generated display area (e.g., a portion of an extended display area) and / or blurring a portion of the representation of the display (e.g., blurring a portion of a physical display within a CGR environment).

[0069] FIG. 2A to FIG. 2F Exemplary techniques for dynamically controlling the size of a display in a CGR environment according to aspects of the present disclosure are shown. Specifically, Figure 2A A user 202 and an electronic device 200 are shown. In some embodiments, the electronic device 200 may be a wearable electronic device (eg, an HMD). Examples of wearable electronic devices are described herein, such as with respect to the above reference Figure 1A to Figure 1B The electronic device 100a.

[0070] Note that while the following discussion focuses on wearable electronic devices, such as HMDs, this is by way of example and should not be construed as limiting in any way. In fact, the techniques disclosed herein are applicable to any specific implementation and form factor of an electronic device configured to present a CGR environment as described above (e.g., HMDs, HUDs, vehicle windshields with integrated display capabilities, windows with integrated display capabilities, etc.).

[0071] In an embodiment, the electronic device 200 may be configured to present a CGR environment to the user 202. The CGR environment presented to the user 202 may include presenting a display 204. In various aspects, the display 204 may be a representation of a physical display (e.g., a monitor or screen of a physical computer). For example, the display 204 may be a computer-generated simulation of a physical display. In some embodiments, the computer-generated simulation of a physical display may be configured to be perceived as a physical display by the user 202, but the display 204 may be a computer-generated simulation of a physical display, and / or may be configured to behave as a physical display. In some embodiments, the computer-generated simulation of a physical display may be a computer-generated display area without a corresponding physical counterpart and / or representation. In these cases, the computer-generated simulation of a physical display may be a display area that may not simulate a physical display. In these embodiments, the computer-generated simulation of a physical display may not be configured to be perceived as a physical display by the user 202, but may be configured to be perceived as a virtual display area, and / or may not be configured to behave as a physical display.

[0072] In some embodiments, the display 204 may represent a physical display presented to the user 202 via the electronic device 200. For example, the display 204 may include an actual physical display, and may not be a computer-generated simulation of the display. In this case, the display 204 may be presented to the user 202 so that the user 202 can view the physical display instead of viewing the computer-generated simulation of the physical display. In some embodiments, presenting the physical display to the user 202 may be performed via a transparent or translucent display of the electronic device 200, or may be performed via a pass-through video as described above, so that the user 202 can perceive the actual physical display. In yet other embodiments, presenting the physical display to the user 202 may be performed by superimposing a realistic image and / or video of the physical display on a simulated environment.

[0073] In other embodiments, the display 204 may include a computer-generated simulation of a physical display as described above (e.g., a computer-generated simulation of a physical display or a purely virtual display). The computer-generated simulation of the physical display may be presented to the user 202 via the electronic device 200. In some embodiments, presenting the computer-generated simulation of the physical display to the user 202 may include displaying the computer-generated simulation of the physical display on the display of the electronic device 200. In some implementations, the computer-generated simulation of the physical display may be superimposed on a physical environment or a representation thereof (e.g., as in MR implementations), so that the user 202 may perceive the computer-generated simulation of the physical display superimposed on the physical environment. In other implementations, the computer-generated simulation of the physical display may be a virtual object that is part of a fully CGR environment (e.g., as in a VR application).

[0074] Note that the display 204 (whether a representation of a physical display or a computer-generated simulation) can be presented as a 3D object. Therefore, the user 202 can perceive the display 204 as a 3D representation of a physical display. In some embodiments, where the display 204 is a computer-generated simulation of a physical display, the display 204 can be presented to the user with a blurred background and / or a semi-transparent background, so that objects behind the display 204 can be visible or at least visible to the user 202 to a certain extent. Where the display 204 is a physical display, the display 204 can be presented to the user 202 without a blurred or semi-transparent background, so that objects behind the display 204 may not be visible to the user 202.

[0075] In an embodiment, the display 204 may be represented as having a specific size. For example, in the case where the display 204 is an actual physical display, the size of the display 204 may be determined by the actual size of the display. In a specific implementation where the display 204 is a computer-generated simulation, the size of the simulated display may be predetermined and may be perceived by the user 202. Note that in a computer-generated simulation, the size of the display 204 may be the size perceived by the user in a CGR environment. In these cases, the user 202 may perceive the display 204 as a specific size as determined by the operation and functionality of the electronic device 200.

[0076] In some embodiments, the display 204 may include a display area 206. The display area 206 may be configured to function as a display area of ​​the display 204, such that a visual object (e.g., a visual object 210) to be displayed by the display 204 may be presented on the display area 206. In an embodiment, the display area 206 may have a particular size, which may include a portion or all of the display 204. In some embodiments, the display area 206 may include the entire display 204, such that the display area 206 encompasses an area from edge to edge of the display 204. In other embodiments, the display 204 may include a border 208 of a particular width surrounding the display 204. In this case, the display area 206 may encompass an area of ​​the display 204 that does not include the border 208.

[0077] Visual object 210 may be a visual object configured to be displayed on display 204. In some aspects, visual object 210 may be a 2D object, and may be any visual object that may be configured to be displayed on a display. For example, visual object 210 may be an icon, a 2D window, an image, etc.

[0078] In some embodiments, such as in a specific implementation where display 204 can be a physical display, visual object 210 can be an object displayed on the physical display. For example, visual object 210 can be displayed as a graphic object of display 204. In these cases, visual object 210 can be presented to user 202 via a transparent or semi-transparent display of electronic device 200 or via a pass-through video, so that user 202 can perceive the graphic object displayed on display 204.

[0079] In other embodiments, the visual object 210 may be a visual object displayed on a display of the electronic device 200. In some implementations, the visual object displayed on the electronic device 200 may be superimposed on a representation of the display 204. In the case where the display 204 is a physical display, the user 202 may perceive the visual object as being displayed on the display 204, but the visual object 210 may not be displayed on the display 204, but rather displayed on the display of the electronic device 200. In the case where the display 204 is a computer-generated simulation of a physical display, the user 202 may perceive the visual object as being displayed on the display 204, where both the visual object 210 and the display 204 may be displayed on the display of the electronic device 200.

[0080] The display 204 may also be configured to include a proximity threshold 212. In an embodiment, the proximity threshold 212 may be visible or invisible on the display 204. The proximity threshold 212 may be predetermined to be a specific distance 214 from an edge of the display area 206 or from an edge of the display 204. Since a conventional display may have four edges, in some embodiments, a proximity threshold may be configured for each edge or for any subset of the four edges. For example, a proximity threshold may be configured for the right and left edges of the display 204, but not for the top and bottom edges of the display 204. As will be described in more detail below, the proximity threshold 212 may be used to determine whether the display 204 may be resized according to aspects of the present disclosure. Note that in some implementations, the display 204 does not include a proximity threshold for any edge.

[0081] Figure 2B An example of a visual object 210 moving from a first location 220 to a second location 222 of a display area 206 in a direction 216 is shown. In various aspects, the movement of the visual object 210 from the first location 220 to the second location 222 can be in response to a request to move the visual object 210. The request to move the visual object 210 can be included in a user input. For example, a user (e.g., user 202 or another user) can drag or otherwise move the visual object 210 from the first location 220 to the second location 222. The dragging or movement of the visual object 210 can be performed using an input sensor, as described above, which is configured to detect user input that interacts with a visual object of a CGR environment. In some embodiments, the input sensor may include a mouse, a stylus, a touch-sensitive surface, an image sensor, etc.

[0082] like Figure 2B As shown, the movement operation of the visual object 210 from the first location 220 to the second location 222 in the direction 216 may not meet the criteria for resizing the display 204. In an embodiment, the criteria for resizing the display 204 may include any one or more of the following: the second location is within a proximity threshold of the edge of the display; the second location is at least partially outside the display area of ​​the display; and / or the movement operation is in a specific direction. When the movement operation of the visual object 210 meets any of the above criteria, the display 204 can be resized according to various aspects of the present disclosure. However, when the movement operation of the visual object 210 does not meet any of the above criteria, resizing the display 204 can be abandoned.

[0083] As used herein, the second location being within a proximity threshold of an edge of a display may include a situation in which a visual object moved to the second location at least partially enters the proximity threshold when in the second location. Similarly, the second location being at least partially outside a display area of ​​the display may include a situation in which a visual object moved to the second location at least partially enters an area outside the display area of ​​the display when in the second location.

[0084] For example, Figure 2B As shown, the second location 222 may be within the display area 206. However, in this case, the second location 222 may not be within a predetermined proximity (e.g., the proximity threshold 212) of the edge of the display 204, or may be at least partially outside the display area 206. Figure 2B In the example shown, the display 204 may not be resized. In some embodiments, the criteria for resizing the display are not met until the entire visual object being moved enters a proximity threshold or an area outside the display area.

[0085] Figure 2C Another example of a movement operation of the visual object 210 from a first location 220 to a second location 224 in a direction 216 is shown. Figure 2C As shown, a request may be received to move a visual object 210 from a first location 220 to a second location 224. The second location 224 is a location where at least a portion of the visual object 210 is within a proximity threshold 212 of an edge 230 of the display 204. In this case, the second location 224 is a location where at least a portion of the visual object 210 is within a proximity threshold 212 of an edge 230 of the display 204, and the criterion for resizing the display 204 is satisfied. In response to the second location satisfying the criterion for resizing the display, the display 204 may be resized.

[0086] Figure 2D 2 shows yet another example of a movement operation of the visual object 210 from the first location 220 to the second location 226 in the direction 216. Figure 2D As shown, a request to move the visual object 210 from the first location 220 to the second location 226 may be received. The second location 226 is a location where at least a portion of the visual object 210 is outside the display area of ​​the display 204. In this case, the second location 226 is a location where at least a portion of the visual object 210 is outside the display area of ​​the display 204, and the criterion for resizing the display 204 is satisfied. In response to the second location satisfying the criterion for resizing the display, the display 204 may be resized.

[0087] In an embodiment, resizing the display 204 may include extending the display area of ​​the display 204 by displaying the display area 206 and the additional display area 260. The additional display area 260 may be a computer-generated display area that may be attached or otherwise added to the display 204 so that the user 202 may perceive the extended display 204 as larger. In some embodiments, the additional display area 260 may be added to the display 204 so that the extended display 204 may be perceived by the user 202 as a single cohesive and integrated display. In other words, in these embodiments, the user 202 may not perceive that the additional display area 260 has been attached to the display 204. In specific implementations where the display 204 is a computer-generated simulation, extending the display 204 may include displaying the display 204 at a larger size, including the additional display area 260.

[0088] In an embodiment, extending the display 204 may include moving the display from an edge 230 of the current display 204 (e.g., an edge of the physical display or an edge of a computer-simulated representation of the physical display) to an edge of the extended display 204 (e.g., Figure 2C The edge of 240 or Figure 2D 242 in ) is extended by a certain amount 250. In these embodiments, edge 230 can be determined to be an extended edge because the request to move visual object 210 includes a request to move visual object 210 toward edge 230.

[0089] The size of the extension (e.g., the size of the additional display area 260 and / or the size of the amount 250) may be determined by the size of the visual object 210 and may be sized to accommodate displaying the visual object 210 in the extension area. In other embodiments, the size of the extension may be determined by a second location (e.g., Figure 2C The second location in 224 or Figure 2D The display 204 may be extended by an amount x, based on the amount by which the edge 242 in the image has moved toward the edge beyond the proximity threshold 212 and / or the amount by which the visual object 210 has moved toward the edge beyond the proximity threshold 212. For example, if the second location 224 and / or the visual object 210 has moved a distance x beyond the edge 230, the display 204 may be extended by an amount x. In yet other embodiments, the size of the extension may be determined based on ergonomic design, such as to reduce neck strain.

[0090] In some embodiments, for example, where display 204 is a physical display and visual object 210 is a visual object displayed on display 204, when visual object 210 enters an extended display area of ​​display 204, visual object 210 may transition to a visual object displayed on the display of electronic device 200. In this sense, user 202 may be able to perceive visual object 210 even after visual object 210 may be moved outside of the physical display. This transition may also allow for seamless presentation of visual object 210 when it moves into the extended area.

[0091] Note that while this discussion focuses on expansion of the display from the left edge of the display to the left, the techniques herein are equally applicable in situations where expansion may be performed from the right edge to the right in response to moving a visual object toward the right edge of the display. Furthermore, this also applies to expansion of the top edge and / or the bottom edge. Additionally, in some embodiments, note that expansion of the display may be limited to the left and right, and moving an object into proximity of the top edge or the bottom edge and / or outside of a region near the top edge or the bottom edge may not result in expansion of the display.

[0092] In an embodiment, once the visual object 210 crosses the proximity threshold 212 (e.g., Figure 2C ) or edge 230 (e.g., as Figure 2D 210 ), the expansion of the display 204 may be performed, and the display 204 may continue to expand as the visual object 210 further crosses the proximity threshold 212 or the edge 230. In other embodiments, until at least a predetermined portion of the visual object 210 has crossed the proximity threshold 212 (e.g., as Figure 2C ) or edge 230 (e.g., as Figure 2D The display 204 may be expanded until the entire visual object 210 has crossed the proximity threshold 212 or the edge 230 .

[0093] In some embodiments, expanding the display 204 may include animating the expansion operation. Figure 2D , when the display 204 is extended from the current edge 230 to the extended edge 242, the extension may be displayed in an animated manner, for example by displaying the edge of the display 204 as moving or shifting from the current edge 230 to the extended edge 242. The speed of the animation display may be based on the speed at which the visual object 210 is moved, or may be based on a predetermined value and / or a fixed value. In some embodiments, as will be described in more detail below, when the display 204 is extended, the extended display may be curved.

[0094] In an embodiment, the resizing operation (e.g., expansion and / or reduction) of the display 204 may include rearranging the structural elements of the display 204. For example, the representation of the display 204 may include a display shelf 270. The display shelf 270 may be a visual element that provides more realism to the representation of the display 204. In an embodiment, resizing the display 204 may also include shifting the display shelf 270 to an appropriate location relative to the extended display so that the proportions of the display can be maintained. For example, the display shelf 270 may be shifted from a location 272 in the unextended display 204 to a location 274 in the extended display 204. Thus, after the expansion operation of the display 204, the shelf remains approximately in a location in the middle portion of the display 204, thereby maintaining the proportions of the display. In an embodiment, the displacement of the display shelf 270 may be displayed in an animated manner, and the shelf may be shown sliding from the location 272 to the location 274 when the display 204 is extended.

[0095] In some embodiments, the expansion of the display 204 may be limited to a specific size. For example, a threshold size may be used, and when the size of the extended display 204 reaches the threshold size, the expansion of the display 204 may be abandoned. For example, based on the movement operation of the visual object 210 meeting the resizing standard (e.g., the standard for extending the display), the display 204 may be determined to be extended to a specific size as described above. For example, in order to accommodate the visual object 210 in the extended display, it may be determined that the display 204 is extended by an amount 250 by adding an additional display area 260, thereby obtaining the target extended size of the extended display area of ​​the display 204. In the absence of a threshold size, the display 204 may be extended as determined. However, in the presence of a threshold size, the threshold size may be compared with the target extended size, and in the case where the target extended size of the display 204 exceeds the threshold size, the expansion of the display 204 may be abandoned. In some embodiments, the display 204 may still be extended, but the extension may be limited to the threshold size.

[0096] Figure 2E Another example of a dynamic resizing operation of a display is shown. Figure 2EIn the example shown, a request to move the visual object 210 from the first location 220 to the second location 228 may be received. In this example, the second location 228 may be a location where at least a portion of the visual object 210 may satisfy the resizing criteria relative to more than one edge at the same time. For example, the second location 228 may be a location where at least a portion of the visual object 210 is outside the display area near both the left edge and the top edge of the display 204. In this case, the display 204 may be extended from both the left edge 230 and the top edge 232 according to the above description. The extension of the left edge 230 may reach the edge 244 and may include animating the extension. Similarly, the extension of the top edge 232 may reach the edge 246 and may include animating the extension. In various aspects, the extension of both the left edge 230 and the top edge 232 may be performed simultaneously or nearly simultaneously, and the user 202 may perceive the extension of the two edges as a diagonal extension.

[0097] In some embodiments not shown, the user 202 may first move the visual object 210 from the first location 220 to a second location that meets the resizing criteria relative to a single display edge (e.g., a second location outside the display area near the left edge of the display 204), and then move the visual object 210 from the second location to a third location that meets the resizing criteria relative to another display edge (e.g., a third location outside the display area near the top edge of the display 204). In this case, the display 204 may first expand from the left edge in response to the movement of the visual object 210 to the second location, and then expand from the top edge in response to the movement of the visual object 210 to the third location. Alternatively, the display 204 may not expand first from the left edge and then from the top edge when the visual object moves to the second location and the third location, respectively, but may expand from the left edge and the top edge simultaneously or nearly simultaneously when the visual object moves to the third location. In this sense, the second location may be considered a transition location and may not result in a resizing operation. In various aspects, the transition location may be determined based on whether the moving operation has stopped for a predetermined amount of time. For example, user 202 may first move visual object 210 to a second location, stop for a period of time, and then move the visual object to a third location. In this case, if the period of time does not exceed a predetermined threshold, the second location may be determined to be a transition location, and expansion of display 204 based on the second location (e.g., expansion from the left edge) may be abandoned, even if the second location otherwise meets the resizing criteria (e.g., it is a location within the proximity threshold or a location outside the display area of ​​display 204).

[0098] Figure 2FAn example of dynamically controlling a resizing operation of a display including shrinking the display is shown. Figure 2F In the example shown, a request may be received to move a visual object 210 from a first location 221 to a second location 229 in a direction 217. In an embodiment, the first location 221 may be part of an extended display area of ​​the display 204. For example, the display 204 may have previously been extended by extending a certain amount 256 from edge 234 to edge 248 to include an additional display area 260. In this case, it may be determined that moving the visual object 210 into the display area 206 that may not include the additional display area 260 satisfies the resizing criteria. In this case, the resizing criteria may include shrinking the extended display back to the original display size by shrinking from edge 248 to edge 234 in the direction 217. Edge 234 may be an actual edge of the display 204 (e.g., an edge of the display 204 before the previous expansion). As described above, relative to the expansion operation, the shrinking operation may include animating the shrinking of the edge so that the edge of the display 204 appears to move from edge 248 to edge 234.

[0099] In some embodiments, moving the visual object 210 to a second location beyond the bottom edge of the display 204 may result in a zoom-out operation that may completely minimize the display 204. In this case, minimizing the display 204 may include collapsing the entire display into a representation indicating that the display is in a minimized state. The user 202 may interact with the minimized representation to restore the display 204 to its previous size.

[0100] In some embodiments not shown, the second location may not be completely within the display area 206 and may include at least a portion of the additional display area 260. In this case, it may be determined that the resizing criteria have been met and the display 204 may be reduced. In this case, the display 204 may be reduced, but only to a display size that can still accommodate displaying the visual object 210 within the reduced display area. The partial reduction may be repeated when the visual object 210 is moved to a location that is completely within the display area 206.

[0101] In other embodiments, the first location 221 may not represent a portion of the extended display area of ​​the display 204, but may be a portion of the unextended display area 206. In this example, the original size of the display 204 may include the additional display area 260 up to the edge 248. For this example, the original size of the display 204 may include the display area 206 and the additional display area 260. In this case, it may be determined that moving the visual object 210 to the second location 229 may support a smaller display area of ​​the display 204 while still accommodating the display of the visual object 210. This may be determined to meet the resizing criteria. In this case, the resizing criteria may include shrinking the display area of ​​the display 204 to a smaller size. The display area of ​​the display 204 may shrink from the edge 248 to the edge 234 in the direction 217. The edge 248 may be the actual edge of the display 204 (e.g., the edge of the display 204 before shrinking). As described above, relative to the expansion operation, the shrinking operation may include animating the shrinking of the edge so that the edge of the display 204 appears to move from the edge 248 to the edge 234.

[0102] In embodiments where the display 204 is a physical display, reducing the previously unexpanded display may include blurring a portion of the display. For example, when the display 204 is reduced after the visual object 210 is moved to the second location 229, the additional display area 260 may be blurred such that the blurred portion of the display 204 may not be used to display the visual object. One application of this functionality may be in AV applications, where the blurred portion of the display may be blurred by replacing the blurred portion of the display with an appropriate portion of a computer-generated simulation, such that the user 202 may perceive the display 204 as smaller while being able to perceive virtual objects behind the blurred portion of the display area.

[0103] Figure 3 An example of a shift operation of a display is shown. In this example, moving the visual object 210 from the first location 223 to the second location 225 may satisfy the above-mentioned FIG. 2A to FIG. 2F The resizing criteria described above. However, in this example, rather than resizing the display 204, the display 204 is shifted in the direction of the move operation. The amount of shift may be based on the amount of movement after the visual object has been placed at a location that satisfies the resizing criteria discussed above. When manipulating a 2D visual object displayed on the display 204, the user 202 may employ this functionality to move the display 204 to another location.

[0104] Figure 4A and Figure 4B An example of a resizing operation according to aspects of the present disclosure is shown. Figure 4A2 and an electronic device 200 are shown. As described above, in some embodiments, the electronic device 200 may be a wearable electronic device (e.g., an HMD). Also as described above, in some embodiments, the electronic device 200 may be similar to the above referenced device. Figure 1A and Figure 1B An electronic device 100a is described.

[0105] Figure 4A A CGR environment including a display 204 is shown presented to a user 202. As described above, the display 204 can be a physical display or a computer-generated simulation of a physical display. The display 204 can include visual objects 410 and 411 within a display area 206. The visual objects 410 and 411 and the display area 206 can be as described above. In some aspects, the display 204 can be presented to the user 202 as a curved display, or can be presented as a flat display.

[0106] As described above, the movement operation of visual objects 410 and / or 411 may result in dynamic resizing of display 204. In an embodiment, the dynamic resizing of display 204 may include the expansion of display 204. For example, a request to move visual object 210 from place 410 to place 422 in direction 416 may be received. It may be determined that moving visual object 210 from place 410 to place 422 in direction 416 meets the resizing (e.g., expansion) criteria, as described herein. In response to moving visual object 210 from place 410 to place 422 in direction 416, display 204 may be expanded from edge 430 to edge 440 to include additional display area 460. Additional display area 460 may be a computer-generated virtual display area configured similarly to additional display area 260 described above. Similarly, a request to move visual object 211 from place 411 to place 423 in direction 417 may be received. It may be determined that moving visual object 211 from location 411 to location 423 in direction 417 satisfies resizing (e.g., expansion) criteria, as described herein. In response to moving visual object 211 from location 411 to location 423 in direction 417, display 204 may expand from edge 431 to edge 441 to include additional display area 461. Additional display area 461 may be a computer-generated virtual display area configured similarly to additional display area 260 described above.

[0107] In an embodiment, when display 204 expands from edge 430 to edge 440 and / or from edge 431 to edge 441, the expanded display area may be curved, such as in Figure 4A. In some embodiments, before expansion, the display 204 may or may not be curved, but the expansion operation may cause the display 204 to be curved regardless. In these cases, after expansion, not only may the additional area 460 be curved, but the original pre-expansion display may also be curved. In this sense, the extended display is configured to be perceived by the user 202 as a single cohesive and integrated curved display. In some embodiments, the extended display may be reduced based on the movement operation of the visual object 410, as described above. In this case, regardless of whether the original pre-expansion display 204 was curved, after reduction, the display 204 may remain curved. In other embodiments, after the reduction operation, the display 204 may return to a flat configuration.

[0108] Figure 4B An exemplary top view configuration of a resizing operation according to aspects of the present disclosure is shown. Specifically, Figure 4B The bending aspect of the expansion operation is shown. For example, when the display 204 is expanded in direction 450, the expanded display is curved. Similarly, when the display 204 is expanded in direction 451, the expanded display is also curved. In some embodiments, the bending of the expanded display may cause the expanded display to partially or completely surround the user 202, so that the user may perceive the display as a display surrounding him or her. In some embodiments, the expansion operation in a single direction 452 may continue from one edge of the display 204 to another edge. For example, the user 202 may drag the visual object from a location within the display area of ​​the display 204 toward the edge 432 of the display 204 in direction 452. The user 202 may continue to drag the visual object to a location outside the display 204 near the edge 432, so that the display 204 is expanded from the edge 430 in direction 452. The user 202 may continue to drag the visual object in direction 452 through the expansion of the display 204, so that the display 204 continues to expand and bend. If the user 202 continues to drag the visual object through the extension until the leading edge 434 of the extension meets the edge 433 of the display 204, the result will be a 360 degree curved extended display completely around the user 202, as shown in FIG. Figure 4B Note again that, as described above, although the display 204 is Figure 4B Although shown as a flat display, the display 204 may also be a curved display.

[0109] In some embodiments, when the user 202 moves his head to the left or right, the extended display area around the user 202 can rotate to provide an ergonomically beneficial method that can reduce neck strain. For example, when the user 202 moves his head to the right, the extended display area around the user 202 can rotate toward the left to "bring" a portion of the extended display area into the field of view of the user 202. In this way, the user 202 does not have to turn his neck excessively to view the portion of the extended display area that is not within the front periphery of the user 202.

[0110] Figure 5 is a flow chart showing a method 500 for dynamically controlling the size of a display in a CGR environment. In some embodiments, the method 500 may be performed by an electronic device (such as the one described above with reference to FIG. Figure 1A and Figure 1B The electronic device 100a and / or the electronic device 200 described herein are executed.

[0111] At block 502, a representation of a physical display within a CGR environment is presented to a user. For example, an electronic device (such as electronic device 200) may be used to present a representation of a physical display (e.g., display 204) to a user (e.g., user 202). In some embodiments, the representation of the display may be an actual physical display presented to the user via a semi-transparent display or pass-through video of the electronic device. In other embodiments, the representation of the display may be a computer-generated simulation of a physical display displayed on a display of the electronic device. In an embodiment, the representation of the physical display may include a first display area (e.g., display area 206).

[0112] At block 504, a visual object (e.g., visual object 210) may be displayed at a first location within a first display area of ​​a representation of a physical display. In some embodiments, the visual object may be displayed via a display of a wearable electronic device. In some embodiments, for example, where the representation of the physical display is an actual physical display, the visual object may be displayed on the physical display. In various aspects, the visual object may be a 2D object and may be any visual object that may be configured to be displayed on a display. For example, the visual object may be an application icon, a 2D application window, an image, a thumbnail, etc.

[0113] At block 506, a request to move the visual object to a second location within the CGR environment may be received. For example, the user input may include a request to drag the visual object from the first location to the second location. The user input may include a drag operation performed using any input sensor configured to operate within the CGR environment, such as a mouse, stylus, touch-sensitive surface, image sensor, etc.

[0114] At box 508, in response to receiving a request to move the visual object to a second location, an extended display area of ​​the representation of the physical display is displayed when the second location meets the criteria. The extended display area may include at least a portion of the first display area and the second display area. In an embodiment, the second display area (e.g., additional display area 260) can be a computer-generated display area that can be added to the representation of the physical display to generate the extended display area. In some embodiments, the second display area can be added to the representation of the physical display so that the extended display area can be perceived by the user as a single cohesive and integrated display.

[0115] In some embodiments, the criteria satisfied by the second location in order to expand the display area may include any one or more of the following: the second location is within a proximity threshold of an edge of a representation of the physical display; the second location is at least partially outside the first display area; and / or the request to move the visual object is in a particular direction.

[0116] In an embodiment, an extended display area showing a representation of a physical display may include extending the representation of the physical display in the direction in which the visual object may have moved. The extension may include extending from the edge of the representation of the physical display in the direction in which the visual object moves. In various aspects, the extension operation may be displayed in an animated manner. In some embodiments, when the visual object has moved to a location that satisfies the resizing criteria for the top edge or bottom edge of the representation of the physical display (e.g., a second location is a location within a proximity threshold of the top edge or bottom edge, or when the second location is a location at least partially outside the first display area near the top edge or bottom edge), the display of the extended display area of ​​the representation of the physical display may be abandoned. Instead, the size of the representation of the physical display is maintained.

[0117] In some embodiments, the size of the second display area (e.g., the size of the expansion) may be determined by a size sufficient to accommodate the visual object in the second location. In some cases, the size of the second display area (e.g., the width of the second display area) may be less than the size of the visual object (e.g., the width), such as when at least a portion of the visual object remains within the first display area. In other cases, the size of the second display area may be at least the size of the visual object, such as when the entire visual object is outside the first display area when in the second location.

[0118] In an embodiment, extending the representation of the physical display may include limiting the extension of the representation of the physical display to a predetermined threshold size. For example, the expansion area of ​​the target size may be determined to accommodate the visual object at the second location. In this case, the target size may exceed the predetermined threshold size. In response to the target size exceeding the predetermined threshold size, the expansion area may be limited to the threshold size, or the extension of the display area may be abandoned entirely.

[0119] In some embodiments, the visual object may be moved from the second location to the third location within the first display area. In these embodiments, it may be determined that the extended display area no longer needs to accommodate the visual object. In these cases, the extended display area may be reduced back to its original pre-expanded size. For example, the second area may no longer be displayed, and only the first display area may be displayed. In an embodiment, the reduction of the extended area may be displayed in an animated manner.

[0120] In an embodiment, when the representation of the physical display is extended, such as by displaying an extended display area, the representation of the physical display can be curved. For example, when the extended display area is displayed, the first display area and the second display area can be curved. In some embodiments, only the second display area can be curved, while the first display area remains in a flat form. In some embodiments, after the extended display area is reduced, the reduced display area can maintain a curved form. In other embodiments, after the extended display area is reduced, the reduced display area can be displayed in a flat form.

[0121] In some embodiments, after expansion, at least a portion of the visual object may be displayed on the first display area, and at least another portion of the visual object may be displayed on the second display area. In this way, the user may perceive the visual display object as being displayed in a single coherent display without indicating that the display is an extended display consisting of the original first display area and the additional second display area.

[0122] Aspects of the present disclosure relate to the following systems and technologies that provide functionality for controlling the display of a representation of a system bar in a display within a CGR environment. In an embodiment, controlling the display of a representation of a system bar in a display may include: displaying a first representation of a system bar in a first area of ​​a display (e.g., a physical display or a virtual display) of the CGR environment; when the system bar is not selected, detecting an event in a second area of ​​the display; and when the event satisfies a system bar movement criterion, displaying a second representation of the system bar in the second area. In an embodiment, the system bar movement criterion may include events that may indicate potential and / or actual user interactions with elements in the second area of ​​the display. For example, the system bar movement criterion may include the opening of an application in the second area of ​​the display, detecting a user gaze directed to the second area of ​​the display, detecting movement of a cursor to the second area of ​​the display, selection of an interactive element for displaying a second representation of the system bar in the second area of ​​the display, and the like.

[0123] Fig. 6A and Figure 6B Exemplary techniques for controlling the display of a representation of a system bar in a display within a CGR environment are shown in accordance with aspects of the present disclosure. Specifically, Fig. 6Aand Figure 6B An example is shown in which display of a representation of a system bar is controlled based on events that meet system bar movement criteria including a change in user gaze or a change in cursor position.

[0124] Fig. 6A 2 shows a user 202 wearing an electronic device 200, which is configured to allow the user 202 to view a CGR environment 290. As described above, in some embodiments, the electronic device 200 may be similar to the device described above with reference to FIG. Figure 1A and Figure 1B An electronic device 100a is described.

[0125] like Fig. 6A As shown, the CGR environment 290 includes a display 600. In an embodiment, the display 600 can be a representation of a display (e.g., a physical display, such as a monitor or screen of a physical computer), such as a computer-generated simulation of a physical display. In some embodiments, the computer-generated simulation of a display can be a computer-generated display area that has no corresponding physical counterpart and / or representation. In these cases, the computer-generated simulation of a physical display can be a display area that may not simulate a physical display. In some implementations, the computer-generated simulation of a physical display can be superimposed on a physical environment or a representation thereof (e.g., as in an MR implementation), so that the user 202 can perceive the computer-generated simulation of a physical display superimposed on the physical environment. In other implementations, the computer-generated simulation of a physical display can be a virtual object that is part of a complete CGR environment (e.g., as in a VR application).

[0126] In some embodiments, the display 600 may represent a physical display presented to the user 202 via the electronic device 200. For example, the display 600 may include an actual physical display rather than a computer-generated simulation of a display. In this case, as described above with respect to Figure 2A As described with reference to display 204 of FIG. 1 , display 600 may be presented to user 202 so that user 202 may view a physical display rather than a computer-generated simulation of a physical display. In yet other embodiments, presenting a physical display to user 202 may be performed by superimposing realistic images and / or videos of a physical display on a simulated environment.

[0127] like Fig. 6AAs shown, display 600 may include regions 601, 602, and 603. In some embodiments, some or all of regions 601, 602, and 603 may be regions in a representation of a physical display (e.g., a virtual display), or may be a physical display. In these embodiments, display 600 may represent an aggregate display including more than one display. Therefore, regions 601, 602, and 603 may be determined to be in different representations of physical displays. For example, region 601 may be included in a representation of a first physical display, region 602 may be included in a representation of a second physical display, and region 603 may be included in a representation of a third physical display.

[0128] In some embodiments, display 600 may be a representation of a single continuous display (e.g., a virtual display or a physical display). In these embodiments, regions 601, 602, and 603 may be determined to be regions in the same representation of a physical display.

[0129] The first representation 620 of the system bar may be displayed by the electronic device 200 in an area of ​​the display 600. For example, the first representation 620 may be displayed on the area 603 of the display 600. In an embodiment, the first representation 620 may be displayed at any location within the area 603. For example, although the first representation 620 is shown as being displayed at the bottom of the area 603, the first representation 620 may be displayed at the top, left side, right side, center, etc. of the area 603. In an embodiment, the location of the first representation 620 in the area 603 may be determined by system preferences and / or settings, or may be determined by a user (e.g., user 202), who may position the system bar at a specific location within the area 603.

[0130] As used herein, a system bar may refer to a portion of a display area (e.g., a strip) that may be used to display system elements such as system information (e.g., notifications, status (e.g., status of open applications), etc.), provide navigation of functions and files within the system, display selectable elements for opening applications (e.g., icons, affordances, etc.), display selectable elements for selecting an open application (e.g., icons, affordances, etc.), provide application-specific options and / or functionality (e.g., when provided in association with a text editor, the system bar may present options for opening / closing / saving files, etc.). In some embodiments, the system bar is a menu with multiple options. In some embodiments, selection of each option in the menu causes the electronic device to perform a different operation.

[0131] In an embodiment, an input mechanism may be included to facilitate user interaction. For example, the input mechanism may include a mechanism for a user (e.g., user 202) to interact with the display 600. In an embodiment, the input mechanism may include a representation of the user's appendage (e.g., a representation of a finger, hand, leg, foot, etc.), the user's gaze (e.g., head gaze, eye gaze, etc.), an input device (e.g., a mouse, a stylus, a pen, a touch-sensitive surface, an image sensor, etc.), and the like. In an embodiment, the representation of the user's appendage may include a virtual representation of the appendage and / or may include data representing characteristics of the appendage within the CGR environment (e.g., location, orientation, distance from a particular point, etc.). In various aspects, the input mechanism may be detected using an input sensor (e.g., a touch-sensitive surface, an image sensor, etc.) configured to perform hand tracking, head gaze tracking, eye gaze tracking, finger tracking, and the like. As Fig. 6A As shown, the input mechanism may include gaze 650 and / or cursor 610 of user 202 .

[0132] In an embodiment, a gaze 650 of the user 202 may be detected. The detected gaze 650 may be a head gaze (e.g., a direction in which the user's head is facing), an eye gaze (e.g., a direction in which the user's eyes are looking), a combination thereof, or the like. The gaze 650 of the user 202 may be determined to be focused, placed, or otherwise directed to a particular area of ​​the display 600. For example, it may be determined that the gaze 650 of the user 202 is directed to the area 603 where the first representation 620 of the system bars is displayed. Alternatively or in addition, the cursor 610 may be determined to be positioned and / or displayed at a location within the area 603.

[0133] In an embodiment, the input mechanism may be moved. For example, the gaze 650 of the user 202 may be moved, and the movement of the gaze 650 may be detected (e.g., using an image sensor, etc.). In an embodiment, in response to the detected change in the user's gaze, the direction of the new direction of the gaze may be determined.

[0134] In one specific example, the first representation 620 displayed in the area 603 may be semi-transparent (e.g., translucent). In an embodiment, determining that the new direction of gaze 650 points to a new location in the area 603 that is closer than the previous location at which gaze 650 was directed may cause the first representation 620 to change from semi-transparent to opaque. For example, determining that the change in gaze 650 indicates that gaze 650 is approaching the first representation 620 (e.g., based on determining that the new direction of gaze 650 is a location within a predetermined distance of the first representation 620) may cause the first representation 620 to become opaque. In an embodiment, the opacity of the first representation 620 may be based on the distance from the location at which gaze 650 is directed to the location of the first representation 620. For example, as gaze 650 gets closer to the first representation 620, the opacity of the first representation 620 may increase. In an embodiment, as gaze 650 moves away from the first representation 620, the opacity of the first representation 620 may decrease (e.g., the first representation 620 may become more translucent). In additional or alternative embodiments, such functionality for changing the opacity of the first representation 620 may be based on movement of the cursor 610 toward or away from the first representation 620, and may use similar techniques as discussed above with respect to changes in user gaze (e.g., gaze 650).

[0135] In another example, Figure 6B As shown, it can be determined that the new direction of gaze 650 may point to an area different from area 603, such as area 602. Figure 6B 602. As shown. In various aspects, a representation of the system bar may not currently be displayed in area 602. Therefore, a user's gaze (e.g., gaze 650) may be detected in area 602, which is different from area 603 in which the first representation 620 may currently be displayed. Detection of gaze 650 may be determined to be an event in area 602 of display 600. Note that in some embodiments, rather than detecting movement of the user's gaze from area 603 to area 602 of the display, the user's gaze may be detected in area 602 without detecting actual movement of the user's gaze from area 603. In other words, it may not be detected whether gaze 650 has moved from area 603 to area 602, but gaze 650 may be detected as being directed to area 602. Note that in embodiments, detecting an event in area 602 of display 600 and other steps described below may occur when the system bar is not selected. For example, detecting gaze 650 on area 602 and subsequently displaying the second representation of the system bars may occur without user 202 selecting and / or interacting with the system bars (eg, via first representation 620 ).

[0136] In some embodiments, detecting that the gaze 650 is directed to a particular area may include determining that the gaze 650 has remained directed to a location within the particular area for at least a predetermined period of time. For example, it may be determined that the gaze 650 has remained directed to the area 602 for a period of time that is less than the predetermined period of time (e.g., the direction of the gaze 650 moves to a different direction before the predetermined period of time expires). In this case, it may not be determined that an event in the area 602 will occur. When it is determined that the gaze 650 has remained directed to the area 602 for a period of time that is at least the same as the predetermined period of time (e.g., the direction of the gaze 650 does not move to a different direction before the predetermined period of time expires), it may be determined that an event in the area 602 will be detected.

[0137] In an embodiment, in response to detecting an event in area 602, a determination is made as to whether the event satisfies system bar movement criteria. Figure 6B In the example shown, when the event in area 602 is determined to be a user gaze (eg, gaze 650 ) detected in area 602 , it may be determined that the event in area 602 meets the system bar movement criteria.

[0138] In response to determining that the event in area 602 satisfies the system bar movement criteria (e.g., gaze 650 is detected in area 602), a second representation of the system bar (e.g., second representation 621) is displayed in area 602, such as Figure 6B In an embodiment, the second representation 621 may have the same size as the first representation 620. In an embodiment, the second representation 621 may be similar to the first representation 620. For example, the second representation 621 may include the same theme, the same information, the same interactive elements (e.g., application icons, affordances, etc.), etc. In some embodiments, the second representation 621 may be different from the first representation 620.

[0139] In an embodiment, the first representation 620 can continue to be displayed in the area 603, while the second representation 621 is displayed in the area 602. In some embodiments, the first representation 620 can be displayed as semi-transparent in the area 603, while the second representation 621 is displayed in the area 602. In these embodiments, when it is determined that the gaze 650 moves back to the area 603, the first representation 620 can transition from semi-transparent to opaque. In other embodiments, the first representation 620 can cease to be displayed in the area 603, while the second representation 621 is displayed in the area 602.

[0140] In some embodiments, the second representation 621 may be displayed in the region 602 in the same orientation and / or the same region as the first representation 620 in the region 603. For example, the first representation 620 may be displayed as a horizontal bar across the bottom of the region 603. In this case, the second representation 621 may be displayed as a horizontal bar across the bottom of the region 602. It should be understood that the first representation 620 and the second representation 621 are described as being displayed on the bottom of their respective regions for illustrative purposes and not as a limitation. Therefore, in an embodiment, the first representation 620 and the second representation 621 may be displayed at any one of the bottom side, top side, left side, and / or right side of their respective regions. In an embodiment, the second representation 621 may be displayed in an orientation different from the first representation 620 in the region 603 and / or in a different region.

[0141] Alternatively or in addition, the input mechanism can include a cursor 610. In these embodiments, Figure 6B As shown, cursor 610 may move from a location in area 603 to a location in area 602. The movement of cursor 610 from area 603 to area 602 may be detected (e.g., using an input sensor, an image sensor, etc.). In an embodiment, the detection of cursor 610 in area 602 may be determined to be an event in area 602 of display 600. Note that in some embodiments, rather than detecting the movement of cursor 610 from area 603 to area 602 of the display, cursor 610 may be detected in area 602 without detecting the actual movement of cursor 610 from area 603. In other words, it may not be detected whether cursor 610 has moved from area 603 to area 602, but cursor 610 may be detected in area 602. Note that in an embodiment, detecting an event in area 602 of display 600 and the other steps described below may occur when the system bar is not selected. For example, detecting cursor 610 on area 602 and subsequently displaying the second representation of the system bar may occur without user 202 selecting and / or interacting with the system bar (eg, via first representation 620 ).

[0142] In an embodiment, in response to detecting an event in area 602, it may be determined whether the event satisfies system bar movement criteria. Figure 6B In the example shown, when determining the event in area 602 includes detecting cursor 610 moving to area 602 , it can be determined that the event in area 602 meets the system bar movement criteria.

[0143] In response to determining that the event in area 602 satisfies the system bar movement criteria (e.g., cursor 610 is moved to area 602), a second representation of the system bar (e.g., second representation 621) is displayed in area 602, such as Figure 6Bshown and as described above.

[0144] Note that while the foregoing examples and subsequent examples may focus on a description of a particular event detected in the second area of ​​the display (e.g., detecting user gaze in the second area, detecting a cursor moving to the second area, and / or interacting with a selectable element in the second area), this is done for illustrative purposes and is not intended to be limiting in any way. In some embodiments, other events may be detected, and it may be determined whether these events meet the system movement criteria. It should be understood that any event indicating a user's potential and / or actual interaction with the second area (e.g., interaction with a UI element in the second area) may be considered and / or may trigger an event in the second area.

[0145] Fig. 7A and Figure 7B Another example of a technique for controlling display of a representation of a system bar in a display within a CGR environment according to aspects of the present disclosure is shown. Specifically, Fig. 7A and Figure 7B An example is shown in which display of a representation of a system bar is controlled based on an event that satisfies system bar movement criteria including interaction with a selectable UI element.

[0146] Fig. 7A A CGR environment 290 including a display 600 is shown. As described above, the display 600 can be a representation of a display (e.g., a virtual representation of a physical display), a display area, or a physical display. The display 600 can include regions 601, 602, and 603. Also as described above, in some embodiments, some or all of the regions 601, 602, and 603 can be regions in a representation of a physical display (e.g., a virtual display), or can be a physical display. In some embodiments, the display 600 can be a representation of a single continuous display (e.g., a virtual display or a physical display), and the regions 601, 602, and 603 can be determined to be regions in the same representation of a physical display.

[0147] like Fig. 7A As shown, the first representation 720 of the system bar can be displayed by the electronic device 200 in the bottom area of ​​the area 601. In an embodiment, the first representation 720 can be displayed anywhere (e.g., bottom, top, left, right, center, etc.) within the area 601, as described above. In an embodiment, the representation of the system bar may not currently be displayed in the area 602.

[0148] Selectable UI element 630 may be displayed in area 602 of display 600. In an embodiment, area 602 may include multiple selectable UI elements, and selectable UI element 630 may be included in the multiple selectable UI elements. In an embodiment, selectable UI element 630 may include an icon, an affordance, a mark, a button, an interactive UI element, and / or any other UI element configured to allow a user to select the UI element. Selectable UI element 630 may be associated with an application and may be configured to cause the associated application to open in response to being selected.

[0149] For example, Figure 7B As shown in , a user (e.g., user 202) can interact with selectable UI element 630. In an embodiment, it can be determined that the interaction indicates a selection of UI element 630. In response to the selection of selectable UI element 630, the associated application (e.g., application 631) can be opened and / or displayed in area 602.

[0150] In an embodiment, selection of selectable UI element 630 and / or opening of application 631 may be detected. In an embodiment, selection of selectable UI element 630 and / or opening of application 631 may be determined to be an event in area 602 of display 600. Note that in an embodiment, detecting an event in area 602 of display 600 and other steps described below may occur when the system bar is not selected (e.g., when a user (e.g., user 202) may not have selected and / or interacted with the system bar (e.g., via first representation 720)).

[0151] In an embodiment, in response to detecting an event in area 602, a determination is made as to whether the event satisfies system bar movement criteria. Figure 7B In the example shown, when an event in area 602 is determined to be a selection of a selectable UI element in area 602 (e.g., selectable UI element 630) and / or the opening of an application in area 602 (e.g., application 631), it can be determined that the event in area 602 meets the system bar movement criteria.

[0152] In response to determining that an event in area 602 satisfies system bar movement criteria (e.g., a selection of selectable UI element 630 and / or an opening of application 631 is detected in area 602), a second representation of the system bar (e.g., second representation 721) is displayed in area 602, such as Figure 7BIn an embodiment, the second representation 721 may have the same size as the first representation 720. In an embodiment, the second representation 721 may be similar to the first representation 720. For example, the second representation 721 may include the same theme, the same information, the same interactive elements (e.g., application icons, affordances, etc.), etc. In some embodiments, the second representation 721 may be different from the first representation 720.

[0153] In an embodiment, the first representation can continue to be displayed in area 601 while the second representation 721 is displayed in area 602. In other embodiments, the first representation 720 can cease to be displayed in area 601 while the second representation 721 is displayed in area 602.

[0154] In some embodiments, the second representation 721 can be displayed in area 602 in the same orientation (horizontal, vertical, angled, etc.) and / or in the same region (bottom, top, left, and / or right) as the first representation 720 in area 601. In embodiments, the second representation 721 can be displayed in a different orientation and / or in a different region than the first representation 720 in area 601.

[0155] Fig. 8A and Figure 8B Another example of a technique for controlling display of a representation of a system bar in a display within a CGR environment according to aspects of the present disclosure is shown. Specifically, Fig. 8A and Figure 8B An example is shown in which display of a representation of a system bar in an area of ​​a representation of a display is controlled based on detected events that meet system bar movement criteria.

[0156] Fig. 8A A CGR environment 290 is shown that includes a display 600. As described above, the display 600 can be a representation of a display (e.g., a virtual representation of a physical display), a display area, or can be a physical display. In an embodiment, the display 600 can represent a single display or can represent an aggregate display that includes more than one display. Fig. 8A In the example shown, display 600 may be a representation of a single display. Display 600 may include regions 670 and 680. Regions 670 and 680 may be regions within the single representation of display 600. Region 670 may be defined by border 671, and region 680 may be defined by border 681. In various aspects, borders 671 and 681 may not be displayed on display 600 or may be displayed in display 600.

[0157] In various aspects, display 600 can be a curved display. In some embodiments, display 600 can be sized so that display 600 can partially or completely surround a user (e.g., user 202) located in front of display 600, so that the user can perceive display 600 as a display surrounding him or her.

[0158] like Fig. 8A As shown, the first representation 820 of the system bar can be displayed by the electronic device 200 in the bottom area of ​​the area 670. In an embodiment, the first representation 820 can be displayed anywhere (e.g., bottom, top, left, right, center, etc.) within the area 670, as described above. In an embodiment, the representation of the system bar may not currently be displayed in the area 680.

[0159] Figure 8B An event 840 is shown detected in area 680 of display 600. In an embodiment, event 840 may be an event indicating potential and / or actual interaction of a user (e.g., user 202) with area 680. In an embodiment, it may be determined whether event 840 satisfies system bar movement criteria. For example, it may be determined whether event 840 is at least one of: detection of a user gaze in area 680, detection of a cursor moving into area 680, detection of an interaction with a selectable element in area 680, opening of an application in area 680, any other event indicating potential and / or actual interaction of a user with area 680, and / or any combination thereof.

[0160] Note that in an embodiment, detecting event 840 in area 680 of display 600 and the other steps described below may occur when the system bar is not selected (e.g., when a user (e.g., user 202) may not have selected the system bar and / or interacted with the system bar (e.g., via first representation 720)).

[0161] In response to determining that event 840 in region 680 meets the system bar movement criteria, a second representation of the system bar (eg, second representation 821) is displayed in region 680, such as Figure 8B In an embodiment, the second representation 821 may have the same size as the first representation 820. In an embodiment, the second representation 821 may be similar to the first representation 820. For example, the second representation 821 may include the same theme, the same information, the same interactive elements (e.g., application icons, affordances, etc.), etc. In some embodiments, the second representation 821 may be different from the first representation 820.

[0162] In some embodiments, the second representation 821 can be displayed in area 680 in the same orientation (horizontal, vertical, angled, etc.) and / or in the same region (bottom, top, left, and / or right) as the first representation 820 in area 670. In embodiments, the second representation 821 can be displayed in a different orientation and / or in a different region than the first representation 820 in area 670.

[0163] In an embodiment, the first representation 820 can continue to be displayed in area 670 while the second representation 821 is displayed in area 680. In other embodiments, the first representation 820 can cease to be displayed in area 670 while the second representation 821 is displayed in area 680.

[0164] In some embodiments, the second representation 821 may be displayed at a location within the area 680 based on the location of the event 840. For example, the event 840 may be detected in the area 680 and may be determined to be located at a specific location within the area 680 (e.g., a location in the area 680 where the user's gaze may be directed, a location in the area 680 where the cursor may move to, a location in the area 680 where a UI element may interact with the user, a location in the area 680 where an application may be opened, etc.). In these embodiments, the second representation 821 may be displayed at the location within the area 680 where the event 840 may be detected. In other embodiments, the second representation 821 may be displayed on the bottom side, top side, right side, or left side of the area 680, regardless of where the location within the area 680 where the event 840 may be detected is.

[0165] Fig. 9 900 is a flow chart illustrating a method 500 for controlling display of a representation of a system bar in a display within a CGR environment. In some embodiments, the method 900 may be performed by the system 100 or a portion of the system 100. In some embodiments, the method 900 may be performed by one or more external systems and / or devices. In some embodiments, the method 900 may be performed by the system 100 (or a portion of the system 100) in conjunction with one or more external systems and / or devices.

[0166] At block 902, the system displays a virtual display area representing a physical display within a CGR environment via a display of an electronic device (e.g., a wearable electronic device, an HMD device, etc.). For example, the virtual display area representing the physical display may be displayed via a first display (e.g., a left-eye display panel) or a second display (e.g., a second-eye display panel) of the electronic device.

[0167] In various aspects, the virtual display area may represent a curved display. In some embodiments, the virtual display area may represent a curved display having a size such that the display can partially or completely surround a user located in front of the represented display, so that the user can perceive the display as a display surrounding him or her.

[0168] In an embodiment, the virtual display area may be a representation of a single continuous physical display. In other embodiments, the virtual display area may be a representation of more than one physical display (e.g., an aggregation of more than one physical display). In an embodiment, the virtual display area may include at least one area configured to display UI elements and to display a representation of a system bar.

[0169] At block 904, the system displays a first representation of a system bar in a first area of ​​the virtual display. In an embodiment, the first representation of the system bar may be a virtual representation displayed within the CGR environment. In an embodiment, the first representation may be displayed at any location within the first area (e.g., bottom, top, left, right, center, etc.). In an embodiment, a representation of the system bar may not currently be displayed in another area of ​​the virtual display area. In an embodiment, the system bar may include multiple affordances, icons, and / or selectable UI elements.

[0170] At box 906, the system detects an event in the second area of ​​the virtual display. In various aspects, the second area of ​​the virtual display may be outside the first area of ​​the virtual display. In an embodiment, the detected event may be an event indicating a potential and / or actual interaction of the user with the second area. In an embodiment, it may be determined whether the event satisfies the system bar movement standard. For example, it may be determined whether the event is at least one of the following: detection of a user's gaze in the second area, detection of a cursor moved into the second area, detection of interaction with a selectable element in the second area, opening of an application in the second area, any other event indicating a potential and / or actual interaction of the user with the second area, and / or any combination thereof.

[0171] At block 908, when it is determined that the event satisfies the system bar movement criteria, the system displays a second representation of the system bar in the second area. In an embodiment, detecting the event in the second area and displaying the second representation of the system bar when the event satisfies the system bar movement criteria may occur when the system bar is not currently selected (e.g., when a user (e.g., user 202) may not have selected and / or interacted with the system bar (e.g., via the first representation)).

[0172] In an embodiment, the second representation of the system bar may have the same size as the first representation. In an embodiment, the second representation may be similar to the first representation (e.g., the second representation may include the same subject matter, the same information, the same interactive elements (e.g., application icons, affordances, etc.) as the first representation, etc.). In some embodiments, the second representation may be different from the first representation.

[0173] In some embodiments, the second representation can be displayed in the second region in the same orientation (horizontal, vertical, angled, etc.) and / or in the same region (bottom, top, left, and / or right) as the first representation in the first region. In embodiments, the second representation can be displayed in a different orientation and / or in a different region than the first representation in the first region.

[0174] In an embodiment, the first representation may continue to be displayed in the first area, while the second representation is displayed in the second area. In other embodiments, the first representation may stop being displayed in the first area, while the second representation is displayed in the second area.

[0175] In an embodiment, the first representation may continue to be displayed in the first area as a semi-transparent object, while the second representation may be displayed in the second area as an opaque object. In an embodiment, when it is determined that the user is looking at or the cursor moves into the first area, the first representation may be transformed from semi-transparent to opaque.

[0176] Aspects of the present disclosure relate to the following systems and techniques that provide functionality for controlling the display of a virtual display window based on an articulation angle of a physical device within a CGR environment. In an embodiment, controlling whether a virtual display window appears (e.g., displayed within a CGR environment) may include displaying the virtual display window by an adjacent physical device (e.g., a display of an adjacent physical device) when the articulation angle of the physical device reaches a threshold articulation angle.

[0177] FIG. 10A to FIG. 10J Exemplary techniques for controlling the display of a virtual display window based on an articulation angle of a physical device within a CGR environment are shown in accordance with aspects of the present disclosure.

[0178] Fig. 10AA physical device 1000 is shown within a CGR environment 1090. In an embodiment, the physical device 1000 can be a device having a hinged function, wherein at least two parts of the physical device can be configured to rotate around a hinge point and come together or separate from each other. For example, the physical device 1000 can include a part 1020 and a part 1021, both of which can be configured to hinge around a hinge point 1010. In an embodiment, the parts 1020 and 1021 can be separated from each other while remaining attached at the hinge point 1010. In an embodiment, the physical device 1000 can be any device having such a hinged function (e.g., a laptop, a smart phone, a folding tablet, a briefcase, etc.).

[0179] In an embodiment, the physical device 1000 can be part of the CGR environment 1090. For example, as described above, the physical device 1000 can be presented (e.g., presented to a user (e.g., user 202)) via a transparent or semi-transparent display of the electronic device 200, or can be presented via a pass-through video as described above, so that the actual physical device 1000 can be perceived.

[0180] In an embodiment, the articulation angle of physical device 1000 may be measured by measuring the separation angle between portion 1020 and portion 1021. For example, Fig. 10B As shown, the current articulation angle of the physical device 1000 may be determined to be zero. That is, the portion 1020 and the portion 1021 may be together.

[0181] Fig. 10C The hinge action on the physical device 1000 is shown. In an embodiment, the portion 1021 or the portion 1021 or both can be hinged about the hinge point 1010 to separate from each other. Fig. 10C In the example shown, portion 1020 can be hinged about hinge point 1010 in direction 1040 away from portion 1021. For example, opening physical device 1000 can be determined by the hinge action of portion 1020 away from portion 1021. The hinge action can create a hinge angle 1050 between portions 1020 and 1021.

[0182] Fig. 10D The articulation motion of portion 1020 is shown rotating about hinge point 1010 and moving away from portion 1021 to create a current articulation angle 1050 between portion 1020 and portion 1021. In an embodiment, current articulation angle 1050 can be measured. For example, an image sensor can be used to track the location of portions 1020 and 1021 relative to each other to measure current articulation angle 1050. In other specific implementations, a sensor in physical device 1000 can be used to measure current articulation angle 1050, and physical device 1000 can provide data representing the articulation angle to device 200.

[0183] In an embodiment, a threshold articulation angle 1051 may be provided. The threshold articulation angle 1051 may be a predetermined articulation angle according to which the current articulation angle may be measured. In an embodiment, the threshold articulation angle 1051 may be an angle relative to portion 1021, which is the bottom portion of the physical device 1000. In an embodiment, the threshold articulation angle 1051 may be displayed as a virtual object within the CGR environment (e.g., displayed as a line, dashed line, dot, etc.). In some embodiments, the threshold articulation angle 1051 may not be displayed.

[0184] In an embodiment, it may be determined whether the measured current articulation angle exceeds a threshold articulation angle. For example, the current articulation angle 1050 may be compared with the threshold articulation angle 1051 to determine whether the current articulation angle 1050 exceeds the threshold articulation angle 1051. When it is determined that the current articulation angle does not exceed the threshold articulation angle, the virtual display window may not be displayed. In an embodiment, not displaying the virtual display window may include abandoning displaying the virtual display window according to the following description. Conversely, when it is determined that the measured articulation angle exceeds the threshold articulation angle, the virtual display window may be displayed.

[0185] like Fig. 10C and Fig. 10D As shown, the current articulation angle 1050 does not exceed the threshold articulation angle 1051. According to determining that the current articulation angle 1050 does not exceed the threshold articulation angle 1051, the virtual display window may not be displayed.

[0186] Fig. 10E and Fig.10F Another articulation motion is shown in which portion 1020 rotates about articulation point 1010 and moves further away from portion 1021 in direction 1040. In this example, current articulation angle 1050 has increased. In an embodiment, it may be determined whether current articulation angle 1050 now exceeds threshold articulation angle 1051. For example, current articulation angle 1050 may be compared to threshold articulation angle 1051 to determine whether current articulation angle 1050 exceeds threshold articulation angle 1051. Fig. 10E and Fig.10F As shown, the current articulation angle 1050 does not exceed the threshold articulation angle 1051. According to determining that the current articulation angle 1050 does not exceed the threshold articulation angle 1051, the virtual display window may not be displayed.

[0187] In some embodiments, the physical device 1000 may be configured with a display (e.g., display 1030). The physical device 1000 may also be configured such that the display 1030 may be inactive and / or turned off (e.g., not displaying content or displaying a completely dark or black screen) until a specific articulation angle is reached. When the articulation angle of the physical device 1000 reaches a specific articulation angle, the display 1030 may be turned on or activated. Fig. 10EIn the example shown, articulation angle 1050, although it may not exceed threshold articulation angle 1051, has reached an articulation angle for activating display 1030. In this case, display 1030 may be activated. In an embodiment, activating display 1030 may include displaying at least a portion of content in display 1030. For example, display 1030 may be activated, and content 1031 and content 1032 may be displayed in display 1030. In an embodiment, content 1031 and content 1032 may be different content, may be parts of content, or may be different parts of the same content.

[0188] Figure 10G and Fig. 10H Still another articulation motion is shown in which portion 1020 rotates about hinge point 1010 and moves even further away from portion 1021. In this example, current articulation angle 1050 has increased further. In an embodiment, it may be determined whether current articulation angle 1050 now exceeds threshold articulation angle 1051. For example, current articulation angle 1050 may be compared to threshold articulation angle 1051 to determine whether current articulation angle 1050 exceeds threshold articulation angle 1051. Fig. 10E and Fig.10F As shown, the current articulation angle 1050 exceeds the threshold articulation angle 1051. Based on determining that the current articulation angle 1050 exceeds the threshold articulation angle 1051, a virtual display window may be displayed.

[0189] In an embodiment, the appearance of a virtual display window may include displaying at least one virtual display window relative to a physical device within the CGR environment 1090. In an embodiment, the virtual display window may be displayed in any area within the CGR environment 1090 that is conducive to configuring the virtual display window to be connected to a physical device. In an embodiment, for example, in the case where the physical device may not have a display, the virtual display window may be displayed to represent the display of the physical device. In some embodiments, for example, in the case where the physical device may have a display, the virtual display window may be displayed as a display connected to the physical device (e.g., as an extension of the display of the physical device). In an embodiment, the size of the virtual display window may be the same as the size of the display of the physical device. In other embodiments, the size of the virtual display window may be different from the size of the display of the physical device. In these cases, the size of the virtual display window may be determined based on system preferences and / or settings, and / or may be determined based on environmental features. For example, the size of the virtual display window may be determined to adapt to a desk (e.g., a virtual desk or a physical desk) on which the physical device 1000 may be placed within the CGR environment.

[0190] exist Figure 10GIn the example shown, a virtual display window 1060 may be displayed in an area above the physical device 1000 and connected to the display 1030 of the physical device 1000 when it is determined that the current articulation angle 1050 exceeds the threshold articulation angle 1051. In this example, the virtual display window 1060 extends the display 1030 upward. In some embodiments, additional virtual display windows may be displayed. For example, a virtual display window 1061 may be displayed on the left side of the physical device 1000 to extend the display 1030 to the left, and / or a virtual display window 1062 may be displayed on the right side of the physical device 1000 to extend the display 1030 to the right. It should be understood that additional or fewer virtual display windows may be displayed. For example, in an embodiment, additional virtual display windows may be displayed so that the entire area including the virtual display window and the physical display 1030 represents a single coherent extended display.

[0191] In an embodiment, displaying the virtual display window 1060 and stopping displaying the virtual display window 1060 as described below may be performed gradually. In an embodiment, displaying the virtual display window 1060 may include initially displaying the virtual display window 1060 with a high transparency (e.g., a non-zero transparency) and gradually reducing the transparency of the virtual display window 1060 until the virtual display window 1060 becomes opaque. In some embodiments, the transparency of the virtual display window 1060 may depend on the hinge angle of the physical device 1000. In an embodiment, the size of the virtual display window 1060 may also increase gradually, and in some embodiments, the size of the virtual display window 1060 may depend on the hinge angle.

[0192] In an embodiment, at least a portion of the content displayed in the display 1030 of the physical device 1000 may be displayed in the virtual display window. Figure 10GAs shown, content 1031 previously displayed in display 1030 may be displayed in virtual display window 1060. In an embodiment, it may be automatically determined which portion of the content previously displayed in the display of the physical device may be displayed in the virtual display window when the virtual window is displayed. For example, content 1031 and content 1032 may be portions of content representing a coherent element (e.g., an image, an application window, etc.). In this case, when virtual display window 1060 is displayed in response to the articulation angle of physical device 1000 exceeding threshold articulation angle 1051, the size of the coherent element may increase, portion 1031 may be moved to virtual display window 1060, and portion 1031 may remain displayed in display 1030. In an embodiment, content 1031 and content 1032 may not be portions of content representing a coherent element, but at least a portion of the content may still be moved to virtual display window 1060. In some embodiments, a user (e.g., user 202) may select content to be moved to virtual display window 1060.

[0193] In an embodiment, content that has not been previously displayed in the display of the physical device may be displayed in the virtual display window. For example, the content 1033 may not have been previously displayed in the display 1030 of the physical device 1000. In this case, when the virtual display window 1060 is displayed in response to the articulation angle of the physical device 1000 exceeding the threshold articulation angle 1051, the content 1033 may be displayed in the virtual display window 1060.

[0194] In an embodiment, the content displayed in the virtual window can be simultaneously displayed (e.g., shared) in a physical display. For example, content 1033 can be displayed in display 1030 of physical device 1000. In response to user input (e.g., selection of an interactive element, a drag gesture, a click, etc.), content 1033 can be simultaneously displayed (e.g., shared) in physical display 1030. In this way, a user viewing CGR environment 1090 and perceiving virtual display window 1060 via device 200 can share virtual content from the CGR environment via a real-world physical display (e.g., display 1030) to other users who may not be perceiving CGR environment 1090.

[0195] In an embodiment, the virtual display window 1060 may be displayed at a particular tilt angle relative to the physical device 1000. In an embodiment, the tilt angle of the virtual display window 1060 may be the same as the hinge angle of the portion 1020. In other embodiments, the tilt angle of the virtual display window 1060 may be fixed and / or predetermined. In these cases, the tilt angle of the virtual display window 1060 may remain the same even if the hinge angle of the physical device changes (e.g., as long as the hinge angle of the physical device exceeds a threshold hinge angle).

[0196] Fig.10I and Fig.10J An articulation motion is shown, wherein portion 1020 rotates about hinge point 1010 in direction 1041 toward portion 1021. In this example, based on direction 1041, it can be determined that physical device 1000 is being closed. In this example, it can be determined that current articulation angle 1050 has decreased. In an embodiment, it can be determined whether current articulation angle 1050 exceeds threshold articulation angle 1051. For example, current articulation angle 1050 can be compared to threshold articulation angle 1051 to determine whether current articulation angle 1050 exceeds threshold articulation angle 1051. Fig.10J As shown, the current articulation angle 1050 no longer exceeds the threshold articulation angle 1051. Based on determining that the current articulation angle 1050 does not exceed the threshold articulation angle 1051, the display of the virtual display window (e.g., virtual display windows 1060, 1061, 1062 and / or any combination thereof) is stopped.

[0197] In an embodiment, ceasing to display the virtual display window may include moving at least a portion of the content displayed in the virtual display window to the display 1030 of the physical device 1000. In an embodiment, the content moved to the display 1030 may include content that was previously displayed in the display 1030 but moved to the virtual display window, and / or may include content that was displayed in the virtual display window but was not previously displayed in the display 1030.

[0198] In an embodiment, as described above, stopping displaying the virtual display window may be performed gradually. For example, the transparency of the virtual display window may be gradually increased until the virtual display window is no longer visible. In an alternative or additional embodiment, stopping displaying the virtual display window may include gradually reducing the size of the virtual display window until the window is no longer visible.

[0199] Fig.11 1 is a flow chart showing a method 1100 for controlling the display of a virtual display window based on an articulation angle of a physical device within a CGR environment. In some embodiments, the method 1100 may be performed by the system 100 or a portion of the system 100. In some embodiments, the method 1100 may be performed by one or more external systems and / or devices. In some embodiments, the method 1100 may be performed by the system 100 (or a portion of the system 100) in combination with one or more external systems and / or devices. In an embodiment, the method 1100 may be performed by an electronic device (e.g., a wearable electronic device, an HMD device, etc.).

[0200] At block 1102, the system receives and / or acquires data representing an articulation angle of a physical object. In an embodiment, the articulation angle of a physical device may be measured using an image sensor, a sensor in the physical device, using computer vision techniques that may employ a front-facing camera, and the like.

[0201] In an embodiment, in response to acquiring data representing an articulation angle of a physical device, it may be determined whether the articulation angle of the physical device exceeds a threshold articulation angle. For example, the articulation angle may be compared to a threshold articulation angle to determine whether the articulation angle exceeds the threshold articulation angle. When it is determined that the articulation angle does not exceed the threshold articulation angle, the virtual display window may not be displayed. In an embodiment, not displaying the virtual display window may include forgoing displaying the virtual display window.

[0202] At block 1104, in response to acquiring data representing an articulation angle of the physical device, when it is determined that the articulation angle exceeds a threshold articulation angle, the system displays a virtual display window relative to the physical device via a display of an electronic device (e.g., a wearable electronic device, an HMD device, etc.). For example, a virtual display area relative to the physical device may be displayed via a first display (e.g., a left-eye display panel) or a second display (e.g., a second-eye display panel) of the electronic device.

[0203] In an embodiment, the virtual display window may be displayed at any area within the CGR environment that is conducive to configuring the virtual display window to be displayed as connected to the physical device. In an embodiment, the virtual display window may be displayed as a display connected to the physical device (e.g., as an extension of the display of the physical device). In an embodiment, the size of the virtual display window may be the same as the size of the display of the physical device. In other embodiments, the size of the virtual display window may be different from the size of the display of the physical device.

[0204] In embodiments, the virtual display window may be displayed in the region above the physical device. In some embodiments, additional or alternative virtual display windows (e.g., on the left or right side of the physical device) may be displayed.

[0205] In an embodiment, displaying the virtual display window and stopping displaying the virtual display window may be performed gradually. For example, displaying the virtual display window may include initially displaying the virtual display window with high transparency and gradually reducing the transparency of the virtual display window until the virtual display window becomes opaque. In an embodiment, the size of the virtual display window may be gradually increased until a determined size is reached when the virtual display window is displayed.

[0206] In an embodiment, at least a portion of the content displayed in the display of the physical device may be displayed in the virtual display window. For example, when the virtual display window is displayed in response to the articulation angle of the physical device exceeding the threshold articulation angle, at least a portion of the content previously displayed in the display of the physical device may be moved to the virtual display window.

[0207] In an embodiment, the content displayed in the virtual display window can be simultaneously displayed (e.g., shared) in a physical display. For example, in response to user input (e.g., selection of an interactive element, a drag gesture, a click, etc.), the content displayed in the virtual display window can be simultaneously displayed (e.g., shared) in a display of a physical device. In this way, a user who is presented with a CGR environment and can perceive the virtual display window via an electronic device can share virtual content from the CGR environment with other users who may not be able to perceive the CGR environment via a real-world physical display.

[0208] In an embodiment, the virtual display window can be displayed at a specific tilt angle relative to the physical device. In an embodiment, the tilt angle of the virtual display window can be maintained even if the hinge angle and / or tilt angle of the physical device changes.

[0209] In an embodiment, in response to detecting that the articulation has changed, it can be determined whether the new articulation angle of the physical device exceeds a threshold articulation angle. When it is determined that the new articulation angle does not exceed the threshold articulation angle, the display of the virtual display window can be stopped.

[0210] In an embodiment, ceasing to display the virtual display window may include moving at least a portion of the content displayed in the virtual display window to a display of a physical device. In an embodiment, the content moved to the display of the physical device may include content that was previously displayed in the display of the physical device but moved to the virtual display window, and / or may include content that was displayed in the virtual display window but was not previously displayed in the display of the physical device.

[0211] In an embodiment, as described above, stopping displaying the virtual display window may be performed gradually. For example, the transparency of the virtual display window may be gradually increased until the virtual display window is no longer visible. In an alternative or additional embodiment, stopping displaying the virtual display window may include gradually reducing the size of the virtual display window until the window is no longer visible.

[0212] Aspects of the present disclosure relate to the following systems and techniques, which provide functionality for controlling the display of UI elements based on gaze within a CGR environment. In an embodiment, controlling the display of UI elements (e.g., a cursor, an application window, etc.) based on gaze within a CGR environment may include: displaying the UI element at a first position within the CGR environment; detecting the current gaze of a user within the CGR environment; and upon receiving a request that meets a movement criterion, displaying the UI element at an introduction position and then moving the UI element from the introduction position to a second position determined based on the user's gaze.

[0213] FIG. 12A to FIG. 12E Exemplary techniques for controlling the display of UI elements based on gaze within a CGR environment are shown according to aspects of the present disclosure. Specifically, Fig. 12A 1 shows a user 202 wearing an electronic device 200, which is configured to allow the user 202 to view a CGR environment 1290. As described above, in some embodiments, the electronic device 200 may be similar to the one described above with reference to FIG. Figure 1A and Figure 1B An electronic device 100a is described.

[0214] like Fig. 12A As shown, CGR environment 1290 includes display 1200. In an embodiment, display 1200 can be a representation of a display (e.g., a physical display, such as a monitor or screen of a physical computer), such as a computer-generated simulation of a physical display. In some embodiments, the computer-generated simulation of a display can be a computer-generated display area that has no corresponding physical counterpart and / or representation. In these cases, the computer-generated simulation of a physical display can be a display area that may not simulate a physical display. In some implementations, the computer-generated simulation of a physical display can be superimposed on a physical environment or a representation thereof (e.g., as in an MR implementation), so that user 202 can perceive the computer-generated simulation of a physical display superimposed on the physical environment. In other implementations, the computer-generated simulation of a physical display can be a virtual object that is part of a complete CGR environment (e.g., as in a VR application).

[0215] In some embodiments, display 1200 may represent a physical display presented to user 202 via electronic device 200. For example, display 1200 may include an actual physical display and may not be a computer-generated simulation of a display. In this case, as described above with respect to Figure 2AAs described with display 204 of the embodiment, display 1200 can be presented to user 202 so that user 202 can view a physical display instead of a computer-generated simulation of a physical display. In yet other embodiments, presenting a physical display to user 202 can be performed by superimposing realistic images and / or videos of the physical display on a simulated environment.

[0216] In various aspects, the display 1200 can be a curved display. In some embodiments, the display 1200 can extend so that the display 1200 can partially or completely surround the user 202 so that the user 202 can perceive the display 1200 as a display surrounding him or her.

[0217] In an embodiment, an input mechanism may be included to facilitate user interaction. For example, the input mechanism may include a mechanism for a user (e.g., user 202) to interact with the display 1200 or with a UI element of the display 1200. In an embodiment, the input mechanism may include a representation of the user's appendage (e.g., a virtual representation of a finger, hand, leg, foot, etc.), the user's gaze (e.g., head gaze, eye gaze, etc.), an input device (e.g., a mouse, a stylus, a pen, a touch-sensitive surface, an image sensor, etc.), etc. In an embodiment, the representation of the user's appendage may include a virtual representation of the appendage and / or may include data representing characteristics of the appendage within the CGR environment (e.g., location, orientation, distance from a particular point, etc.). In various aspects, the input mechanism may be detected using an input sensor (e.g., a touch-sensitive surface, an image sensor, etc.) configured to perform hand tracking, head gaze tracking, eye gaze tracking, finger tracking, etc.

[0218] In an embodiment, a UI element associated with an input mechanism may be provided. Fig. 12A As shown, UI element 1210 may be provided and displayed in display 1200. In an embodiment, UI element 1210 may be a cursor, an application window, etc. In this example, the current location of UI element 1210 within CGR 1290 may be location 1220. Additionally, the input mechanism may include gaze 1250 of user 202. In particular, it is noted that gaze 1250 may have a direction associated therewith, and thus gaze 1250 may be directed to a specific location within CGR environment 1290.

[0219] In an embodiment, gaze and gaze direction may be detected. For example, an input sensor and / or an image sensor may be used to detect a gaze 1250 of the user 202, which may be a head gaze (e.g., the direction the user's head is facing), an eye gaze (e.g., the direction the user's eyes are looking at), a combination thereof, etc.

[0220] Fig. 12BIt is shown that the gaze 1250 of the user 202 has moved. In particular, it is shown that the gaze 1250 is directed to the location 1221. The location 1221 can be a location within the area 1201 of the display 1200. As shown, the area 1201 can be defined by the boundary 1202.

[0221] In an embodiment, an input sensor and / or an image sensor may be used to detect a gaze 1250 directed toward a location 1221 in the zone 1201. Thus, it may be determined that the current gaze of the user 202 (e.g., gaze 1250) may be directed toward the location 1221 and / or toward the zone 1201. Furthermore, as described above, the gaze 1250 may be a head gaze, in which case it may be determined that the user's head is directed toward the location 1221 and / or the zone 1201, or it may be an eye gaze, in which case it may be determined that the user's eyes are directed toward the location 1221 and / or the zone 1201.

[0222] In an embodiment, gaze 1250 can be used to determine a second location within CGR environment 1290. The second location can be a location to which UI element 1210 is to be moved. In an embodiment, the second location can be a point (e.g., location 1221) or can be a zone (e.g., zone 1201), and can be determined based on the location pointed to by gaze 1250. For example, when gaze 1250 points to location 1221 and zone 1201, the second location can be determined to be location 1221 or zone 1201. In some embodiments, for example, when the second location is determined to be a zone, the specific location in the zone where UI element 1210 is to be moved can be an alignment point. In these cases, UI element 1210 can be moved to the zone (e.g., zone 1201), but can be aligned to the alignment location (e.g., the location pointed to by the user's gaze or the default location). In an embodiment, the second location can be based on the user's determination of the UI element to be viewed. For example, based on gaze 1250 , it can be determined that user 202 may be viewing UI elements at location 1221 (e.g., specific selectable UI elements, status UI elements (e.g., time icons, etc.), system elements (e.g., system display icons, etc.), etc.).

[0223] In some embodiments, the second location may be determined based on the dwell time of gaze 1250. For example, gaze 1250 may be detected at area 1201, and it may be determined that gaze 1250 is directed to location 1221. In this case, when it is determined that gaze 1250 remains directed to (e.g., remains at) location 1221 for at least a predetermined period of time, location 1221 may be determined to be the second location. If gaze 1250 does not remain directed to location 1221 for at least a predetermined period of time, the second location is not determined.

[0224] In some embodiments, the second location may be determined based on gaze 1250 pointing to a specific UI element. For example, when gaze 1250 points to a specific UI element in zone 1201, zone 1201 or a location within zone 1201 (e.g., an alignment location or a location within zone 1201 to which gaze 1250 may be directed) may be determined as the second location. In an embodiment, a specific UI element may be provided in zone 1201, wherein the specific UI element may be configured to indicate a request to move UI element 1210 to zone 1201 and / or to a location where the specific UI element is displayed. In these cases, when gaze 1250 points to a specific UI element (e.g., for at least a predetermined amount of time) and zone 1201 or a location within zone 1201 (e.g., an alignment location or a location of a specific UI element) may be determined as the second location, the specific UI element may be determined to be selected.

[0225] In an embodiment, it is determined that the mobile UI element 1210 may be based on a request for the mobile UI element 1210 that meets the mobile standard. For example, a request for the mobile UI element 1210 may be received. In an embodiment, the request for the mobile UI element 1210 may include the actual movement of the UI element 1210. For example, an input mechanism (e.g., a mouse, stylus, pen, touch-sensitive surface, image sensor, etc.) associated with the UI element 1210 or configured to control the UI element 1210 may be moved (e.g., by the user 202), which naturally may result in a corresponding movement of the UI element 1210 within the display 1200. In an embodiment, the movement of the input mechanism may be determined to be a request for the mobile UI element 1210. In an embodiment, the request for the mobile UI element 1210 may include clicking and holding the input mechanism. In an embodiment, it may be determined whether the request for the mobile UI element 1210 meets the mobile standard.

[0226] In an embodiment, the request to move UI element 1210 may be determined to satisfy the movement criteria when a second location has been determined based on the user's gaze (e.g., gaze 1250). In these cases, the determination of the second location may indicate that the user's gaze (e.g., gaze 1250) has been detected as pointing to region 1201 and / or location 1221, detected as staying at a location for at least a predetermined period of time, and / or detected as pointing to a specific element in region 1201, as described above.

[0227] In an embodiment, when the movement of the additional input mechanism exceeds a threshold movement amount, it may be determined that the request to move UI element 1210 satisfies the movement criteria. Fig. 12BAs shown, gaze 1250 may point to area 1201 and / or location 1221. In this example, the input mechanism may move an input mechanism movement distance. The input mechanism movement distance may be compared to a threshold movement amount. If the input mechanism movement distance exceeds the threshold movement amount, it may be determined that the request to move UI element 1210 meets the movement criteria. Conversely, if the input mechanism movement distance does not exceed the threshold movement amount, it may be determined that the request to move UI element 1210 does not meet the movement criteria.

[0228] In an embodiment, a request to move UI element 1210 may be determined to satisfy the movement criteria when the movement of the additional input mechanism is in a particular direction. Fig. 12B As shown, gaze 1250 may be directed to area 1201 and / or location 1221. In this example, the input mechanism may move toward area 1201 and location 1221 in direction 1231. In this example, when the direction of movement of the input mechanism is toward the location and / or area directed by gaze 1250, it may be determined that the request to move UI element 1210 meets the movement criteria. On the contrary, if the direction of movement of the input mechanism is not toward the location and / or area directed by gaze 1250, it may be determined that the request to move UI element 1210 does not meet the movement criteria.

[0229] In an embodiment, when additional gaze 1250 is directed to an application of a specific type, it may be determined that the request to move UI element 1210 meets the move criteria. For example, gaze 1250 may be directed to an application displayed in area 1201. In this example, an input mechanism associated with UI element 1210 may be moved. If the application directed by gaze 1250 is of a specific type, it may be determined that the request to move UI element 1210 meets the move criteria. Conversely, if the application directed by gaze 1250 is not of a specific type, it may be determined that the request to move UI element 1210 does not meet the move criteria.

[0230] In an embodiment, as described above, when it is determined that the request to move UI element 1210 meets the move criteria, UI element 1210 may be moved to the second location. Fig. 12BAs shown, according to the above description, the second location can be determined to be location 1221. Also as shown, there is a distance 1230 between the current location 1220 of the UI element 1210 and the second location (e.g., location 1221). Therefore, in order to move the UI element 1210 to the location 1221, it is necessary to traverse the distance 1230. It should be understood that moving the UI element 1210 from the location 1220 to the location 1221 may include dragging the UI element 1210 across the display 1200 to the second location (e.g., location 1221) in accordance with the movement of the input mechanism (e.g., a mouse, a stylus, a pen, a touch-sensitive surface, an image sensor, etc.). However, the dragging movement may take a long time, and in some cases, the user may lose track of the location of the UI element 1210. This is particularly noteworthy when the display in which the UI element is displayed is a large display (e.g., a display that substantially or completely surrounds the user). On the other hand, moving UI element 1210 from location 1220 to location 1221 may include teleporting UI element 1210 from location 1220 to location 1221. For example, UI element 1210 may be displayed at location 1221 while simultaneously ceasing to be displayed at location 1220. However, teleporting UI element 1210 from location 1220 to location 1221 may result in undesirable effects, for example, when UI element 1210 suddenly appears at new location 1221. In addition, the user may no longer be able to see UI element 1210 in location 1221, especially when new location 1221 may include content that is being displayed, which may obscure the location where the UI element is to be displayed.

[0231] In an embodiment, moving UI element 1210 to location 1221 may include a two-stage process. In an embodiment, the two-stage process of an embodiment may be configured to facilitate the user's perception of the movement of the UI element to a new location and / or zone without unexpected surprises, and may make it easier to follow and perceive the movement of UI element 1210 to location 1221. In an embodiment, the two-stage process may include teleporting UI element 1210 from location 1220 to an introduction location different from location 1221, and moving (e.g., dragging) the UI element from the introduction location to location 1221.

[0232] Fig. 12C UI element 1210 is shown teleporting from location 1220 to introduction location 1222. In an embodiment, teleporting UI element 1210 from location 1220 to introduction location 1222 may include displaying UI element 1210 at introduction location 1222 while simultaneously or nearly simultaneously ceasing to display UI element 1210 at location 1220. The result is that UI element 1210 appears to teleport from location 1220 to location 1222.

[0233] In an embodiment, introduction location 1222 may be determined based on the second location (e.g., location 1221). For example, the introduction location may be determined as a location at a predetermined distance from the second location in the direction of location 1220. In an embodiment, introduction location 1222 may also be a location determined based on the zone where the second location is located. For example, each zone of display 1200 may have a predetermined introduction location. For example, introduction location 1222 may be a predetermined introduction location for zone 1201.

[0234] Fig.12D UI element 1210 is shown initially moving from introduction location 1222 to location 1221 after teleporting from location 1220 to introduction location 1222 . Fig.12E UI element 1210 is shown moved to location 1221. In an embodiment, the movement of element 1210 from introduction location 1222 to location 1221 can be displayed in an animated manner, and the user can perceive UI element 1210 moving, sliding, or otherwise dragging from introduction location 1222 to location 1221. In some embodiments, audio can also be played as part of the animation display of the movement of UI element 1210 from introduction location 1222 to location 1221. In some embodiments, different audio can be played when UI element 1210 teleports from location 1220 to introduction location 1222.

[0235] Fig.13 1 is a flow chart illustrating a method 1300 for controlling the display of UI elements based on gaze within a CGR environment. In some embodiments, the method 1300 may be performed by the system 100 or a portion of the system 100. In some embodiments, the method 1300 may be performed by one or more external systems and / or devices. In some embodiments, the method 1300 may be performed by the system 100 (or a portion of the system 100) in conjunction with one or more external systems and / or devices.

[0236] At block 1302, the system displays a UI element at a first location within a CGR environment via a display of an electronic device (e.g., a wearable electronic device, an HMD device, etc.). For example, the UI element can be displayed at the first location within the CGR environment via a first display (e.g., a left-eye display panel) or a second display (e.g., a second-eye display panel) of the electronic device.

[0237] In an embodiment, the first location where the UI element is displayed may be a location within a virtual display. In an embodiment, the virtual display may represent a curved display having a size that allows the display to partially or completely surround the user, so that the user may perceive the display as a display that surrounds him or her.

[0238] In an embodiment, the UI element can be a UI element (e.g., a cursor) associated with an input mechanism (e.g., a mouse, a stylus, a pen, a touch-sensitive surface, an image sensor, etc.), and the input mechanism can be configured to control the UI element. For example, the UI element can move based on corresponding movements of the associated input mechanism. In an embodiment, the UI element can be an application window.

[0239] At block 1304, the system detects the current gaze of the user within the CGR environment. For example, an input sensor and / or an image sensor may be used to detect the gaze of the user. In an embodiment, the gaze of the user may be a head gaze, an eye gaze, or a combination thereof. In an embodiment, detecting the current gaze of the user may include detecting the direction in which the current gaze is directed. In an embodiment, it may be determined that the current gaze is directed to a specific location and / or area within the CGR environment.

[0240] At block 1306, the system determines a second location within the CGR environment based on the user's current gaze. In an embodiment, the second location is different from the first location. For example, the second location can be a location in a different zone and / or location within the CGR environment than the zone and / or location of the first location.

[0241] In an embodiment, for example, in the case where the second position is determined to be a zone, the location within the zone where the UI element is to be moved can be an alignment point. In these embodiments, the UI element can be moved to the second position (e.g., a zone) and can be aligned to the alignment location. In an embodiment, the second position can be based on the UI element that the user determines to be viewed. In an embodiment, the second position can be determined based on the dwell time of the user's gaze. For example, the second position can be determined to be a location where the user's gaze remains pointed to (e.g., dwells) for at least a predetermined period of time.

[0242] At block 1308, the system receives a request to move the UI element. In an embodiment, the request to move the UI element may include movement of an input mechanism associated with the UI element (e.g., a mouse, stylus, pen, touch-sensitive surface, image sensor, etc.). In an embodiment, the request to move the UI element may be a request to move the UI element a distance to move, or may be a request to move the UI element in a direction of movement. In an embodiment, the request to move the UI element may include clicking and holding an input mechanism (e.g., clicking and holding a mouse, stylus, etc.).

[0243] At block 1310, the system displays the UI element at a third location within the CGR environment in response to determining that the request meets the mobile criteria. In an embodiment, the third location may be different from the first location and the second location. In an embodiment, the third location may be an introduction location within the CGR environment as described above.

[0244] In an embodiment, displaying the UI element at a third location within the CGR environment may include teleporting the UI element from the first location to the third location. For example, displaying the UI element at the third location may include displaying the UI element at the third location while simultaneously or nearly simultaneously ceasing to display the UI element at the first location. The result is that the UI element appears to teleport from the first location to the third location.

[0245] In an embodiment, determining that the request satisfies the movement criteria may include determining that the user's gaze has been detected as being directed toward the second location, has been detected as remaining at the second location or on a UI element in the second location for at least a predetermined period of time, and / or has been detected as being directed toward a specific UI element in the second location.

[0246] In an embodiment, determining that the request satisfies the movement criteria may also include determining that the movement distance of the input mechanism exceeds a threshold movement amount. If the input mechanism movement distance exceeds the threshold movement amount, it may be determined that the request to move the UI element satisfies the movement criteria. Conversely, if the input mechanism movement distance does not exceed the threshold movement amount, it may be determined that the request to move the UI element does not satisfy the movement criteria.

[0247] In an embodiment, determining that the request satisfies the movement criteria may also include determining that the movement direction of the input mechanism is in a specific direction. If it is determined that the movement direction of the input mechanism is toward the second position, it may be determined that the request to move the UI element satisfies the movement criteria. Conversely, if it is determined that the movement direction of the input mechanism is not toward the second position, it may be determined that the request to move the UI element does not satisfy the movement criteria.

[0248] In an embodiment, determining that the request satisfies the movement criteria may further include determining that the user's current gaze is directed to an application of a particular type. If it is determined that the application directed to the user's gaze is of a particular type, then it may be determined that the request to move the UI element satisfies the movement criteria. Conversely, if it is not determined that the application directed to the user's gaze is of a particular type, then it may be determined that the request to move the UI element does not satisfy the movement criteria.

[0249] At block 1312, the system moves the UI element to the second position in response to displaying the UI element at the third position. In various aspects, moving the UI element from the third position to the second position may include sliding, dragging, and / or otherwise moving the UI element from the third position to the second position so that the UI element appears to be moved (rather than teleported) from the third position to the second position.

[0250] In an embodiment, moving the UI element from the third position to the second position may include displaying the movement in an animated manner. The animation display may allow the user to perceive that the UI element 1210 slides, drags, and / or otherwise moves (rather than teleports) to the second position from the third position.

[0251] As described above, one aspect of the present technology is to collect and use data that can be obtained from various sources to provide dedicated resource management for low-power devices (e.g., HMD devices), thereby saving battery life for users and providing dedicated content to users of low-power devices. The present disclosure contemplates that, in some instances, this collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0252] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to save the battery life of the user's low-power device. Thus, for example, using such personal information data, the system properly manages resources to save the battery life of the low-power device. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0253] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving the user's informed consent. In addition, such entities should consider taking any necessary steps to defend and safeguard access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0254] Regardless of the foregoing, the present disclosure also contemplates an example in which a user selectively blocks the use or access of personal information data. That is, the present disclosure contemplates providing hardware elements and / or software elements to prevent or block access to such personal information data. For example, with respect to managing resources for low-power devices, the present technology may be configured to allow a user to choose to "opt in" or "opt out" at any time during or after registration for a service to participate in the collection of personal information data. In another example, a user may choose not to provide eye tracking data, such as pupil location, pupil dilation, and / or blink rate, for specialized resource management. In yet another example, a user may choose to limit the length of time that eyeglass tracking data is maintained or to completely prohibit the development of a baseline eyeglass tracking file. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then the user may be reminded again just before the personal information data is accessed by the application.

[0255] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount and / or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how the data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0256] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed examples, the present disclosure also contemplates that various examples may also be implemented without access to such personal information data. That is, various examples of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, resources of a low-power device may be managed, and preferences may be inferred based on non-personal information data or an absolute minimum of personal information (such as content requested by a device associated with a user, other non-personal information available to a system controlling the low-power device, or publicly available information), thereby selecting content (e.g., status updates and / or objects) and delivering them to the user.

Claims

1. A method comprising: displaying a user interface (UI) element at a first location within a computer-generated reality (CGR) environment via a display of the wearable electronic device; detecting a current gaze of the user within the CGR environment; determining a second location within the CGR environment based on the current gaze of the user, wherein the second location is different from the first location; Receiving a request to move the UI element; In response to determining that the request satisfies the movement criteria, displaying the UI element at a third location within the CGR environment, wherein: the third position is different from the first position and the second position; and wherein displaying the UI element at the third position within the CGR environment comprises teleporting the UI element from the first position to the third position, wherein teleporting the UI element from the first position to the third position comprises: displaying the UI element at the third position; and At the same time or almost at the same time, stop displaying the UI element at the first position In response to displaying the UI element at the third position, the UI element is moved to the second position according to the request. 2 . The method of claim 1 , wherein moving the UI element to the second position comprises sliding the UI element from the third position to the second position such that the UI element does not teleport from the third position to the second position. 3 . The method of claim 1 , wherein the second location within the CGR is one of a place and a zone within the CGR environment. 4 . The method of claim 1 , wherein determining the second location within the CGR environment comprises determining the second location based on a location within the CGR environment at which the current gaze of the user is directed. 5 . The method of claim 4 , wherein the second position within the CGR is the location within the CGR environment to which the current gaze of the user is directed. The method of claim 1 , wherein the request to move the UI element comprises a request to move the UI element a movement distance. The method of claim 6 , wherein determining that the request satisfies the movement criteria comprises determining that the movement distance exceeds a movement threshold. The method of claim 1 , wherein the request to move the UI element comprises a request to move the UI element in a movement direction. 9 . The method of claim 8 , wherein determining that the request satisfies the movement criteria comprises determining that the movement direction is a direction toward the second location.

10. The method of claim 8, wherein determining that the request satisfies the movement criteria comprises determining that the movement direction is any direction within the display, and wherein the third position is in a direction away from the second position and opposite to the movement direction.

11. The method of claim 1 , wherein determining the second location within the CGR environment based on the current gaze of the user comprises: detecting that the current gaze of the user is directed toward the second position; as well as It is determined that the current gaze of the user remains directed toward the second location for at least a predetermined period of time.

12. The method of claim 1 , wherein determining the second location within the CGR environment based on the current gaze of the user comprises: Detecting that the current gaze of the user is directed to a type of UI object in the second position; as well as It is determined that the type of the UI object in the second position is a specific type.

13. The method of claim 1 , wherein determining the second location within the CGR environment based on the current gaze of the user comprises: Detecting that the current gaze of the user is directed to an application of a type in the second location; as well as It is determined that the type of the application in the second location is a specific type. The method of claim 1 , wherein the current gaze of the user is one of head gaze, eye gaze, and a combination thereof.

15. The method of claim 1, wherein the UI element is an application window, and further comprising: detecting user input including clicks of input mechanisms; as well as In response to determining that the user input is maintained, the application window is moved to the second position within the CGR environment determined based on the current gaze of the user.

16. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a wearable electronic device including a display, the one or more programs comprising instructions for executing the method according to any one of claims 1-14.

17. A wearable electronic device, comprising: monitor; one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1-14.

18. A wearable electronic device, comprising: monitor; Apparatus for carrying out the method according to any one of claims 1-14.

19. A computer program product, comprising one or more programs configured to be executed by one or more processors of a wearable electronic device, the one or more programs comprising instructions for performing the method according to any one of claims 1-14.

Images