Out-of-Process Effect of Electronic Device

Abstract

The remote user interface (UI) rendering effect provides an improvement in privacy and efficiency in a computer user input system. In one aspect, an application specifies a remote UI effect that is managed and rendered on UI elements separately from the application, such as by an effect component that runs outside of the application's operating system process. When user input indicates a preliminary interaction with a UI element, the remote UI effect can be rendered without the application's knowledge of the preliminary interaction, thus protecting the user's privacy from the application for the preliminary UI interaction.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 358,070, filed on July 1, 2022, entitled "OUT - OF - PROCESS EFFECTS FOR ELECTRONIC DEVICES"; U.S. Provisional Patent Application No. 63 / 402,435, filed on August 30, 2022, entitled "OUT - OF - PROCESS EFFECTS FOR ELECTRONIC DEVICES"; U.S. Provisional Patent Application No. 63 / 449,945, filed on March 3, 2023, entitled "OUT - OF - PROCESS AUDIO EFFECTS FOR ELECTRONIC DEVICES"; and U.S. Provisional Patent Application No. 63 / 470,952, filed on June 4, 2023, entitled "OUT - OF - PROCESS EFFECTS FOR ELECTRONIC DEVICES", the disclosures of each of which are hereby incorporated by reference in their entirety.

[0002] This disclosure relates to human - computer interface technology and computer privacy, including, for example, out - of - process effects for electronic devices.

Background Art

[0003] Operating system software generally provides an abstraction layer between user interface hardware and applications that run on the operating system. Multiple applications can be executed in separate operating system processes using corresponding user interface windows presented on a computer display managed by the operating system.

Brief Description of the Drawings

[0004] The specific features of the technology of this application are set forth in the appended claims. However, for purposes of illustration, some implementations of the technology of this application are shown in the following figures.

[0005]

Figure 1

[0006]

Figure 2

[0007]

Figure 3A

[0008]

Figure 3B

[0009]

Figure 4

[0010]

Figure 5

[0011]

Figure 6

[0012] [[ID=4,4]]

Figure 7

[0013]

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0014] The following detailed description is intended as an explanation of various configurations of the technology of the present application and is not intended to represent the only configuration in which the technology of the present application can be practiced. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details to provide a complete understanding of the technology of the subject matter. However, the technology of the present application is not limited to the specific details shown herein and can be implemented using one or more other implementation forms. In one or more implementation forms, the structures and components are shown in block diagram form to avoid obscuring the concepts of the technology of the present application.

[0015] Aspects of the present disclosure include technologies for providing improved privacy and / or efficiency in a computer user input system. Although any computer input system can benefit from these technologies, systems that can capture user data that can identify an individual, such as an augmented reality system that tracks a user's hand gestures and / or line of sight location, can particularly benefit from these technologies.

[0016] In one aspect of the present disclosure, a user input and rendering system can receive and render user input and can filter user input available to a computer software application (or an "app" herein). User input filtering enables a user's privacy from an application by at least restricting access of the application to the user's filtered input data. In some aspects, the computer user input and rendering system may be more trustworthy than applications running on the same computer because, for example, the user input and rendering system is provided by a trustworthy operating system vendor and the application is provided by a less trustworthy third-party application vendor.

[0017] In one aspect, a user can receive rendered feedback of the user's preliminary interaction with a user interface element, such as a button or a scroll bar, without making the preliminary interaction available to the application itself. The preliminary interaction with the UI element can include an intentional initial interaction with the user interface, such as exploring the interface of the application (e.g., to discover that a rectangle with a square inside is actually a scroll bar), and the preliminary interaction can also include an unintentional and even unconscious interaction with the application's UI elements (e.g., when the user's eyes move across the user interface without the user paying attention to the user interface).

[0018] Aspects of the present disclosure provide techniques for efficiently maintaining the privacy of a user's preliminary UI interactions from an application while still providing the user with rendered feedback of the preliminary interactions. Rendered feedback of a preliminary interaction with a UI element can include, for example, rendered visual cues (such as glows, lifts, shape changes, transparency changes, etc.) and / or audio cues when a hand gesture occurs near the UI element, when the user's hand hovers over or near the UI element, when the user's line-of-sight location is near or hovers near or over the UI element, or when a pointer controlled by a trackpad or pointer device (such as a mouse, joystick, etc.) hovers at or near the UI element. In one aspect, when it is determined that the user intends to interact with an application (or a UI element of the application), the user input can be considered no longer preliminary (e.g., can be considered confirmed), and some user input can be provided to the application. In one aspect, an application can declare or define a rendered feedback effect of a preliminary interaction with a UI element before the preliminary interaction occurs, giving the application control over the nature of the preliminary interaction feedback effect even when the application never knows about the preliminary interaction with that UI element. The application can provide the declaration or definition of the effect to an operating system or another software component for managing the rendering of the preliminary interaction. When the rendering of the effect is managed in an operating system process separate from the application that proves the declaration or definition of the effect, the effect is referred to herein as a remote effect or an "out-of-process effect".

[0019] As described above, aspects of the present disclosure may be applied to an extended reality system. The physical environment refers to the physical world that people can perceive and / or interact with without the aid of an electronic device. The physical environment may include physical features such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly perceive and / or interact with the physical environment through vision, touch, hearing, taste, smell, etc. In contrast, an extended reality (XR) environment refers to an environment that is wholly or partially simulated and with which people perceive and / or interact through an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. Using an XR system, user input may include tracking a person's physical movements, and in response, the XR system can render adjustments to one or more characteristics of one or more virtual objects simulated within the XR in a manner that conforms to at least one physical law. For example, if a user's hand gesture is rendered at a location near but not touching a rendered application UI element in the XR system, the UI element may be rendered in a glow or jiggle to indicate a preliminary interaction with that UI element (e.g., to indicate that the UI element is interactive and / or to guide the user towards targeting that UI element) without notifying the application of the preliminary user interaction. In an alternative example, if a user's hand gesture touches or grasps a UI element (e.g., and / or a user's hand gesture occurs near or away from the UI element while the user's line of sight intersects the UI element), it may be determined that the user intentionally intends to interact with the UI element, and then a user input indicating the touch or grasp of the UI element may be provided to the application.

[0020] Many different types of electronic user input and rendering systems can enable a human user to perceive and / or interact with various XR environments. Examples include head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display functionality, windows with integrated display functionality, displays formed as lenses designed to be placed over a person's eyes (similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mountable system may have one or more speakers (singular or plural) and an integrated opaque display. Alternatively, a head-mountable system may be configured to accept an external opaque display (e.g., a smartphone). A head-mountable system may incorporate one or more imaging sensors for capturing an image or video of the physical environment and / or one or more microphones for capturing the audio of the physical environment. A head-mountable system may have a transparent or translucent display instead of an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed towards a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scan light source, or any combination of these technologies. The medium may be an optical waveguide, hologram medium, optical coupler, optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface.

[0021] Aspects of the present disclosure provide a user with a rendered confirmation of user engagement with an application user interface before providing user input to the application. The user interface (UI) can be presented to appear at a location within a physical environment that is remote from the user input device and / or remote from the display device that presents the user interface (e.g., within a three-dimensional XR display environment). For example, physical movement of a mouse input device can be rendered as movement of a mouse cursor on a display at a physical location different from the physical mouse (perhaps just a few inches). As another example, a gaze cursor can indicate the location of the user's gaze, and the location of the user's gaze is remote from one or more cameras facing the eyes used to determine the gaze location. Similarly, hand gestures can be rendered as hand objects in a virtual space and presented to the user. Thus, the rendered effect of the user input can be rendered or presented to the user at a location physically separated from the physical location of the sensor that acquired the sensed user input. In a virtual space or other augmented reality environment, the user input can also occur at a location remote from the application UI and / or the display component that presents the representation of the user's hand, and may appear to the user to be at a location remote from the display component.

[0022] Further, a user can move a hand and / or direct a gaze towards or near a rendered application user interface and / or perform a hand gesture for interaction with one displayed application UI without intending to interact with the application UI (e.g., during a conversation with another person not associated with the electronic device, due to the user's normal movement around the physical environment), and without intending to provide input to an application associated with another displayed application UI.

[0023] Aspects of the present disclosure include receiving, in a system process of an electronic device, a definition of an effect on a first user interface (UI) element managed by an application from the application being executed on the electronic device. While the first user interface element is being displayed by the electronic device without the effect applied to the first user interface element, the system process may receive user input. In response to a determination that the user input corresponds to the first user interface element displayed without the effect, the system process renders the effect on the first UI element according to the definition without providing the user input to the application. In one or more implementations, the system process can also provide lighting, mirroring, and / or anchoring instead of the application, such as without providing information regarding lighting, mirroring, and / or anchoring to the application.

[0024] FIG. 1 shows an exemplary environment 100 for implementing aspects of the present disclosure. In the exemplary environment 100, a user 110 interacts with a computing device 120 having a display 115. As shown, the computing device 120 can include one or more speakers, such as speaker 117 (e.g., a plurality of speakers operable to generate spatialized audio perceived by the user 110 as being emitted from a location separate from the location of speaker 117 in the physical environment of the user 110). In one or more examples described herein, the user 110 may gaze at the computing device 120 and the device may sense the user's line-of-sight location 114. In one or more implementations, the user's line-of-sight location can constitute user input to the device.

[0025] The computing device 120 can be a smartphone, a tablet device, or a wearable device such as a smartwatch or a head-mounted portable system, including a display system capable of presenting a visualization of an augmented reality environment to the user 110. The computing device 120 can be powered using a battery and / or any other power source. In one example, the display system of the computing device 120 provides a stereoscopic presentation of the augmented reality environment, enabling the user to have a three-dimensional visual indication of the rendering of a particular scene. In one or more implementations, instead of or in addition to accessing the augmented reality environment using the computing device 120, the computing device 120 is utilized.

[0026] The computing device 120 can include one or more cameras (e.g., visible light camera, infrared camera, etc.). Further, the computing device 120 can include various sensors capable of detecting user input, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, LIDAR sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, short-range communication sensors, etc. Further, the computing device 120 can include hardware elements capable of receiving user input such as hardware buttons or switches. The user input detected by such sensors and / or hardware elements corresponds to various input modalities for interacting with virtual content displayed within a given augmented reality environment. For example, such input modalities can include, but are not limited to, face tracking, eye tracking (e.g., line of sight direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, respiration, temperature, brain waves, smell), recognizing speech or audio (e.g., specific hotwords), and activating buttons or switches. The computing device 120 may also detect and / or classify physical objects within the physical environment of the computing device 120.

[0027] In one or more implementations, computing device 120 may be communicatively coupled to a base device. Such a base device may generally include more computing resources and / or available power compared to computing device 120. In one example, computing device 120 may operate in various modes. For example, computing device 120 may operate in a stand-alone mode independent of any base device.

[0028] Computing device 120 may also operate in a wireless tethering mode (e.g., connected via a wireless connection to a base device) that operates in cooperation with a given base device. Computing device 120 may also operate in a connected mode in which computing device 120 is physically connected to a base device (e.g., via a cable or some other physical connector), and may utilize power resources provided by the base device (e.g., when the base device is charging computing device 120 while physically connected).

[0029] When computing device 120 operates in wireless tethering mode or connection mode, at least some of processing user input and / or rendering an augmented reality environment is offloaded to the base device, thereby reducing the processing burden on computing device 120. For example, in one implementation, computing device 120 operates in cooperation with the base device to generate an augmented reality environment that includes physical and / or virtual objects that enable different forms of interaction (e.g., visual, auditory, and / or physical or tactile interactions) between the user and the augmented reality environment in real time. In one example, computing device 120 provides rendering of a scene corresponding to an augmented reality environment that can be perceived and interacted with by the user in real time. Additionally, as part of presenting the rendered scene, computing device 120 may provide audio and / or tactile feedback to the user. The content of a given rendered scene may depend on the available processing power, network availability and capacity, available battery power, and current system workload.

[0030] Computing device 120 may also detect events that occur within a scene of the augmented reality environment. Examples of such events include detecting the presence of a living being such as a person or pet in the scene, a particular person, entity, or object.

[0031] Figure 2 shows an exemplary system 200 that provides an out-of-process effect. System 200 may be implemented, for example, on computing device 120. System 200 includes an application (app) process 202 corresponding to application 260, and a system process 204 that includes a rendering system 270 (e.g., a rendering process) and an effect component 280 (e.g., an effect process). In some aspects, processes 202 and 204 may be separate processes created and managed by an operating system and executed on a common device such as computing device 120. Aspects of system process 204 (e.g., including rendering system 270, effect component 280, and / or hit test 282) may be implemented in hardware, software, or a combination thereof.

[0032] As shown in FIG. 2, the system process 204 includes both the rendering system 270 and the effect component 280. However, the present disclosure is not so limited. For example, the rendering system 270 may be executed in a process separate from the effect component 280, the rendering system 270 and the effect component 280 may be executed in a common process, or the rendering system 270 and the effect component 280 may be further distributed to be executed in three or more processes, and all of these processes may be separate from the app process 202. Similarly, in other aspects, the app 260 may be distributed to be executed in two or more app processes. Operating system processes such as the app process 202 and the system process 204 can provide security and / or privacy between processes when executed on a common device such as the computing device 120 of FIG. 1 or the computing device of FIG. 8. For example, the app process 202 and the system process 204 can operate in different virtual memory spaces so as to be controlled by the operating system and executed by the memory controller hardware of the processor. Separate app and system processes can help prevent the app 260 from accessing data within the system process 204. In other aspects, the system process 204, the rendering system 270, and the effect component 280 may include one or more operating system drivers that are not executed within the virtual memory space.

[0033] The operation of system 200 may include receiving user input and outputting the rendered effect as feedback for the user input, which may be performed outside of app process 202 and without the knowledge of app 260. App 260 can provide a description of its UI element(s) 250 to rendering system 270, and app 260 can provide a definition 252 of the effect that will be rendered in response to future user input. When effect component 280 receives user input corresponding to the definition of the effect received from app 260, effect component 280 may cause rendering system 270 to render effect 256 as a user output.

[0034] In an optional aspect of system 200, the effect component 280 can know the location of the UI element (e.g., the location of a button or scroll bar), for example, via an optional UI element 258 message from the rendering system 270. In one or more implementations, the effect component 280 can then perform an optional hit test 282 between the current location of the UI element and the current location of the user input. In one or more other implementations, the hit test 282 is separate from the effect component 280 and can be performed by another system process that performs hit tests for multiple different purposes (e.g., for the effect component 280 as well as other components and / or processes). When a particular user input is identified as being associated with a particular UI element of app 260 by a hit test 282 between the user input and the particular UI element of the app 260 (e.g., by the effect component 280 or another system process of the computing device 120), the hit test 282 can identify a preliminary interaction with the particular UI element of app 260. In one or more implementations, the hit test 282 can include a "fuzzy" hit test operation that intentionally biases the hit test results towards the interactive UI element (e.g., away from the non-interactive UI element). As an example, when the user's line of sight is approximately midway between an interactive UI element (e.g., a button, slider, list, text entry field, etc.) and a non-interactive UI element (e.g., static text and / or an unlinked image), and / or when the user's line of sight intersects the interactive UI element and the non-interactive element at different depths, the hit test 282 can determine that the line of sight is associated with the interactive UI element. When a preliminary interaction is identified, the effect 256 can be rendered as a user output without notifying app 260 of the user input or the identified interaction.Alternatively, when an alternative user input is identified as a confirmed user interaction with a UI element, the effect component 280 or another system process may notify the app 260 of the confirmed interaction as a filtered user input message 254.

[0035] The effect definition 252 can describe various effects that instruct the app to render on one or more UI elements by the system process 204. Each effect defined in the effect definition 252 can correspond to a specific type of user input interaction with a specific UI element. In an aspect, the effect definition 252 can describe multiple effects that are rendered in response to a single user input interaction with a single UI element. For example, the effect definition 252 can indicate that when the user's line of sight is within a specific range of a specific button UI element, the button starts emitting light up to a specific luminance level and / or emitting sound, and when the line of sight is within a closer distance range to the button, the UI element can both emit light at a brighter level and generate a staccato sound and / or a buzzing sound, and when the line of sight is within a third closest range (presumably, directly gazing at the button and / or directly gazing at the button for at least a predefined dwell time such as 1 second, 1 / 2 second, 1 / 10 second, or a fraction of 1 second that is smaller), the user interaction can be identified as a confirmed interaction (e.g., the interaction intended by the user as the activation of the button).

[0036] In an alternative optional aspect, app 260 can directly provide the effect component 280 with information that describes the UI elements of app 260 (e.g., instead of the effect component 280 receiving UI information from the rendering system 270). Additionally, instead of location-based hit testing, hit test 282 may more generally determine that a particular user input corresponds to an interaction with a particular user interface element. For example, a verbal audio input from a user saying "red button" can cause hit test 282 to associate that audio input with a red button UI element.

[0037] In one aspect, the effect definition 252 can be a declarative definition. In this aspect, app 260 can provide all the information necessary to a software component external to the app process 202, such as the effect component 280, to render the desired effect of the app without the knowledge or further participation of the app. The declarative definition of an effect can include, for example, the identification of UI elements provided by a user interface framework or operating system, the identification of user input that triggers it, and the identification of the effect that is rendered when the triggering user input corresponds to a first UI element. In one or more embodiments, the identified effect may include a selected one of a plurality of default effects available from a system process (e.g., a highlight effect, a glow effect, or a lift effect), may include an indication for the system process to automatically select a default effect, or may include instructions to render a custom effect provided by app 260.

[0038] In one or more implementations, the identified effects can be applied to the entire UI element or to subviews or sub-elements of the UI element (e.g., the internal background subview of a text field, sub-elements of a slider such as the thumb of the slider, sub-elements of a segmented control element, cells of a collection view, cells of a table view, or sub-elements of a date picker or other picker). In one or more implementations, the same effect can be applied to multiple sub-elements of a UI element. In one or more implementations, different default or custom effects can be applied to different sub-elements of a UI element. In an example of a set of cells or a table, the effects can be defined based on heuristics such as whether a given cell is highlightable or selectable. For example, these heuristics can model the intention to show a hovering effect when the cell is interactive (e.g., zero effect can be used when the cell is non-interactive). In one or more implementations, when the app explicitly sets the effect style of a cell, the heuristic may be disabled and the explicit style may be used. For example, the hovering effect can be disabled for the list rows, view hierarchies, etc. of the application. In one or more implementations, the definition of an effect can include multiple definitions of multiple effects (e.g., the same or different effects) for multiple respective sub-elements of a UI element (e.g., the same or different effects can be applied when a hovering interaction is detected at or near a selectable date or time within a picker).

[0039] In one or more implementations, providing the definition 252 of the effect may include providing a definition of the shape of the effect and / or the shape of the underlying UI element. For example, the shape of the effect may include a capsule shape, a linear shape, a circular shape, or other predefined or custom shape. In one or more implementations, the definition of the shape may include one or more definitions of a corner shape, a corner radius, a corner curve, a corner masking, and / or other corner and / or edge definitions for the shape. In one or more implementations, the shape may not have a unique frame or size and instead may be broken down into a specific shape within a given frame (e.g., this may provide more expressive shapes that can automatically adapt to the view to which the shape is associated). For example, it is possible to provide a predefined capsule shape that can include corners that are automatically resizable based on the frame provided for the shape.

[0040] In one or more implementations, the definition of a shape can include the definition of an inset shape (e.g., for an effect that is inset within a UI element or occurs within the bounds of a UI element) and / or an outset shape (e.g., that is outset from a UI element, such as by padding via a negative inset, or a shape that occurs outside the bounds of a UI element). In one or more implementations, the definition of a shape can automatically insert a rounded linear shape that preserves concentric corner radii proportional to its size. In one or more implementations, the definition of a shape can include the definition of a dynamic shape (e.g., that can be resolved within a given context). For example, a dynamic shape definition can be used to create a shape that uses a fractional corner radius. A dynamic shape can have dynamic edges and / or corners and / or can include dynamic transparency, luminance, or other dynamic characteristics of the overall shape. In one or more implementations, a shape for an effect can be expressed as a modification of the content shape of a view (e.g., a UI element) or the shape of its container. For example, app 260 can modify the effect shape of a button UI element to provide padding while maintaining its original shape heuristics. In one or more implementations, a shape can be defined to correspond to the container in which the shape is rendered. In one or more implementations, a shape can be defined as a zero shape that causes the use of an implicit default shape.

[0041] In one or more implementations, the effect definition 282 may be provided without a shape definition. In one or more implementations, when no shape definition is provided, a default shape may be determined by the system process. In one or more implementations, when no shape definition is provided for pointer-style input, a default preview shape for the pointer may be used for the shape of the effect. In one or more implementations, the hovering effect may be deactivated during a drag touch input by the user. In one or more implementations, when the hovering effect is applied (e.g., to other UI elements), a drag touch input (e.g., for some UI elements) may be disabled. In one or more implementations, the UI elements may be sold from one or more files such as a Universal Scene Description (USDZ) file. In another implementation, the UI elements referred to in the declarative definition of the app may be provided by the operating system.

[0042] Figure 3A shows an exemplary method of rendering a UI effect. As shown, the method of Figure 3A may include receiving, in system process 300, a definition of an effect for a UI element of an app (box 302). The UI elements of the app may include elements of a UI window such as a virtual button, virtual switch, virtual slider, virtual list (e.g., dropdown list), text entry field, collection view (or its cells), or table view (or its cells), other interactive virtual content with which a user can interact to provide user input to the app, or any or all of non-interactive virtual content. The UI elements of the app may be two-dimensional elements or may be a rendering of a three-dimensional (3D) model. The UI elements of the app can include static virtual content (e.g., images, text, static shapes, borders, colors, etc.) and / or can include dynamic virtual content such as video and / or 2D or 3D animations. The UI elements may include virtual content that is displayed within or as part of the surrounding UI window or may be stand-alone virtual content that is displayed separately from other virtual content of the application and / or other UI elements. The user input may be received by system process 300 (box 304), and if it is determined (e.g., by the system process) that the received user input corresponds to the UI element of the received effect definition (box 306), the effect may be rendered (e.g., by system process 300) on the UI element according to the effect definition (box 308). In one aspect, the operations of Figure 3A may be performed in a system process 300 separate from app process 350 in which the app is executed.

[0043] The method of FIG. 3A may include optional additional operations, such as receiving additional user input that is different from the user input received at box 304 by system process 300 (box 310). Based on the original user input and / or the additional user input, when the system process 300 identifies the intention of the user to interact with the app or a UI element of the app (box 312), some or all of the user input for which the user's intention has been identified may optionally be provided from the system process 300 to the app process 350 (or otherwise made available) (box 314).

[0044] The UI elements referred to throughout FIGS. 3A and 3B may be part of the user interface of the app. For example, the UI elements may be defined by the app, and the purpose of the UI elements may be to enable the user to interact with the app (after the intention of the user interacting with the app has been identified) in one or more implementations forms. In one aspect, the UI elements may include one or more properties, such as (by way of example) the location, size, orientation, color, transparency, or luminance of the UI element, and the effect definition may include a modification of one or more of those properties. For example, the effect definition for a button UI element may include a lift effect to raise the location property of the button and a brightening effect to increase the luminance property of the button. Further, the app UI elements can include a default "remote state" that corresponds to the default value(s) of the properties of the UI element. The remote state of the app's UI elements can be considered remote in that while the app can define the UI elements, their properties, and their remote state, only system components (such as the rendering system 270 and the effect component 280) within a separate system process can manage or have knowledge of the remote state of the UI elements. This can prevent the app from discovering what remote state the UI elements are in at any given time, allowing the preliminary interaction of the user with the UI elements to be rendered while remaining private from the app.

[0045] In one aspect, the UI elements referred to throughout FIGS. 3A and 3B can also be obtained directly from the app, or from alternative sources such as the UI framework and file format described above, or from the operating system. These UI elements provided by alternative sources can function as part of the user interface of the app (such as the user interface of app 260 in FIG. 2). UI elements from alternative sources can have associated properties and a "remote state" defined by the alternative source and / or by the app using it. For example, even if a UI element can originate from an alternative source, such a UI element can still be managed by the app using it and can cause remote effects on system processes while maintaining privacy from the managing app.

[0046] In one aspect, the UI elements can be specified as a layer tree or as a rendering tree. For example, application 260 can provide a layer tree or a rendering tree of UI elements to rendering system 270, and application 260 can provide a definition 252 of an effect that includes the layer tree or the rendering tree of UI elements to an operating system or other component that manages out-of-process UI effects.

[0047] The effect definition (box 302) may be received from a specific app executed in the app process 350 and may describe an effect to be rendered on a UI element belonging to that specific app. However, implementations of the present disclosure are not so limited. For example, the effect definition received at box 302 may be received indirectly from other sources, the effect definition may be a predetermined effect, or the UI element may be a predetermined UI element. In one example, the button UI element may be default, and the "lift" effect may be default to occur when user input of a line of sight, a hand, or a pointer is determined to correspond to the default button UI element (e.g., hovering over or at the default button UI element). In the case of a default effect and / or a default UI element, the received effect definition may include a reference to the default element without fully defining such element.

[0048] In one aspect, the effect definition may identify an effect to be rendered for a UI element by identifying a remote state (with default rendering property values) that is used when user input is determined to correspond to the UI element. In a further aspect, the effect definition may identify an animation used when transitioning between default remote states.

[0049] User input (such as that received in box 304 or 310 of FIG. 3) can include many types of input from a user. For example, in addition to user input received from a user input device such as a mouse, keyboard, and / or audio input device, the received user input can also include the location and / or orientation tracking of the user's body or body part (such as a fingertip or hand), gesture recognition (such as hand shape, hand movement, and / or facial expression), or the location of the user's optical line of sight (e.g., line of sight location 114). The user input can include time or speed measurements such as a dwell time of the user's line of sight remaining at a particular location or within a particular distance or range of a UI element, or a hovering dwell that measures the duration that the user's hand or other body part remains within a particular two - or three - dimensional distance of a particular location or UI element.

[0050] In one aspect, a UI effect (such as that defined in box 302 of FIG. 3 and rendered in box 308) can include any rendered output that a user can perceive, including audio, images, and / or haptic effects. Examples of audio effects on a UI element can include an audible sound designed to be perceived as emanating from the location of the UI element, or a distortion of audio already emanating from the UI element prior to the rendering of the audio effect. As an example of an image effect, a UI element may be displayed to the user or otherwise visually presented prior to the rendering of the effect, and the rendered effect may change the appearance or location of the UI element. For example, a glow effect can cause an existing UI element to glow. A highlight effect can add a new visual cue, such as a circle or check mark, near the location of the UI element. In one aspect, the effect definition can specify a "reverse" rendering effect in which the specified effect is generally rendered except when the user input corresponds to the UI element.

[0051] In one aspect, the correspondence between user input and a first UI element (box 306) may include identifying a plurality of user interface elements with which the UI element can interact, and then determining that the plurality of identified UI elements includes the first UI element. For example, the user input may be a line-of-sight location. The system process may identify a list of potential UI elements that the user may be attempting to interact with, such as by identifying all UI elements from all applications located within a threshold distance of the line-of-sight location and / or all UI elements that the line-of-sight direction intersects, in a hit test process. If the effect definition includes the app UI element within that list, the corresponding effect may be rendered on that app UI element. In a use case where the line-of-sight direction intersects a plurality of UI elements, each having an effect definition, the system process may render the effect on the UI element that appears closest to the user among the plurality of UI elements to be displayed and / or on the UI element among the plurality of UI elements where the line-of-sight location intersects most centrally.

[0052] In one aspect, a single effect may be defined for a group of UI elements. For example, when a user input correspondence is determined for any of the UI elements within the group (306), the defined effect may be rendered on all of the UI elements within the group. In another example, the user input correspondence within box 306 may be determined by only a subset of the UI elements within the group. The definition of an effect for a group of UI elements may include an indication of which subset of the UI elements within the group to perform a hit test on. If the user input location (such as the user's line of sight or the location of a hand gesture) is within a particular proximity (e.g., a default distance defined by the system process or in the effect definition) of any element within the indicated subset of UI elements, the effect may be rendered on all of the UI elements within the group.

[0053] In one aspect, a single effect definition can define a group of effects to be rendered. Once the mapping to the UI element is determined (box 306), the UI element can be rendered with multiple different effects (box 308). For example, the effect definition for a button may include a lift effect (e.g., the button moving in the opposite direction of being pressed), a glow effect (e.g., brightening), and an audio cue (e.g., an audio tone emitted from the button).

[0054] In one aspect, an effect definition can combine multiple aspects described above. For example, a single effect definition can include multiple UI elements, multiple effects to be rendered, and / or multiple types of user input that can trigger the rendering of one or more effects.

[0055] In one aspect, even after identifying the intention of the user interacting with the application (box 312), only a summary or subset of the user input may be provided to the application (box 314). For example, when the dwell time of the user's gaze on an app button exceeds a threshold, the intention to interact with the application can be identified. However, instead of providing the user's gaze location or dwell time to the application, the application can be provided with an indication that the user intended to press the app button without providing any knowledge about the user's gaze to the application.

[0056] Using the list selection UI, more complex examples of out-of-process effects that combine the various aspects described above can be shown. A common list selection UI task can include presenting the user with a list of multiple options that includes an indication of the current selection. Examples of list selection UIs include the "combobox" in MacOS and the "dropdown list box" in Windows. Each of the options in the list may be rendered, for example, as a visual box that includes text describing the option, and the list may be presented as a row of text boxes without highlighting, except for the current selection highlighted in a first color.

[0057] In an exemplary out-of-process implementation of the list selection UI, the application 260 can provide the rendering system 270 with a definition of the UI element 250 that includes a row of text boxes. The definition of each text box can include three default remote states: 1) a non-selected (idle) remote state with a highlight property (or no highlight) set to a first color, 2) a current selection remote state with a highlight property set to a second color, and 3) a pre-selection remote state with a highlight property set to a third color. The application 260 can further provide an indication that the first text box in the list is the current selection, and the application 260 can provide the effect component 280 with an effect definition 252 indicating how a preliminary user input should affect the remote state of the text box. The rendering system 270 can render a list of text boxes, each having its corresponding remote state, and the rendering system 270 can provide the effect component 280 with the definition of the text box, along with the current location of each text box, via the UI element 258 message. The effect component 280 can use these definitions to manage the remote state of the text box. When the effect component 280 receives a user input that includes a location (e.g., a line-of-sight location or a hand gesture location), the effect component 280 can perform a hit test of the user input location against the location of the text box to determine a correspondence between the user input and one of the rendered text boxes.If the user input location corresponds to a second text box in a list that is different from the currently selected first text box, the effect component 280 can cause the rendering system 270 to render an effect 256 including a change in the highlight property from a non - selected remote state to a pre - selected remote state without notifying the application of the change in the remote state and highlight property of the user input location or the second text box.

[0058] Instead of or in addition to the above pre - selection, when the user input indicates the user's completed selection, the application can be notified of the user's intention to change the current selection in the list. For example, after rendering the above pre - selection, in response to receiving additional user input (box 310) that identifies the user's intention to change the current selection to the second box (for example, when the gaze dwell duration threshold on the second box is exceeded, or when a button press gesture occurs on the second box), the application can be notified of the user's intention to change the current selection in the list.

[0059] In one aspect, the remote effect definition can specify an animation for rendering upon transition between remote states. For example, in the above example of the list selection UI, the remote state transition animation can be defined to include a fade transition from the non - selected remote state to the pre - selected remote state, from the first color of the highlight (no highlight for the idle state) to the third color of the highlight (for the pre - selected state).

[0060] Figure 3B shows an exemplary method of rendering a UI effect according to an aspect of the present disclosure. As shown, the method of Figure 3B may include identifying (box 552), by an app process 350, app management rendering effect(s) and remote rendering effect(s) for corresponding UI elements. The definition of the identified remote rendering effect(s) may be provided, by the app process 350, to a remote effect rendering service provider (box 554 such as the rendering system 270 and / or the effect component 280 of Figure 2). User input may be received, by the app process 350, for UI elements having corresponding app management effects (box 556). The app management UI effect may be rendered, by the app process 350, on its corresponding UI element (box 558).

[0061] In one aspect, the operations of the method of Figure 3B may be performed in an app process 350 separate from a system process 300 in which remote / out-of-process UI effects are managed. In another aspect, the method of Figure 3A may be an exemplary method of providing a remote effect rendering service, and the effect definition provided in box 554 may be received in box 302. In yet another aspect, additional user input provided in box 314 may be received in box 556.

[0062] In other aspects, effects for preliminary user interactions may be identified as remote UI effects, and effects for confirmed user input may be identified as app management UI effects (box 552). In the example of the list selection UI described above, preliminary selection rendering effects may be identified as remote UI effects, while completed selection rendering effects may be identified as app management UI effects.

[0063] In one or more implementations, a delegate-based method may be provided to dynamically update a hovering effect based on a change in the view state of a UI element. In one or more implementations, a delegate-based method may be provided to dynamically customize the hovering effect behavior. In one or more implementations, an additional customized definition of a remote effect can be provided (e.g., to provide the ability to match effects across multiple views). In one or more implementations, multiple hovering effects may be composed of, or added to, a single view (e.g., a single UI element). In one or more implementations, an automatic default shape may be provided for some or all effects, such as for hovering effects, focus effects, drag-and-drop effects, context menu previews, etc. In one or more implementations, the shape definition may define the shape as a clip shape (e.g., to automatically set appropriate properties on a given layer by automatically setting the corner radius for a rounded rectangle or the mask for a custom shape). In one or more implementations, the shape of an effect may be resolved based on one or more device properties (e.g., based on the corner radius of the device display) or based on a container shape (e.g., the shape of the superview in which the shape is framed). In one or more implementations, the hovering effect may be defined based on the type of content associated with the UI element (e.g., an image or video) and / or based on the input type of user input (e.g., eye gaze, hand, or pointer style). In one or more implementations, the effect may be defined to reveal additional UI information and / or content (e.g., in response to eye gaze at a particular location), to expand the UI element to display more content, and / or to scale up or transform the UI element.In various implementations, out-of-process (e.g., remote) effects and / or application management effects may be defined and / or realized using application programming interfaces (APIs) and / or block-based programming.

[0064] In one or more implementations, an application process (e.g., application process 202) can identify application management user interface (UI) effects and remote UI effects for a first UI element (e.g., in box 552) and provide a definition of the remote UI effect and an identification of the first UI element to a system process (e.g., system process 204). The application process can receive user input corresponding to the first UI element and render an application management UI effect on the first UI element. In one or more embodiments, the user input can include a confirmed user input (e.g., a confirmed user input to the first UI element as determined by a hit test) that follows a preliminary user input corresponding to the first UI element (e.g., a preliminary user input to the first UI element as determined by a hit test) and is confirmed by the system process, and the remote UI effect is rendered by the system process. For example, the confirmed user input can include a confirmed user input to a second UI element generated by the system process in response to the preliminary user input to the first UI element. In one or more use cases, the first UI element can include a list selection UI that is extended to include selectable options in response to the preliminary user input, and at least one of the selectable options includes a second UI element. In one or more other use cases, the first UI element can represent an option within a list selection UI, the remote UI effect can correspond to a preliminary selection of an option within the list selection UI (e.g., be rendered and / or indicated in response to its detection), and the application management UI effect can correspond to a completed selection of an option within the list selection UI (e.g., be rendered and / or indicated in response to its detection).

[0065] In various examples described herein, examples of remote UI effects include remote effects (e.g., glow, highlight, lift, flicker, buzz, etc.) that are generated in response to identifying preliminary user input that matches a user interface element. In another exemplary use case, one or more remote effects may be generated by a system process (e.g., system process 300 described herein) in response to environmental conditions and / or characteristics, without providing an indication to the application that the remote effect is being generated.

[0066] For example, system process 300 may generate lighting effects on virtual content such as a rendered three-dimensional (3D) model and / or UI elements in response to identifying lighting conditions in the physical environment of computing device 120. For example, system process 300 may receive a request for lighting effects on virtual content (e.g., a rendered 3D model and / or a user interface (UI) element) managed by application 260 from application 260 (e.g., in relation to box 554). The request for lighting effects may be a system-defined request for lighting effects or may include a definition of lighting effects (e.g., a declarative definition) defined by application 260. For example, lighting effects may include brightening, darkening, or otherwise modifying the appearance of various portions of a UI element based on lighting conditions within the physical environment (e.g., the direction and / or luminance of one or more light sources). For example, system process 300 may identify lighting conditions in the physical environment while the virtual content is being displayed by computing device 120. Identifying lighting conditions may include receiving sensor data and / or images from one or more cameras and / or sensors (e.g., in box 304, with or instead of user input). System process 300 may then render lighting effects on the virtual content based on the identified lighting conditions without providing the lighting conditions to application 260.

[0067] As another example, system process 300 may generate a mirroring effect for virtual content (e.g., a rendered 3D model and / or user interface (UI) elements), where the mirroring effect mirrors one or more portions of the physical environment of computing device 120. For example, system process 300 may receive, from application 260 (e.g., in relation to box 554), a request for a mirroring effect for virtual content managed by application 260. The request for the mirroring effect may be a system-defined mirroring effect request or may include a definition of the mirroring effect (e.g., a declarative definition) as defined by application 260. For example, the mirroring effect may include displaying an image of a portion of the physical environment such that it appears to be reflected from a portion (e.g., a reflective surface or texture on the rendered 3D model and / or UI elements) of the virtual content (e.g., the rendered 3D model and / or UI elements). For example, system process 300 may identify one or more features of the physical environment of computing device 120 (e.g., by capturing one or more images of one or more portions of the physical environment) while the UI element is being displayed by computing device 120. Identifying the one or more features may include receiving sensor data and / or images from one or more cameras and / or sensors (e.g., in box 304, with or instead of user input) and using the sensor data and / or images to identify the one or more features (e.g., in box 306). System process 300 may then render a reflection of one or more features of the physical environment on the UI element without providing the one or more features of the physical environment to the application, based on the identified one or more features.Rendering the reflection of one or more features on virtual content can include, for example, obtaining an image of a portion of the physical environment and modifying the image based on the position and / or shape of the virtual content and / or the reflectivity of the surface of the virtual content (e.g., rotating, scaling, warping, distorting, etc.), and rendering the modified image over a portion or all of the virtual content. For example, virtual content in the form of a reflective sphere can be provided by application 260, and system process 300 can display a distorted mirror image of the physical environment on the surface of the reflective sphere without providing information about the physical environment to application 260.

[0068] In one or more implementations, a system process of an electronic device, such as computing device 120, can provide one or more other out-of-process services to an application, such as application 260 (e.g., without providing information about the service to the application, which can help protect the privacy of the user of the electronic device with respect to the application). For example, in one or more implementations, system process 300 can provide out-of-process anchoring for one or more applications in computing device 120.

[0069] For example, in one or more embodiments, system process 300 can receive (e.g., in connection with box 554) a request from an application such as application 260 to anchor virtual content of the application (e.g., a rendered 3D model and / or UI elements such as UI elements that may also provide one or more out-of-process effects) to a physical object within the physical environment of computing device 120. The system process 300 can then anchor the virtual content to the physical object (e.g., as part of the rendering at box 308) without providing information associated with the physical object or the physical environment to application 260 in response to the request. By way of example, the physical objects that may be requested for anchoring virtual content of an application can include tables, walls, chairs, floors, horizontal planes, vertical planes, hands, fingers, or any other physical object that can be identified by computing device 120. For example, the anchoring can be provided by the system process without providing to the application conversion, images, sensor data, or any other information that describes the location of the physical object or the physical environment or the physical objects within the physical environment. For example, the system process can provide to the requesting application only an indication of the success or failure of the anchoring.

[0070] In one or more implementations, anchoring virtual content to a physical object may include identifying a physical object in a physical environment by the system process 300 (e.g., using sensor data, one or more cameras, and / or computer vision operations). Receiving, from the application 260 in the system process 300, rendering information for the virtual content, and using the rendering information to render the virtual content by the system process at a location on a display of a device corresponding to the location of the physical object in the physical environment (e.g., when the display 115 is viewed by the user, a location on the display 115 that makes the virtual content appear to be at the location of the physical object in the physical environment). In various implementations, the rendering information received from the application 260 may include a rendered 3D model, a rendered UI element, or instructions (e.g., a layer tree) for rendering the 3D model or UI element. Anchoring virtual content to a physical object may also include detecting, by the system process 300 (e.g., using sensors and / or cameras of the computing device 120), the movement of the physical object with respect to the computing device 120, and modifying the rendering of the virtual content by the system process 300 to track the movement of the physical object with respect to the computing device 120 without providing information about the movement of the physical object to the application.

[0071] In one or more implementations, the system process 300 can provide the application 260 with confirmation that the virtual content is anchored to a physical object without providing the application with information related to the physical object or physical environment.

[0072] In one or more implementations, the virtual content anchored to a physical object by the system process 300 in an out-of-process anchoring operation can be non-interactive virtual content that includes context information about the physical object. In one or more implementations, the physical object can include an image within a physical environment. For example, the image can include a logo (e.g., a company, team, or product logo). In one or more implementations, the virtual content anchored to a physical object by the system process 300 in an out-of-process anchoring operation can include context information about the physical object, such as context information about the logo (e.g., company information or product information).

[0073] For example, in one or more use cases, a request from the application 260 to the system process 300 to anchor virtual content to a physical object within the physical environment of the computing device 120 may include an electronic version of the image. In response to a request that includes an electronic version of the image, the system process 300 can use the electronic version of the image to monitor the physical environment of the image during the operation of the device during the period before the virtual content is displayed. For example, in a use case where the image includes a logo, the electronic version of the image can include an electronic version of the logo, and the system process 300 can monitor the physical environment for an image that (e.g., using computer vision operations on camera frames captured by one or more cameras 117) matches the electronic version of the image (e.g., within a threshold).

[0074] System process 300 can also display virtual content in response to detecting an image in the physical environment during monitoring. For example, in a use case where the image includes a logo, system process 300 can detect a product (or advertisement or other physical object) in the physical environment where the logo is printed or otherwise displayed (e.g., using computer vision operations), and the virtual content to be displayed can include information about the company or other product corresponding to the logo. In various implementations, information about the company or other product may be stored locally on computing device 120 before detecting the logo in the physical environment (e.g., to prevent information about detecting the logo in the user's environment from being sent from the device to protect the user's privacy), or may be obtained from a server or other remote device before display in response to detecting the logo in the physical environment (e.g., after obtaining permission from the user). Thereafter, system process 300 can continuously update the location of the displayed virtual content to track the location of the physical object (e.g., by anchoring the virtual content to the physical object outside of the application process) while the physical object remains within the field of view of computing device 120 (e.g., of one or more cameras of computing device 120).

[0075] In one or more implementations, multiple applications on computing device 120 can provide anchoring requests to system process 300. System process 300 holds multiple anchoring requests while monitoring the physical environment of the physical objects requested during the operation of computing device 120, and can render virtual content for multiple applications (e.g., one or more anchored and rendered 3D models or anchored UI elements) for display when an individual requested physical object for an individual application is detected.

[0076] Figure 4 shows an exemplary use case where one or more UI elements 406 are displayed in a visible display area 400 of computing device 120. For example, a display 115 of computing device 120 (see, e.g., FIG. 1) may be operable to display one or more user interface windows 402 corresponding to underlying applications (e.g., app 260) running on computing device 120 in a visible display area 400 of the display 115. As shown, each of the user interface windows 402 may include one or more user interface elements 406. By way of example, user interface elements 406 may include virtual buttons, virtual switches, virtual lists (e.g., drop-down lists), tabs, scroll bars, application icons, and / or other interactive elements. As shown in FIG. 4, in one or more implementations, user interface elements 406 may be displayed separately from user interface windows 402. As described herein, system process 300 may render one or more effects on user interface element 406 when user input to computing device 120 corresponds to UI element 406 (e.g., when the user's line of sight location 114 is within or within the bounds of UI element 406, and / or when the user's hand or finger (or other pointer) hovers over a location within or within the bounds of UI element 406 for a predetermined time, etc.). These effects may be out-of-process effects defined by the application underlying UI element 406 and rendered without providing user information (e.g., location of user input) to the application.

[0077] As described herein, an out-of-process effect associated with a UI element, such as UI element 406, may include an audio effect. For example, when UI element 406 is implemented as a virtual button, the user's hand and / or line of sight near the button may cause a declaratively defined sound, such as a click sound, a lifting sound, a buzz sound, a bouncing sound, or other sound, to be output by one or more speakers of the electronic device (e.g., a speaker array including speaker 117), such that the sound is perceived by the user of the electronic device as originating from the location of UI element 406. These audio effects can be provided by system process 300 using a visual effect, such as a highlight or glow effect, or in combination with a visual effect.

[0078] In addition to protecting user privacy with respect to the application that is the basis of UI element 406, providing these audio effects as out-of-process effects, as described herein, can be beneficial, for example, in reducing latency between a user action associated with UI element 406 and the output of an audio effect corresponding thereto. For example, instead, if an audio effect is generated by the application in which UI element 406 is displayed, latency can occur due to the time used to transmit user input information to the application, the time the application uses to generate sound, and the time used for the sound to be output from the speaker(s). Further, in the examples described herein where the audio output is generated to be perceived as originating from the location of UI element 406, the location of UI element 406 where the user input occurs is used for the generation of the audio effect. Preparing the audio output by the application can reduce some of this latency, but the application running on the device may not have access to location information for user input, and thus, in some implementations, may not be able to prepare the sound for its own UI element(s).

[0079] Even when providing an out-of-process audio effect generated by system process 300, since the location of the UI element for which the audio effect is generated is not known until a user action occurs at or near that location, the dependency of the audio output on the location of the UI element can result in latency between the user action and the output resulting from the audio effect.

[0080] One option for reducing such latency is for the system process to pre-prepare the respective audio stream for each of all the UI elements 406 that are displayed. However, this can be undesirably costly, especially in terms of processing power and / or memory usage (in some cases), when there are multiple (e.g., many) UI elements 406 being displayed simultaneously. Another option is to start the audio stream each time the user's line of sight, hand, or finger crosses the location of a UI element. However, this option still includes latency and, in particular, in use cases where multiple (e.g., many) UI elements are displayed and the user is looking around the display area or making gestures with their hand without intending to interact with a UI element, it can result in a large number of unused audio streams being generated, which can also be undesirably costly, especially in terms of processing power and / or memory usage (in some cases).

[0081] According to an aspect of the present technology, an ongoing audio stream can be continuously generated such that any non-zero audio content within the audio stream is perceived to be at (e.g., emitted from) a location 408 corresponding to a user action (e.g., the location of the user's hand or finger and / or the line-of-sight location 114 at which the user's eyes are focused at any given time). The ongoing audio stream can follow the location 408 as the location 408 of the user action moves around. As shown in FIG. 4, when there is no UI element at the location 408 of the user action, the audio content of the continuously running audio stream can be null audio content 410 that contains only zero (e.g., silence can be correspondingly output by the speaker 117 as perceived at the user action location 408). As shown in FIG. 5, when the user action location 408 corresponds to the location of a UI element 406 (e.g., according to a hit test as described herein, such as in relation to the box 306 of FIG. 3A), the specific audio content 500 of that UI element (e.g., non-zero audio content) can be inserted into the ongoing audio stream that is being generated to be perceived at that location for output (e.g., replacing the null audio content 410 within the audio stream). For example, the audio content 500 can be declaratively defined by an application process 350, and the declarative definition of the audio content 500 can be provided to a system process 300 as described herein before the output of the audio content 500. In this way, the audio effect for the UI element 406 (defined in the audio content 500) is immediately rendered (e.g., box 308 of FIG. 3A) for perception by the user at the location of the user interface element 406 when the user action location 408 coincides with the location of the user interface element 406 and can be output by the speaker 117.

[0082] In one or more implementations, computing device 120 may include a buffer corresponding to an audio stream that operates continuously, and based on the location 408 of a user action (e.g., the line-of-sight location 114 of the user's line of sight and / or the location of the user's hand or finger), to generate an audio output that includes the content of that buffer perceived at the location 408 of the user action, spatial processing may be applied to the audio content (e.g., null audio content 410 or audio content 500) in that buffer. For example, the spatial processing may cause the audio output from the plurality of speakers 117 to be perceived by the user of the computing device 120 as originating from the location 408 of the user action, by causing the plurality of speakers 117 to cooperate to output the audio content in the buffer, and generating a plurality of audio signals provided to each of the plurality of speakers 117 of the computing device 120.

[0083] In the example of FIG. 4, when the location 408 of the user action does not intersect the UI element 406 (e.g., as determined using other methods of hit testing as described herein), the audio content in the buffer can be zero (e.g., null audio content 410). In this use case, the spatialization process may be applied to the zero in the buffer, and the spatialization process is applied to the null audio content 410, but the audio signal is provided to the speakers to output the null audio content for perception at location 408. However, since the null audio content is zero, the user will not receive any audio output from the speaker 117. In the example of FIG. 5, when the location 408 of the user action intersects the UI element 406 (e.g., as determined using other methods of hit testing as described herein), the buffer can be filled with audio content 500 (e.g., non-zero audio content value) corresponding to the sound of that UI element 406, the spatialization process can be applied to the audio content 500, and the resulting audio signal to the speakers can cause the audio content 500 to be perceived as being output at location 408.

[0084] In this way, when the user 110 gazes at a virtual button, for example, the sound corresponding to the virtual button can be inserted into an already active audio stream that is already generating a spatial audio output for perception at approximately the location of the button (e.g., at location 408 corresponding to the location of the UI element 406). In this example, the location input for the spatialization process can be switched from the location 408 of the user action (e.g., line of sight location) to the location of the virtual button (e.g., the center of the button) when, or immediately before, the audio content for the button is inserted into the audio stream.

[0085] For example, FIG. 6 shows an example where the location 408 of the user action is within the boundary 600 of the UI element 406 (e.g., a displayed visible boundary or a non-displayed boundary not visible to the user). As shown in FIG. 6, the location input into the spatialization process (in order to make the sound generated according to the audio content 500 be perceived as coming from a specific spatial location within the user's physical environment) can be switched from the location 408 of the user action to the location 602 of the UI element 406. In the example of FIG. 6, the location 602 of the UI element 406 is shown as being at the center of the rectangular UI element, but this is merely an illustration, and the shape and / or size of the UI element 406 may be any suitable shape and / or size, and the location of the UI element may be any location on or near the UI element 406 (e.g., within the boundary 600 of the UI element 406). In this example, the jump from location 408 to location 602 is small (e.g., the jump is not started until location 408 coincides with at least a part of the UI element 406, so it is smaller than the displayed size of the UI element 406), so the location jump may be imperceptible to the user even if the location jump occurs while the sound corresponding to the audio content 500 for the UI element 406 is being output (e.g., by the speaker 117).

[0086] In one or more implementations, when the audio content for a particular UI element is inserted into an audio stream (e.g., loaded into a buffer for that audio stream) and the location of that audio stream is set to the location of that particular UI element, a new continuous audio stream can be generated that moves with the location 408 of the user action (e.g., after a virtual button is clicked and the click sound of the virtual button is output using the previous audio stream snapped to location 602, as the user's line of sight moves away from the virtual button, it moves with the user's line of sight).

[0087] FIG. 7 shows an exemplary process 700 for providing an out-of-process audio effect according to one or more implementations. For purposes of explanation, process 700 is primarily described herein with reference to computing device 120 of FIG. 1. However, process 700 is not limited to computing device 120 of FIG. 1, and one or more blocks (or operations) of process 700 may be performed by one or more other components of other suitable devices. For further explanation, some of the blocks of process 700 are described herein as occurring continuously or linearly. However, multiple blocks of process 700 may be performed in parallel. Additionally, the blocks of process 700 need not be executed in the order shown, and / or one or more blocks of process 700 need not be executed and / or may be replaced by other operations.

[0088] In the example of FIG. 7, at block 702, an electronic device (e.g., computing device 120) can display a user interface element (e.g., UI element 406) (e.g., using display 115). For example, the user interface element may be displayed within a user interface window (e.g., user interface window 402 as in the examples of FIGS. 4 and 5) for an application (e.g., app 260), or may be displayed without surrounding or related UI windows. The user interface element may be managed by an underlying application (e.g., app 260) running on the electronic device.

[0089] In block 704, the electronic device may track the location of a user action (e.g., location 408) while, for example, a user interface element is being displayed. For example, the location of the user action may include a line-of-sight location (e.g., line-of-sight location 114). As another example, the location of the user action may include the location of at least a portion of the hand of a user of the electronic device. Tracking the location of the user action may include using one or more cameras and / or other sensors of the electronic device (e.g., including one or more cameras that capture an image of the user's eye(s) and / or one or more cameras that capture an image of the user's hand(s)) to track the location of the user action.

[0090] In block 706, while tracking the location of the user action, the electronic device may generate an audio stream corresponding to the tracked location of the user action, the audio stream including null audio content (e.g., zero such as in null audio content 410). For example, generating the audio stream may include applying a spatialization process to the null audio content based on the tracked location. In one or more implementations, generating the audio stream may include loading audio content (e.g., null audio content 410) into a buffer, providing the location of the user action to the spatialization process, applying the spatialization process to the audio content in the buffer to generate one or more audio output streams for respective ones of the one or more speakers, activating one or more audio components (e.g., amplifier, filter, etc.), and providing one or more audio streams through the one or more audio components for output by the one or more speakers (e.g., including the audio content to which the spatialization process has been applied).

[0091] In one or more implementations, an electronic device can determine whether to generate an audio stream before generating the audio stream. For example, the electronic device can determine that an audio stream should be generated when certain conditions of the electronic device are met. For example, the conditions can include that the electronic device is held, worn, or otherwise engaged by a user, that other audio content has already been generated, that the location of a user action has recently contacted or passed within a predetermined range of a UI element having an audio effect (e.g., freshness determined using a keep-alive timer that counts down from a predetermined value and is initiated by contact with or passage of the location of the user action), that at least one user interface element having an audio effect is displayed, and / or that the electronic device is not thermally limited (e.g., when the temperature of the electronic device or a part thereof is below a temperature threshold). In one or more implementations, the electronic device can stop generating the audio stream if one or more of the above conditions are no longer met while the audio stream is being generated.

[0092] In block 708, in response to determining that the location of the user action corresponds to the location of the user interface element (e.g., location 602), the electronic device may replace null audio content in the audio stream with audio content corresponding to the audio effect for the user interface element (e.g., audio content 500). For example, determining that the location of the user action corresponds to the location of the user interface element can include performing a hit test, as described herein (e.g., in connection with box 306 of FIG. 3A), to determine whether the location of the user action is within the boundary of the user interface element (e.g., boundary 600 of FIG. 6) for, e.g., at least a threshold amount of time.

[0093] In block 710, the electronic device may generate an audio output corresponding to an audio stream including audio content corresponding to an audio effect, using one or more speakers of the electronic device (e.g., speaker 117). For example, generating the audio output may include applying a spatialization process (e.g., a spatialization process applied to null audio content) to an audio stream including audio content corresponding to an audio effect, based on the location of a user interface element, and operating a plurality of speakers of the electronic device to generate an audio output according to the audio stream including audio content corresponding to the audio effect and to which the spatialization process has been applied based on the location of the user interface element. For example, the audio effect to which the spatialization process has been applied based on the location of the user interface element may be configured to be perceived as being emitted from the location of the user interface element by a user (e.g., user 110) of the electronic device. In one or more implementations, applying the spatialization process based on the location of the user interface element may include replacing, in the spatialization process, the tracked location of the user action with the location of the user interface element in response to a determination that the location of the user action corresponds to the location of the user interface element (e.g., and then applying the spatialization process to the audio content). In one or more other implementations, the spatialization process may continue to use the location of the user action even while the location of the user action corresponds to the location of the user interface element.

[0094] In one or more implementations, while generating an audio output corresponding to an audio stream that includes audio content corresponding to an audio effect, the electronic device may generate a new audio stream corresponding to a tracked location of a user action (e.g., location 408) and including null audio content (e.g., null audio content 410 such as all zeros). For example, the new audio stream that includes null audio content may then follow the location of the user action 408 that is away from the location of the user interface element (e.g., by continued application of the spatialization process).

[0095] In one or more embodiments, process 700 may also include receiving, in a system process of the electronic device (e.g., system process 300) (e.g., before displaying the user interface element), from an application (e.g., app process 350) running on the electronic device, a definition of an audio effect for a user interface element (e.g., within effect definition 252) as described herein in connection with box 302 of FIG. 3A. For example, the user interface element may be managed by the application. For example, generating the audio output may include generating the audio output (e.g., according to the definition) without providing the location of the user action to the application.

[0096] In one or more implementations, process 700 may also include identifying, by the system process (e.g., after or while generating the audio output), the intent of the user interacting with the application based on additional user input (e.g., as described herein in connection with box 312 of FIG. 3A), and, when the intent of the user is identified, providing the additional user input to the application (e.g., as described herein in connection with box 314 of FIG. 3A).

[0097] In one or more implementations, the definition of an audio effect can include one or more remote states of the audio effect, including an idle state, and / or the identification of one or more properties and corresponding sets of values for each of the one or more remote states.

[0098] In one or more embodiments, the definition of an audio effect is included in a set of definitions of effects for user interface elements, the set of definitions including the definition of the audio effect and the definition of an animation of user interface elements associated with transitions between two or more different states of one or more remote states of the effect.

[0099] In one or more implementations, the definition of an audio effect can be a declarative definition, and the definition of the audio effect can include the identification of a user interface element, the identification of a user action that triggers, and the identification of an audio effect (e.g., audio content 500) that is rendered when the user action that triggers corresponds to the user interface element.

[0100] In various examples described herein, an out-of-process effect is described as being generated by a system process when a user's gaze or hand hovers over or near a UI element displayed within the XR environment. However, it is also understood that an out-of-process effect can be provided by an electronic device that does not detect gaze and / or hand (e.g., gesture) input, and / or by an electronic device that receives user input via a trackpad, touchpad, mouse, touch screen, stylus, or other pointer-style device or component. For example, an application programming interface (API) provided on an XR-capable device can include subclasses that can be used to provide an out-of-process effect as described herein (e.g., a set of pointer-specific effects that are a subset of the effects available in an XR-capable device) based on input using a pointer-style device. In one or more implementations, one or more subclasses can map to one or more respective legacy effect APIs for effects of pointer-style devices or components. In this way, an API (or other cross-platform compatible definition of an effect applicable in various electronic devices having different hardware configurations) can be provided that functions automatically across various input types and / or across various platforms and that operates within the privacy constraints presented by an XR system that utilizes user-specific inputs such as gaze, hand, and / or gesture input.

[0101] In one or more implementations, subclasses can be used to define hidden styles that do not produce a hovering effect for any input type, and system styles that allow the system process to automatically select an effect. For example, a system or automatic style may be useful for representing the default effect for UI elements that may have one or more other built-in effects (e.g., for selecting an appropriate effect for a given context, such as based on a background property to enable internal resolution by a view). In one or more implementations, an out-of-process or remote effect can be defined to apply to some input types (e.g., to eye gaze or manual input) but not to others (e.g., not to pointer-style input). In one or more implementations, different out-of-process or remote effects can be defined to apply to different input types (e.g., a glow effect for eye gaze input, a lift effect for manual input, and / or a shape (morphing) effect for pointer-style input).

[0102] In the various examples described herein, effects such as the hovering effect, which is implemented as an out-of-process or remote effect, including a glow effect, a highlight effect, a lift effect, a custom effect, an effect including shape definitions (e.g., including static and dynamic shape definitions), a full UI element effect, a sub-element effect, etc. However, it is also understood that any or all of the various effects and / or their features described herein can also be provided by an application (e.g., as an application-generated effect, such as in a use case where the trigger for the effect is not a user-specific trigger, such as in a use case where the trigger corresponds to a pointer-style device or component).

[0103] As described above, one aspect of the present technology is the collection and use of data available from certain legitimate sources for providing out-of-process audio to an electronic device. The present disclosure contemplates that, in some cases, the collected data may include personal information data that can be used to uniquely identify or identify a particular person. Such personal information data can include audio data, voice data, demographic data, location-based data, online identifiers, phone numbers, email addresses, home addresses, encrypted information, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other personal information.

[0104] The present disclosure recognizes that the use of personal information data in the present technology can be a use that benefits the user. For example, personal information data can be used to provide out-of-process audio to an electronic device.

[0105] In the present disclosure, it is contemplated that entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data comply with well-established privacy policies and / or privacy practices. Specifically, such entities are expected to implement and consistently apply privacy practices that meet or exceed industry or government requirements for maintaining user privacy. Such information regarding the use of personal data should be prominently and easily accessible to the user and should be updated as the data collection and / or use changes. Personal information from users should be collected only for legitimate uses. Further, such collection / sharing should be done after receiving user consent or based on other legitimate grounds specified in applicable laws. Moreover, such entities should consider taking all necessary measures to protect and secure access to such personal information data and to ensure that others with access rights to the personal information data faithfully adhere to their privacy policies and procedures. Additionally, such entities can subject themselves to third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. In addition, the policies and practices should be tailored to the specific types of personal information data collected and / or accessed and should comply with applicable laws and standards, including jurisdiction-specific considerations that may impose higher standards. For example, in the United States, the collection or access to certain health data may be subject to federal and / or state laws such as the Health Insurance Portability and Accountability Act (HIPAA). On the other hand, health data in other countries may be subject to other regulations and policies and should be addressed accordingly.

[0106] Notwithstanding the foregoing, the present disclosure also contemplates embodiments that selectively prevent a user from using or accessing personal information data. That is, the present disclosure is intended that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, in an example of providing out-of-process audio to an electronic device, the technology may be configured to enable a user to select an “opt-in” or “opt-out” of participating in the collection and / or sharing of personal information data during or at any time after registration of the service. In addition to providing “opt-in” and “opt-out” options, the present disclosure is intended to provide notice regarding access or use of personal information. For example, the user may be notified when downloading an app that will access the user's personal information data, and then may be alerted again immediately before the personal information data is accessed by the app.

[0107] Furthermore, it is an aspect of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. The risk can be minimized by restricting data collection and deleting data when it is no longer needed. In addition, anonymization of data can be used to protect a user's privacy when applicable in certain health-related applications. Anonymization can be facilitated, when appropriate, by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level instead of at the address level or at a scale insufficient for face recognition), controlling how data is stored (e.g., aggregating data across users), and / or other methods such as differential privacy.

[0108] Therefore, while the present disclosure encompasses the use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that it is possible to implement those various embodiments without the need to access such personal information data. That is, the various embodiments of the present technology are not rendered inoperable by the absence of all or part of such personal information data.

[0109] FIG. 8 shows an exemplary computing device 800 in which aspects of the present technology may be implemented according to one or more implementations. Computing device 800 can be any computing device or server for generating the features and processes described above, including, without limitation, a laptop computer, a smartphone, a tablet device, a wearable device such as Google Glass or glasses, and / or a part thereof. Computing device 800 can include various types of computer-readable media and an interface for various other types of computer-readable media. Computing device 800 includes a persistent storage device 802, a system memory 804 (and / or buffer), an input device interface 806, an output device interface 808, a bus 810, a ROM 812, one or more processing units 814, one or more network interfaces 816, and / or subsets and variations thereof.

[0110] Bus 810 collectively represents all system buses, peripheral buses, and chipset buses that communicatively connect a number of internal devices of computing device 800. In one or more implementations, bus 810 communicatively connects one or more processing units(s) 814 to ROM 812, system memory 804, and persistent storage device 802. From these various memory units, one or more processing units(s) 814 retrieve the instructions to execute and the data to process in order to execute the processes of the present disclosure. The one or more processing units(s) 814 can be a single processor or a multi-core processor in different implementations.

[0111] ROM 812 stores static data and instructions required by one or more processing units(s) 814 of computing device 800 and other modules. On the other hand, persistent storage device 802 may be a read / write memory device. Persistent storage device 802 can be a non-volatile memory unit that stores instructions and data even when computing device 800 is off. In one or more implementations, a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) can be used as persistent storage device 802.

[0112] In one or more implementations, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive, etc.) can be used as the persistent storage device 802. Similar to the persistent storage device 802, the system memory 804 can also be a read-write memory device. However, unlike the persistent storage device 802, the system memory 804 can be a volatile read-write memory such as random access memory. The system memory 804 can store either instructions and data that one or more processing units (singular or plural) 814 may require during execution. In one or more implementations, the processes of the present disclosure are stored in the system memory 804, the persistent storage device 802, and / or the ROM 812. From these various memory units, one or more processing units (singular or plural) 814 retrieve the instructions to be executed and the data to be processed in order to execute the processes of one or more implementations.

[0113] The bus 810 is also connected to input and output device interfaces 806 and 808. The input device interface 806 enables a user to communicate information to and select commands for the computing device 800. Input devices that can be used with the input device interface 806 can include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 808 can enable, for example, the display of images generated by the computing device 800. Output devices that can be used with the output device interface 808 can include printer and display devices such as, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information.

[0114] One or more implementations can include a device that functions as both an input and output device, such as a touch screen. In these implementations, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic input, voice input, or tactile input.

[0115] Finally, as shown in FIG. 8, bus 810 also couples computing device 800 to one or more networks and / or one or more network nodes via one or more network interfaces 816. In this way, computing device 800 can be part of a network, such as a computer network (e.g., a local area network (LAN), a wide area network (WAN), or an intranet), or the Internet. Any or all components of computing device 800 can be used in conjunction with the present disclosure.

[0116] Implementations within the scope of the present disclosure can be implemented, in part or in whole, using a tangible computer-readable storage medium (or one or more types of multiple tangible computer-readable storage media) that encodes one or more instructions. The tangible computer-readable storage medium can also be substantially non-transitory.

[0117] A computer-readable storage medium can be any storage medium that can be read, written to, or otherwise accessed by a general-purpose or special-purpose computing device, including any processing electronic device and / or processing circuit capable of executing instructions. For example, without limitation, a computer-readable medium can include any volatile semiconductor memory such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. A computer-readable medium can also include any non-volatile semiconductor memory such as ROM, PROM, EPROM, EEPROM, NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and millipede memory.

[0118] Furthermore, a computer-readable storage medium can include any non-semiconductor memory such as optical disk storage devices, magnetic disk storage devices, magnetic tapes, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more implementations, a tangible computer-readable storage medium can be directly coupled to a computing device, and in other implementations, a tangible computer-readable storage medium can be indirectly coupled to a computing device via, for example, one or more wired connections, one or more wireless connections, or any combination thereof.

[0119] The commands can be made directly executable or can be used to develop executable commands. For example, the commands can be realized as executable or non-executable machine code, or as high-level language instructions that can be compiled to generate executable or non-executable machine code. Further, the commands can also be realized as data or can include data. Computer-executable instructions can also be structured in any format including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As will be recognized by those skilled in the art, details including but not limited to the number, structure, order, and structure of the instructions can be varied significantly without changing the basic logic, function, processing, and output.

[0120] The above considerations mainly refer to a microprocessor or multi-core processor that executes software, but one or more implementations are executed by one or more integrated circuits such as an ASIC or FPGA. In one or more implementations, such an integrated circuit executes instructions stored in the circuit itself.

[0121] The use of personally identifiable information should be well understood to comply with privacy policies and practices that generally meet or exceed industry or government requirements for maintaining user privacy. In particular, personally identifiable information data should be managed and handled to minimize the risk of unintended or unauthorized access or use, and the nature of the authorized use should be clearly shown to the user.

[0122] One of ordinary skill in the art will appreciate that the various illustrative blocks, modules, elements, components, methods, and algorithms described herein can be implemented as electronic hardware, computer software, or combinations of both. In the foregoing, to illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, components, methods, and algorithms have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. One of ordinary skill in the art may implement the described functionality in varying ways for each particular application. The various components and blocks may be arranged differently (e.g., arranged in a different order or partitioned differently) without departing from the scope of the technology of the present application at all.

[0123] It will be understood that any particular order or hierarchy of blocks in the disclosed processes is an example of an illustrative approach. Based on design preferences, it will be understood that the particular order or hierarchy of blocks in a process may be rearranged or that all of the illustrated blocks may be performed. Any of the blocks may be performed concurrently. In one or more implementations, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components (e.g., computer program products) and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0124] As used in the specification and claims of this application, the terms "base station", "receiver", "computer", "server", "processor", and "memory" all refer to electronic or other technical devices. These terms exclude a person or group of people. For purposes of this specification, the term "display" or "displaying" means displaying on an electronic device.

[0125] As used herein, the phrase "at least one" preceding a series of items modifies the list as a whole, rather than each element (i.e., each item) of the list, together with the term "and" or "or" separating any of the items. The phrase "at least one" does not require at least one selection of each of the listed items; rather, the phrase enables the meaning of including at least one of any one of the items, and / or at least one of any combination of the items, and / or at least one of each of the items. By way of example, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" each refer to only A, only B, or only C, any combination of A, B, and C, and / or at least one of each of A, B, and C.

[0126] The predicates "configured to", "operable to", and "programmed to" do not mean a particular physical or non-physical modification of an object, but rather are intended to be used interchangeably. In one or more implementations, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Similarly, a processor configured to execute code can be interpreted as a processor programmed to execute the code or operable to execute the code.

[0127] Phrases such as "one aspect", "that aspect", "another aspect", "some aspects", "one or more aspects", "one implementation", "that implementation", "another implementation", "some implementations", "one or more implementations", "one embodiment", "that embodiment", "another embodiment", "some embodiments", "one or more embodiments", "one configuration", "that configuration", "another configuration", "some configurations", "one or more configurations", "the technology of the present application", "the disclosure", "this disclosure", "other variations thereof", and similar phrases are for convenience, and do not imply that the disclosure regarding such phrases (singular or plural) is essential to the technology of the present application or that such disclosure applies to all configurations of the technology of the present application. The disclosure regarding such phrases (singular or plural) can apply to all configurations or one or more configurations. The disclosure regarding such phrases (singular or plural) can provide one or more examples. Phrases such as "aspect" or "some aspects" can refer to one or more aspects, and vice versa, and this applies similarly to the other aforementioned phrases.

[0128] The word "exemplary" is used in this specification to mean "serving as an example, instance, or illustration". Any embodiment described herein as "exemplary" or "an example" should not necessarily be construed as being preferred or advantageous over other implementations. Further, to the extent that terms such as "include", "have", etc. are used in the specification or claims, such terms are intended to be inclusive in the same manner as the term "comprise" when "comprise" is used as a transitional term in the claims.

[0129] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure, whether known now or later to become known to those skilled in the art, are hereby expressly incorporated by reference and intended to be embraced within the claims. Further, nothing disclosed herein is dedicated to the public, whether or not such disclosure is explicitly recited in the claims. No element of any of the claims is to be construed under the provisions of 35 U.S.C. § 112, paragraph 6, unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, the element is not recited using the phrase "step for".

[0130] The foregoing description is provided to enable a person skilled in the art to make and use the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language of the claims, and references to singular elements are not intended to mean "one and only one" unless specifically stated otherwise, but rather "one or more". Unless otherwise noted, the term "some" refers to one or more. Pronouns in the masculine form (e.g., he) include the feminine and neuter genders (e.g., she and it), and vice versa. Headings and subheadings, if any, are used for convenience only and do not limit the disclosure of the application.

Claims

Claim 1 A method comprising: In a system process of an electronic device, receiving, from an application running on the electronic device, a definition of an effect on a first user interface (UI) element managed by the application; Receiving user input by the system process while the first user interface element is being displayed by the electronic device without the effect being applied to the first user interface element; Rendering the effect on the first UI element according to the definition by the system process without providing the user input to the application, in response to a determination that the user input corresponds to the first user interface element displayed without the effect. Claim 2 The method according to claim 1, wherein the system process is an operating system process separate from an application process in which the application runs. Claim 3 Determining by the system process that the user input corresponds to the first user interface element displayed without the effect; The method according to claim 1, further comprising determining by the system process that a line of sight direction of a user of the electronic device intersects the first user interface element displayed without the effect. Claim 4 Determining by the system process that the user input corresponds to the first user interface element displayed without the effect; The method according to claim 1, further comprising determining by the system process by identifying a hand gesture at a location within a range of the first user interface element displayed without the effect. Claim 5 Determining by the system process that the user input corresponds to the first user interface element displayed without the effect; The method according to claim 1, further comprising determining by the system process that a line of sight direction of a user of the electronic device intersects the first user interface element for a period longer than a dwell time threshold. Claim 6 by the system process, the user input corresponding to the first user interface element displayed without the effect, identifying a plurality of UI elements having a potential interaction with the user input; the method according to claim 1, further comprising determining that the plurality of identified UI elements include the first UI element. **Claim 7** by the system process, identifying the intention of a user interacting with the application based on additional user input; the method according to claim 1, further comprising providing the additional user input to the application when the intention of the user is identified. **Claim 8** identifying the intention of the user includes identifying the intention of the user based on the user input and the additional user input, and the method further comprises providing the user input to the application after the first user interface is displayed using the effect in response to identifying the intention of the user. The method according to claim 7. **Claim 9** the definition of the effect including one or more remote states of the effect including an idle state; the method according to claim 1, including identifying a set of one or more properties and corresponding values for each of the one or more remote states. **Claim 10** the definition of the effect the method according to claim 9, further including identifying an animation of the first UI element associated with a transition between two or more different states among the one or more remote states of the effect. **Claim 11** the method according to claim 1, wherein the definition of the effect further includes identifying the shape of the effect. **Claim 12** the method according to claim 11, wherein the shape includes a dynamic shape. **Claim 13** the method according to claim 11, wherein the identification of the shape includes a selection of the shape from a plurality of predefined shapes. **Claim 14** the method according to claim 11, wherein the identification of the shape includes instructions for rendering a custom shape for the application. **Claim 15** the method according to claim 11, wherein the shape is inserted into the first UI element. **Claim 16** the method according to claim 11, wherein the shape is offset with respect to the first UI element. **Claim 17** The method according to claim 1, wherein the definition of the effect includes a definition of a single effect applied to a plurality of sub-elements of the first UI element.

18. The method according to claim 1, wherein the definition of the effect includes a definition of a plurality of effects applied to respective sub-elements of the first UI element.

19. The method according to claim 1, wherein the user input includes pointer-style user input.

20. Rendering the effect on the first UI element according to the definition includes rendering the effect according to the definition and based on background content at a location corresponding to the first UI element, according to the method of claim 1.

21. The method according to claim 1, wherein the definition of the effect includes a cross-platform compatibility definition applicable in the electronic device and at least one other electronic device having a hardware configuration different from that of the electronic device.

22. The first UI element is the first entry in a list having a current selection of a second entry in the list, In response to the determination that the user input corresponds to the first entry in the list, the system process renders the effect for the first entry in the list and an alternative effect for the second entry in the list without providing the user input to the application, according to the method of claim 1.

23. The method according to claim 1, wherein the effect includes an audio effect.

24. The method according to claim 1, wherein the effect is included in a group of effects, and when the effect is rendered, all effects in the group are rendered.

25. Rendering the effect on the first UI element according to the definition without providing the user input to the application by the system process is Receiving, by the system process, a description of a user interface of the application, the description including a description of the first UI element, rendering the user interface according to the description of the user interface by the system process; rendering the first UI element within the user interface according to the description and the definition of the first UI element by the system process, the method according to claim 1, comprising:

26. The definition of the effect is a declarative definition, and the definition of the effect includes identification of the first UI element, identification of user input that triggers, and identification of the effect that is rendered when the triggering user input corresponds to the first UI element, the method according to claim 1.

27. receiving, by the system process, from the application, a request for an illumination effect for virtual content managed by the application; identifying, by the system process, illumination conditions of a physical environment of the electronic device while the virtual content is being displayed by the electronic device; rendering, by the system process, the illumination effect on the virtual content based on the identified illumination conditions without providing the illumination conditions to the application, the method according to claim 1, further comprising:

28. receiving, by the system process, from the application, a request for a mirroring effect for virtual content managed by the application; identifying, by the system process, one or more characteristics of a physical environment of the electronic device while the virtual content is being displayed by the electronic device; rendering, by the system process, a reflection of the one or more characteristics of the physical environment on the virtual content based on the identified one or more characteristics without providing the one or more characteristics of the physical environment to the application, the method according to claim 1, further comprising:

29. receiving, by the system process, from the application, a request to anchor virtual content to a physical object within a physical environment of the electronic device; The method according to claim 1, further comprising: anchoring the virtual content to the physical object by the system process without providing the information associated with the physical object or the physical environment to the application according to the requirement.

30. The method according to claim 29, further comprising: providing the application with a confirmation that the virtual content is anchored to the physical object by the system process without providing the information associated with the physical object or the physical environment to the application.

31. The method according to claim 29, wherein the virtual content is non-interactive virtual content including context information about the physical object.

32. Anchoring the virtual content to the physical object includes: identifying the physical object in the physical environment by the system process; receiving rendering information of the virtual content from the application in the system process; rendering the virtual content at a location on the display of the electronic device corresponding to the location of the physical object in the physical environment by the system process. The method according to claim 29.

33. The method according to claim 32, further comprising: detecting the movement of the physical object with respect to the electronic device by the system process; and modifying the rendering of the virtual content to track the movement of the physical object with respect to the electronic device by the system process without providing the information about the movement of the physical object to the application.

34. The method according to claim 29, wherein the physical object includes an image in the physical environment.

35. The method according to claim 34, wherein the image includes a logo, and the virtual content includes context information of the logo.

36. The requirement of anchoring the virtual content to the physical object in the physical environment of the electronic device includes an electronic version of the image. The method according to claim 34.

37. ​ In response to the request including the electronic version of the image by the system process, monitoring the physical environment of the image during the operation of the electronic device during a period before displaying virtual content using the electronic version of the image; and further comprising, in response to detecting the image in the physical environment during the monitoring, displaying the virtual content, the method according to claim 36. **Claim 38** The method according to claim 29, wherein the virtual content includes the first UI element. **Claim 39** A non-transitory computer-readable medium storing instructions that, when executed by a processor, in a system process of an electronic device, cause the system to receive, from an application running on the electronic device, a definition of an effect on a first user interface (UI) element managed by the application; receive user input by the system process while the first user interface element is being displayed by the electronic device without the effect being applied to the first user interface element; and in response to a determination that the user input corresponds to the first user interface element displayed without the effect, cause the system process to render the effect on the first UI element according to the definition without providing the user input to the application. **Claim 40** The instructions further cause the system process to identify the intention of a user interacting with the application based on additional user input; and when the intention of the user is identified, cause the additional user input to be provided to the application, the non-transitory computer-readable medium according to claim 39. **Claim 41** The definition of the effect is a declarative definition, and the definition of the effect includes identification of the first UI element, identification of user input that triggers the effect, and identification of the effect that is rendered when the triggering user input corresponds to the first UI element, the non-transitory computer-readable medium according to claim 39. ​ ​ ​ ​ In the system process of an electronic device, receive a definition of an effect on a first user interface (UI) element managed by the application from the application running on the electronic device. While the first user interface element is being displayed by the electronic device without the effect being applied to the first user interface element, receive user input by the system process. In response to a determination that the user input corresponds to the first user interface element displayed without the effect, render the effect on the first UI element according to the definition by the system process without providing the user input to the application.

43. The instruction further causes the system to Identify the intention of the user interacting with the application based on additional user input by the system process. The system according to claim 42, wherein when the intention of the user is identified, the additional user input is provided to the application.

44. The definition of the effect is a declarative definition, and the definition of the effect includes identification of the first UI element, identification of the user input that triggers it, and identification of the effect to be rendered when the triggering user input corresponds to the first UI element. The system according to claim 42.

45. A method comprising: In an application process, identifying an application-managed user interface (UI) effect and a remote UI effect on a first UI element. Providing, by the application process, to a system process, a definition of the remote UI effect and an identification of the first UI element. Receiving, by the application process, user input corresponding to the first UI element. Rendering, by the application process, the application-managed UI effect on the first UI element.

46. The method according to claim 45, wherein the user input is a confirmed user input that is confirmed by the system process following a preliminary user input corresponding to the first UI element, and the remote UI effect is rendered by the system process. **Claim 47** The method according to claim 46, wherein the confirmed user input includes a confirmed user input to a second UI element generated by the system process in response to the preliminary user input to the first UI element. **Claim 48** The method according to claim 47, wherein the first UI element includes a list selection UI that is extended to include selectable options in response to the preliminary user input, and at least one of the selectable options includes the second UI element. **Claim 49** The method according to claim 46, wherein the first UI element represents an option within a list selection UI, the remote UI effect corresponds to a preliminary selection of the option within the list selection UI, and the application management UI effect corresponds to a completed selection of the option within the list selection UI.

Images