Out-of-process effects of electronic devices

Abstract

The remote user interface (UI) rendering effect provides an improvement in privacy and efficiency in a computer user input system. In one aspect, an application specifies a remote UI effect that is managed and rendered on UI elements separately from the application, such as by an effect component that runs outside of the application's operating system process. When user input indicates a preliminary interaction with a UI element, the remote UI effect can be rendered without the application's knowledge of the preliminary interaction, thus protecting the user's privacy from the application for the preliminary UI interaction.

Description

Technical Field

[0004] , , ,

[0003] ,

[0001] (Cross - reference to related applications) This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 358,070, entitled "OUT - OF - PROCESS EFFECTS FOR ELECTRONIC DEVICES", filed on July 1, 2022; U.S. Provisional Patent Application No. 63 / 402,435, entitled "OUT - OF - PROCESS EFFECTS FOR ELECTRONIC DEVICES", filed on August 30, 2022; U.S. Provisional Patent Application No. 63 / 449,945, entitled "OUT - OF - PROCESS AUDIO EFFECTS FOR ELECTRONIC DEVICES", filed on March 3, 2023; and U.S. Provisional Patent Application No. 63 / 470,952, entitled "OUT - OF - PROCESS EFFECTS FOR ELECTRONIC DEVICES", filed on June 4, 2023, the disclosures of each of which are hereby incorporated by reference in their entirety.

[0002] This disclosure relates to human - computer interface technology and computer privacy, including, for example, out - of - process effects for electronic devices.

Background Art

[0003] Operating system software generally provides an abstraction layer between user interface hardware and applications running on the operating system. Multiple applications can be run in separate operating system processes using corresponding user interface windows presented on a computer display managed by the operating system.

Brief Description of the Drawings

[0004] The specific features of the technology of this application are set forth in the appended claims. However, for purposes of illustration, several implementations of the technology of this application are shown in the following figures.

[0005] [Figure 1] This document provides an exemplary environment for realizing the aspects of this disclosure.

[0006] [Figure 2] This disclosure illustrates an exemplary system that provides an out-of-process effect.

[0007] [Figure 3A] This disclosure illustrates an exemplary method for rendering UI effects.

[0008] [Figure 3B] This disclosure illustrates an exemplary method for rendering UI effects.

[0009] [Figure 4] This disclosure illustrates an exemplary use case in which an audio stream containing null audio content is generated in response to the location of a user action.

[0010] [Figure 5] This disclosure illustrates exemplary use cases in which an audio stream, generated to be perceived at the location of a user action, is used to generate audio effects on user interface elements.

[0011] [Figure 6] This disclosure provides exemplary use cases in which the location of a user action is replaced with the location of a user interface element.

[0012] [Figure 7] A flowchart illustrating exemplary actions that may be performed to provide out-of-process audio effects according to aspects of this disclosure is provided.

[0013] [Figure 8] This document describes exemplary computing devices in which aspects of this disclosure can be realized. [Modes for carrying out the invention]

[0014] The detailed description below is intended to describe various configurations of the present technology and is not intended to represent only one configuration in which the present technology can be put into practice. The accompanying drawings are incorporated herein and constitute part of the detailed description. The detailed description includes certain details to provide a complete understanding of the subject technology. However, the present technology is not limited to the specific details shown herein and can be implemented using one or more other implementations. In one or more implementations, the structure and components are shown in block diagram form to avoid obscuring the concepts of the present technology.

[0015] Aspects of this disclosure include technologies for providing enhanced privacy and / or efficiency in computer user input systems. Any computer input system can benefit from these technologies, but systems that can capture personally identifiable user data, such as augmented reality systems that track a user's hand gestures and / or gaze location, can particularly benefit from these technologies.

[0016] In one aspect of this disclosure, a user input and rendering system can receive and render user input and filter user input that is available to a computer software application (or "App" as herein). User input filtering enables user privacy from the application by restricting the application's access to the filtered user input data. In some aspects, the computer user input and rendering system may be more trustworthy than the application running on the same computer, for example, if the user input and rendering system is provided by a trusted operating system vendor and the application is provided by a less trusted third-party application vendor.

[0017] In one embodiment, a user may receive rendered feedback of their preliminary interactions with user interface elements, such as buttons or scrollbars, without making the preliminary interactions available to the application itself. Preliminary interactions with UI elements may include intentional initial interactions with the user interface, such as exploring the application interface (e.g., to discover that a rectangle with a square inside is actually a scrollbar), and preliminary interactions may also include unintentional, or even unaware, interactions with the application's UI elements (e.g., when the user's eyes move across the user interface without the user paying attention to it).

[0018] Aspects of this disclosure provide techniques for efficiently preserving the privacy of a user's preliminary UI interactions from an application while still providing the user with rendered feedback of the preliminary interaction. Rendered feedback of preliminary interactions with UI elements may include, for example, rendered visual cues (e.g., glow, lift, shape change, transparency change, etc.) and / or audio cues when a hand gesture occurs near a UI element, when a user's hand hovers over or near a UI element, when a user's gaze location is near a UI element, or when a pointer controlled by a trackpad or pointer device (e.g., mouse, joystick, etc.) hovers over or near a UI element. In one aspect, when it is determined that a user intends to interact with the application (or a UI element of the application), user input may no longer be considered preliminary (e.g., considered confirmed), and any user input may be provided to the application. In one embodiment, an application can declare or define a rendered feedback effect of a preliminary interaction with a UI element before the preliminary interaction occurs, giving the application control over the nature of the preliminary interaction feedback effect even when the application is never aware of the preliminary interaction with that UI element. The application can provide the declaration or definition of the effect to an operating system or another software component for managing the rendering of the preliminary interaction. When the rendering of the effect is managed in an operating system process separate from the application that proves the declaration or definition of the effect, the effect is referred to herein as a remote effect or "out-of-process effect".

[0019] As described above, aspects of this disclosure may be applied to augmented reality systems. A physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic devices. A physical environment may include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly perceive and / or interact with a physical environment through their senses such as sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people perceive and / or interact with through electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. Using an XR system, user input may include tracking a person's physical movement, and accordingly, the XR system may render adjustments to one or more properties of one or more virtual objects simulated in XR in a manner that conforms to at least one physical law. For example, if a user's hand gesture is rendered in an XR system at a location near but not touching a rendered application UI element, the UI element may be rendered with a glow or jiggle, or possibly indicate a preliminary interaction with that UI element (for example, to indicate that the UI element is interactive and / or to guide the user toward targeting that UI element), without notifying the application of the preliminary user interaction. Alternatively, if a user's hand gesture touches or grasps a UI element (for example, and / or the user's hand gesture occurs at a location near or away from the UI element while the user's line of sight crosses the UI element), it may be determined that the user intends to interact with the UI element, and then user input indicating a touch or grasp of the UI element may be provided to the application.

[0020] Many different types of electronic user input and rendering systems can enable human users to perceive and / or interact with various XR environments. Examples include head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be positioned over a person's eyes (similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mountable system may have one or more speakers and an integrated opaque display. Alternatively, a head-mountable system may be configured to accept an external opaque display (e.g., a smartphone). A head-mountable system may incorporate one or more imaging sensors for capturing images or videos of the physical environment and / or one or more microphones for capturing sounds of the physical environment. A head-mountable system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical coupler, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology to project a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface.

[0021] Aspects of the present disclosure provide a user with a rendered confirmation of user engagement with an application user interface before providing user input to the application. The user interface (UI) can be presented to appear at a location within a physical environment that is remote from the user input device and / or remote from the display device that presents the user interface (e.g., within a three-dimensional XR display environment). For example, a physical movement of a mouse input device can be rendered as a movement of a mouse cursor on a display at a physical location different from the physical mouse (perhaps only a few inches). As another example, a gaze cursor can indicate the location of the user's gaze, and the location of the user's gaze is remote from one or more cameras facing the eye used to determine the gaze location. Similarly, hand gestures can be rendered as hand objects in a virtual space and presented to the user. Thus, the rendered effect of the user input can be rendered or presented to the user at a location physically separated from the physical location of the sensor that acquired the perceived user input. In a virtual space or other augmented reality environment, the user input can also occur at a location remote from the application UI and / or the display component that presents the representation of the user's hand, and may appear to the user to be at a location remote from the display component.

[0022] Furthermore, a user can move a hand and / or direct a gaze towards or near a rendered application user interface and / or perform a hand gesture for interaction with one displayed application UI without intending to interact with the application UI (e.g., during a conversation with another person not associated with an electronic device, due to the user's normal movement around the physical environment), and without intending to provide input to an application associated with another displayed application UI.

[0023] Aspects of the present disclosure include receiving, in a system process of an electronic device, a definition of an effect on a first user interface (UI) element managed by an application from the application running on the electronic device. While the first user interface element is being displayed by the electronic device without the effect applied to the first user interface element, the system process may receive user input. In response to a determination that the user input corresponds to the first user interface element displayed without the effect, the system process renders the effect on the first UI element according to the definition without providing the user input to the application. In one or more implementations, the system process can also provide lighting, mirroring, and / or anchoring, etc., without providing information regarding lighting, mirroring, and / or anchoring to the application, instead of the application.

[0024] FIG. 1 shows an exemplary environment 100 for implementing aspects of the present disclosure. In the exemplary environment 100, a user 110 interacts with a computing device 120 having a display 115. As shown, the computing device 120 can include one or more speakers, such as speaker 117 (e.g., multiple speakers operable to generate spatialized audio perceived by user 110 as being emitted from a location separate from the location of speaker 117 in the physical environment of user 110). In one or more examples described herein, user 110 may look at computing device 120 and the device may sense the user's line of sight location 114. In one or more implementations, the user's line of sight location can constitute user input to the device.

[0025] The computing device 120 may be a smartphone, tablet device, or wearable device such as a smartwatch or head-mounted portable system, including a display system capable of presenting a visualization of an augmented reality environment to the user 110. The computing device 120 may be powered by a battery and / or any other power source. In one example, the display system of the computing device 120 provides a stereoscopic presentation of the augmented reality environment, enabling the user to see a three-dimensional visual indication of the rendering of a particular scene. In one or more implementations, the computing device 120 is used in addition to, or instead of, accessing the augmented reality environment using the computing device 120.

[0026] The computing device 120 may include one or more cameras (e.g., a visible light camera, an infrared camera, etc.). Furthermore, the computing device 120 may include, but is not limited to, a variety of sensors capable of detecting user input, including cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, LiDAR sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, and near-field communication sensors. Furthermore, the computing device 120 may include hardware elements capable of receiving user input, such as hardware buttons or switches. User input detected by such sensors and / or hardware elements corresponds to a variety of input modalities for interacting with virtual content displayed in a given augmented reality environment. For example, such input modalities may include, but are not limited to, face tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, respiration, temperature, electroencephalogram, olfaction), speech or audio (e.g., specific hotwords), and activating buttons or switches. The computing device 120 may also detect and / or classify physical objects within the physical environment of the computing device 120.

[0027] In one or more implementations, the computing device 120 may be communicatively coupled to a base device. Such a base device may generally contain more computing resources and / or available power compared to the computing device 120. In one example, the computing device 120 may operate in various modes. For example, the computing device 120 may operate in a standalone mode, independent of any base device.

[0028] The computing device 120 can also operate in wireless tether mode (e.g., connected via a wireless connection to a given base device) in conjunction with a given base device. The computing device 120 may also operate in connected mode, where it is physically connected to the base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., charging the computing device 120 while the base device is physically connected).

[0029] When the computing device 120 operates in wireless tether mode or connected mode, at least part of processing user input and / or rendering the augmented reality environment can be offloaded to the base device, thereby reducing the processing load on the computing device 120. For example, in one implementation, the computing device 120 works in conjunction with the base device to generate an augmented reality environment that includes physical and / or virtual objects that enable different forms of interaction between the user and the augmented reality environment in real time (e.g., visual, auditory, and / or physical or tactile interaction). In one example, the computing device 120 provides a rendering of a scene corresponding to an augmented reality environment that can be perceived and interacted with in real time by the user. Additionally, as part of presenting the rendered scene, the computing device 120 may provide the user with sound and / or tactile feedback. The content of a given rendered scene may depend on available processing power, network availability and capacity, available battery power, and the current system workload.

[0030] The computing device 120 may also detect events occurring within the scene of the augmented reality environment. Examples of such events include detecting the presence of living beings such as people or pets, specific people, entities, or objects in the scene.

[0031] Figure 2 shows an exemplary system 200 that provides out-of-process effects. System 200 may be implemented, for example, on a computing device 120. System 200 includes an application (app) process 202 corresponding to an application 260, and a system process 204 including a rendering system 270 (e.g., a rendering process) and an effects component 280 (e.g., an effects process). In some embodiments, processes 202 and 204 may be separate processes created and managed by an operating system and run on a common device such as the computing device 120. Embodiments of system process 204 (e.g., including the rendering system 270, the effects component 280, and / or hit test 282) may be implemented in hardware, software, or a combination thereof.

[0032] As shown in Figure 2, the system process 204 includes both the rendering system 270 and the effects component 280. However, the disclosure is not limited thereto. For example, the rendering system 270 may run in a separate process from the effects component 280, the rendering system 270 and the effects component 280 may run in a common process, or the rendering system 270 and the effects component 280 may be further distributed to run in three or more processes, all of which may be separate from the app process 202. Similarly, in other embodiments, the app 260 may be distributed to run in two or more app processes. Operating system processes such as the app process 202 and the system process 204 can provide inter-process security and / or privacy when running on a common device such as the computing device 120 in Figure 1 or the computing device in Figure 8. For example, the app process 202 and the system process 204 may operate in different virtual memory spaces, controlled by the operating system and executed by the processor's memory controller hardware. Separate app and system processes can help prevent the app 260 from accessing data within the system process 204. In other embodiments, the system process 204, the rendering system 270, and the effect components 280 may include one or more operating system drivers that do not run in virtual memory space.

[0033] The operation of system 200 may include receiving user input and outputting rendered effects as feedback to the user input, which may be performed outside of the app process 202 and without knowledge of app 260. App 260 may provide the rendering system 270 with a description of its UI element(s) 250, and app 260 may provide a definition 252 of its effects to be rendered in response to future user input. When effect component 280 receives user input corresponding to the effect definition received from app 260, effect component 280 may cause the rendering system 270 to render the effect 256 as user output.

[0034] In an optional embodiment of system 200, the effect component 280 may know the location of a UI element (e.g., the location of a button or scroll bar) via an optional UI element 258 message from, for example, the rendering system 270. In one or more implementations, the effect component 280 may then perform an optional hit test 282 between the current location of the UI element and the current location of user input. In one or more other implementations, the hit test 282 may be separate from the effect component 280 and be performed by another system process that performs hit tests for several different purposes (e.g., for the effect component 280 and other components and / or processes). If the hit test 282 between user input and a particular UI element of app 260 (e.g., by the effect component 280 or another system process of computing device 120) identifies that a particular user input is associated with a particular UI element, the hit test 282 may identify a preliminary interaction with the particular UI element of app 260. In one or more implementations, the hit test 282 may include "fuzzy" hit test behavior that intentionally biases the hit test results toward interactive UI elements (e.g., toward non-interactive UI elements). For example, when the user's gaze is approximately midway between an interactive UI element (e.g., a button, slider, list, text entry field, etc.) and a non-interactive UI element (e.g., static text and / or an unlinked image), and / or when the user's gaze intersects the interactive and non-interactive elements at different depths, the hit test 282 may determine that the gaze is associated with an interactive UI element. Once a preliminary interaction is identified, the effect 256 may be rendered as user output without notifying the app 260 of user input or the identified interaction.Alternatively, when alternative user input is identified as a confirmed user interaction with a UI element, the effect component 280 or another system process may notify the app 260 of the confirmed interaction as a filtered user input message 254.

[0035] Effect definitions 252 can describe various effects that the app instructs the system process 204 to render on one or more UI elements. Each effect defined in effect definition 252 can correspond to a specific type of user input interaction with a particular UI element. In some embodiments, effect definition 252 can describe multiple effects that are rendered in response to a single user input interaction with a single UI element. For example, effect definition 252 can indicate that when the user's line of sight is within a specific range of a particular button UI element, the button may begin to emit light and / or sound up to a specific brightness level, and when the line of sight is within a closer distance range of the button, the UI element may both emit light at a brighter level and produce a quivering sound and / or buzzing sound, and when the line of sight is within a third closest range (perhaps directly gazing at the button and / or directly gazing at the button for at least a predetermined dwell time such as 1 second, 1 / 2 second, 1 / 10 second, or a smaller fraction of a second), the user interaction may be identified as a confirmed interaction (e.g., an interaction intended by the user as a button activation).

[0036] In an alternative, optional mode, app 260 may directly provide the effect component 280 with information describing the UI elements of app 260 (for example, instead of the effect component 280 receiving UI information from the rendering system 270). Additionally, instead of location-based hit testing, hit testing 282 may more generally determine that a particular user input corresponds to an interaction with a particular user interface element. For example, a user's verbal audio input saying "red button" may cause hit testing 282 to associate that audio input with the red button UI element.

[0037] In one embodiment, the effect definition 252 may be a declarative definition. In this embodiment, the app 260 can provide all the necessary information to external software components of the app process 202, such as the effect component 280, so that the app can render a desired effect without the app's knowledge or further participation. The declarative definition of an effect may include, for example, the identification of a UI element provided by a user interface framework or operating system, the identification of a triggering user input, and the identification of an effect to be rendered when the triggering user input corresponds to a first UI element. In one or more embodiments, the identified effect may include one selected from a plurality of default effects available from a system process (e.g., a highlight effect, a glow effect, or a lift effect), an indication for the system process to automatically select a default effect, or an instruction to render a custom effect provided by the app 260.

[0038] In one or more implementations, an identified effect may be applied to the entire UI element or to a subview or subelement of a UI element (e.g., an internal background subview of a text field, a slider subelement such as a slider thumb, a subelement of a segmented control element, a cell in a collection view, a cell in a table view, or a subelement of a date picker or other picker). In one or more implementations, the same effect may be applied to multiple subelements of a UI element. In one or more implementations, different default or custom effects may be applied to different subelements of a UI element. In an example of a collection of cells or a table, the effect may be defined based on heuristics, such as whether a given cell is highlightable or selectable. For example, these heuristics can model the intention to show a hovering effect when a cell is interactive (e.g., a zero effect can be used when the cell is non-interactive). In one or more implementations, if the app explicitly sets the effect style for a cell, the heuristics may be disabled and the explicit style may be used. For example, the hovering effect may be disabled for list rows, view hierarchies, etc., in an application. In one or more implementations, the definition of an effect may include multiple definitions of multiple effects (e.g., the same or different effects) for multiple sub-elements of a UI element (e.g., the same or different effects may be applied when a hover interaction is detected on or near a selectable date or time within a picker).

[0039] In one or more implementations, providing an effect definition 252 may include providing definitions for the shape of the effect and / or the shape of the underlying UI element. For example, the shape of an effect may include a capsule shape, a linear shape, a circular shape, or other default or custom shapes. In one or more implementations, the shape definition may include definitions for one or more of the following: corner shape, corner radius, corner curve, corner masking, and / or other corner and / or edge definitions for the shape. In one or more implementations, a shape may not have an inherent frame or size, and instead may be broken down into concrete shapes within a given frame (for example, this may provide more expressive shapes that can automatically adapt to the view to which the shape is associated). For example, a default capsule shape may be provided that can include corners that can be automatically resizable based on the frame provided for the shape.

[0040] In one or more implementations, a shape definition may include definitions of inset shapes (e.g., for effects that are inset into or occur within the boundaries of a UI element) and / or outset shapes (e.g., shapes that are outset into or occur outside the boundaries of a UI element, such as by padding via a negative inset). In one or more implementations, a shape definition may automatically insert rounded linear shapes that store concentric corner radii proportional to their size. In one or more implementations, a shape definition may include definitions of dynamic shapes (e.g., that can be resolved within a given context). For example, a dynamic shape definition can be used to create a shape that uses fractional corner radii. A dynamic shape may have dynamic edges and / or corners and / or may include dynamic transparency, brightness, or other dynamic features of the overall shape. In one or more implementations, a shape for an effect may be represented as a modification of the content shape of a view (e.g., a UI element) or the shape of its container. For example, app 260 can modify the effect shape of a button UI element to provide padding while maintaining its original shape heuristics. In one or more implementations, the shape may be defined to correspond to the container on which the shape is rendered. In one or more implementations, the shape may be defined as a zero shape, which leads to the use of an implicit default shape.

[0041] In one or more implementations, the effect definition 282 may be provided without a shape definition. In one or more implementations, when no shape definition is provided, a default shape may be determined by a system process. In one or more implementations, when no shape definition is provided for pointer-style input, the default preview shape for the pointer may be used for the effect shape. In one or more implementations, the hovering effect may be deactivated during user drag-touch input. In one or more implementations, when the hovering effect is applied (e.g., to other UI elements), drag-touch input (e.g., to some UI elements) may be disabled. In one or more implementations, UI elements may be provided from one or more files, such as a Universal Scene Description (USDZ) file. In another implementation, UI elements referenced in the app's declarative definition may be provided by the operating system.

[0042] Figure 3A illustrates an exemplary method for rendering UI effects. As shown, the method in Figure 3A may include receiving the definition of an effect for the app's UI elements in system process 300 (box 302). The app's UI elements may include any or all of the following: elements of a UI window such as virtual buttons, virtual switches, virtual sliders, virtual lists (e.g., dropdown lists), text entry fields, collection views (or their cells), or table views (or their cells); other interactive virtual content that a user can interact with to provide user input to the app; or non-interactive virtual content. The app's UI elements may be two-dimensional elements or renderings of three-dimensional (3D) models. The app's UI elements may include static virtual content (e.g., images, text, static shapes, borders, colors, etc.) and / or dynamic virtual content such as videos and / or 2D or 3D animations. The UI elements may include virtual content that is displayed within or as part of a surrounding UI window, or they may be standalone virtual content that is displayed separately from the application's other virtual content and / or other UI elements. User input may be received by a system process 300 (box 304), and if the received user input is determined (e.g., by the system process) to correspond to a UI element of the received effect definition (box 306), the effect may be rendered on the UI element (e.g., by the system process 300) according to the effect definition (box 308). In one embodiment, the operation of Figure 3A may be performed in a system process 300 separate from the application process 350 on which the application is run.

[0043] The method in Figure 3A may include optional additional actions, such as the system process 300 receiving additional user input different from the user input received in box 304 (box 310). Once the system process 300 identifies the user's intent to interact with the app or the app's UI elements based on the original and / or additional user inputs (box 312), some or all of the user inputs that identified the user's intent may optionally be provided (or otherwise made available) from the system process 300 to the app process 350 (box 314).

[0044] The UI elements referenced through Figures 3A and 3B may be part of the app's user interface. For example, a UI element may be defined by the app, and its purpose may be to enable a user to interact with the app (after the user's intent to interact with the app has been identified) in one or more implementations. In one embodiment, a UI element may include one or more properties, such as location, size, orientation, color, transparency, or brightness, and an effect definition may include a modification of one or more of those properties. For example, an effect definition for a button UI element may include a lift effect to increase the button's location property and a brightening effect to increase the button's brightness property. Furthermore, app UI elements may include default “remote states,” which correspond to default values ​​(one or more) of the UI element’s properties. The remote states of an app’s UI elements can be considered remote in that the app can define the UI element, its properties, and its remote states, while only system components in a separate system process (such as the rendering system 270 and the effect component 280) can manage or possess knowledge of the UI element’s remote states. This can allow for preliminary user interaction with a UI element to be rendered while remaining private from the app, by preventing the app from discovering what remote state the UI element is in at any given time.

[0045] In one embodiment, the UI elements referenced throughout Figures 3A and 3B may also be obtained from the app, from alternative sources such as the UI frameworks and file formats described above, or directly from the operating system. These UI elements provided by alternative sources can function as part of the app's user interface (such as the user interface of app 260 in Figure 2). UI elements from alternative sources may have associated properties and “remote state” defined by the alternative source and / or defined by the app using it. For example, even if a UI element may originate from an alternative source, such UI element may still be managed by the app using it, allowing system processes to render remote effects while maintaining privacy from the managing app.

[0046] In one embodiment, UI elements may be specified as a layer tree or a rendering tree. For example, application 260 may provide a layer tree or rendering tree of UI elements to a rendering system 270, and application 260 may provide an effect definition 252 including a layer tree or rendering tree of UI elements to an operating system or other component that manages out-of-process UI effects.

[0047] An effect definition (box 302) may be received from a specific application running in the application process 350, and may describe an effect to be rendered on a UI element belonging to that specific application. However, the implementation of this disclosure is not limited thereto. For example, an effect definition received in box 302 may be received indirectly from another source, the effect definition may be a predetermined effect, or the UI element may be a predetermined UI element. For example, a button UI element may be default, and a “lift” effect may be defaulted to occur when user input such as gaze, hand, or pointer is determined to correspond to a default button UI element (e.g., hovering on or near a default button UI element). In the case of default effects and / or default UI elements, the received effect definition may include a reference to such default element without fully defining it.

[0048] In one embodiment, the effect definition may identify an effect to be rendered for a UI element by identifying a remote state (with default rendering property values) that is used when user input is determined to correspond to a UI element. In a further embodiment, the effect definition may identify an animation used when transitioning between default remote states.

[0049] User input (such as that received in boxes 304 or 310 in Figure 3) can include many types of input from the user. For example, in addition to user input received from user input devices such as a mouse, keyboard, and / or audio input device, received user input may also include location and / or orientation tracking of the user's body or body part (such as fingertips or hands), gesture recognition (such as hand shape, hand movement, and / or facial expression), or the location of the user's optical gaze (e.g., gaze location 114). User input may include time or velocity measurements such as gaze dwell duration, which measures the duration for which the user's gaze remains at or within a specific distance of a particular location or UI element, or hover dwell, which measures the duration for which the user's hand or other body part remains within a specific two- or three-dimensional distance of a particular location or UI element.

[0050] In one embodiment, a UI effect (such as the one defined in box 302 and rendered in box 308 in Figure 3) may include any rendered output that the user can perceive, including audio, image, and / or haptic effects. An example of an audio effect on a UI element may include an audible sound designed to be perceived as emanating from the location of the UI element, or a distortion of audio already emanating from the UI element before the rendering of the audio effect. As an example of an image effect, the UI element may be displayed to the user or otherwise visually presented before the rendering of the effect, and the rendered effect may change the appearance or location of the UI element. For example, a glow effect may make an existing UI element glow. A highlight effect may add a new visual cue, such as a circle or checkmark, near the location of the UI element. In one embodiment, an effect definition may specify a “reverse” rendering effect, where the specified effect is generally rendered except when user input corresponds to the UI element.

[0051] In one embodiment, the correspondence between user input and a first UI element (box 306) may include identifying a plurality of user interface elements that the UI element may interact with, and then determining that the plurality of identified UI elements include the first UI element. For example, the user input may be a gaze location. The system process may perform a hit testing process that identifies a list of potential UI elements that the user may be trying to interact with, such as by identifying all UI elements from all applications located within a threshold distance of the gaze location, and / or identifying all UI elements where the gaze direction intersects. If an effect definition includes an app UI element in that list, the corresponding effect may be rendered on that app UI element. In use cases where the gaze direction intersects with a plurality of UI elements, each of which has an effect definition, the system process may render the effect on a UI element among the plurality of UI elements that are displayed closest to the user, and / or on a UI element among the plurality of UI elements where the gaze location intersects most centrally.

[0052] In one embodiment, a single effect may be defined for a group of UI elements. For example, once user input is determined for any of the UI elements in the group (306), the defined effect may be rendered on all UI elements in the group. In another example, user input is determined for only a subset of the UI elements in the group within box 306. Defining an effect for a group of UI elements may include an indication of which subset of UI elements in the group should be hit-tested. If a user input location (such as the location of a user's gaze or hand gesture) is within a certain proximity (e.g., by a system process or a default distance defined in the effect definition) of any element in the indicated subset of UI elements, the effect may be rendered on all UI elements in the group.

[0053] In one embodiment, a single effect definition can define a group of effects to be rendered. Once a correspondence to a UI element is determined (box 306), the UI element may be rendered with multiple different effects (box 308). For example, an effect definition for a button may include a combination of a lift effect (e.g., the button moving in the opposite direction to when it is pressed), a glow effect (e.g., brightening), and an audio cue (e.g., an audio tone emitted from the button).

[0054] In one embodiment, an effect definition can combine multiple embodiments described above. For example, a single effect definition may include multiple UI elements, multiple effects to render, and / or multiple types of user inputs that can trigger the rendering of one or more effects.

[0055] In one embodiment, even after identifying the user's intent to interact with the app (box 312), only a summary or subset of the user input may be provided to the app (box 314). For example, an intent to interact with the app may be identified when the user's gaze dwell time on an app button exceeds a threshold. However, instead of providing the app with the user's gaze location or gaze dwell time, the app may provide an indication that the user intended to press an app button without providing the app with any knowledge of the user's gaze.

[0056] A more complex example of an out-of-process effect combining the various aspects described above can be demonstrated using a list selection UI. A typical list selection UI task may involve presenting the user with a list of multiple options, including an indication of the current selection. Examples of list selection UIs include "combo boxes" in macOS and "dropdown list boxes" in Windows. Each option in the list may be rendered as a visual box containing text describing the option, for example, and the list may be presented as rows of unhighlighted text boxes, with the current selection highlighted in a first color.

[0057] In an exemplary out-of-process implementation of a list selection UI, application 260 can provide rendering system 270 with definitions of UI elements 250, including rows of text boxes. Each text box definition may include three default remote states, including 1) an unselected (idle) remote state with a highlight property set to a first color (or no highlight), 2) a current selected remote state with a highlight property set to a second color, and 3) a preliminary selected remote state with a highlight property set to a third color. Application 260 can further provide an indication that the first text box in the list is the current selection, and application 260 can provide effect definitions 252 to effect component 280 indicating how preliminary user input should affect the remote states of the text boxes. Rendering system 270 can render a list of text boxes, each with its corresponding remote state, and can provide effect component 280 with definitions of the text boxes, along with the current location of each text box, via UI element 258 messages. Effect component 280 can use these definitions to manage the remote states of the text boxes. When the effect component 280 receives user input that includes a location (for example, a gaze location or a hand gesture location), the effect component 280 can perform a hit test of the user input location against the text box location to determine the correspondence between the user input and one of the rendered text boxes.If the user input location corresponds to a second text box in a different list from the currently selected first text box, the effect component 280 may cause the rendering system 270 to render an effect 256, including a change in the highlight property from an unselected remote state to a pre-selected remote state, without notifying the app of any changes in the remote state and highlight properties of the user input location or the second text box.

[0058] In addition to or instead of the preliminary selection described above, when user input indicates a completed selection by the user, the app may be notified of the user's intention to change the current selection in the list. For example, after rendering the preliminary selection described above, if the app receives additional user input (box 310) that identifies the user's intention to change the current selection to a second box (e.g., when the gaze duration threshold is exceeded on the second box, or when a button press gesture occurs on the second box), the app may be notified of the user's intention to change the current selection in the list.

[0059] In one embodiment, the remote effect definition can specify animations to be rendered during transitions between remote states. For example, in the list selection UI example above, the remote state transition animation may be defined to include a fade transition from an unselected remote state to a pre-selected remote state, from a first highlight color (no highlight for the idle state) to a third highlight color (for the pre-selected state).

[0060] Figure 3B illustrates an exemplary method for rendering UI effects according to an aspect of the present disclosure. As shown, the method in Figure 3B may include the app process 350 identifying app-managed rendering effects (one or more) and remote rendering effects (one or more) for corresponding UI elements (box 552). Definitions of the identified remote rendering effects (one or more) may be provided by the app process 350 to a remote effects rendering service provider (box 554, such as the rendering system 270 and / or effect component 280 in Figure 2). User input may be received by the app process 350 for UI elements having corresponding app-managed effects (box 556). App-managed UI effects may be rendered by the app process 350 on their corresponding UI elements (box 558).

[0061] In one embodiment, the operation of the method in Figure 3B may be performed in an application process 350 separate from the system process 300 that manages the remote / out-of-process UI effects. In another embodiment, the method in Figure 3A may be an exemplary method for providing a remote effects rendering service, in which the effect definition provided in box 554 may be received in box 302. In yet another embodiment, additional user input provided in box 314 may be received in box 556.

[0062] In other embodiments, effects on preliminary user interaction may be identified as remote UI effects, and effects on confirmed user input may be identified as app management UI effects (box 552). In the list selection UI example described above, the preliminary selection rendering effect may be identified as a remote UI effect, while the completed selection rendering effect may be identified as an app management UI effect.

[0063] In one or more implementations, a delegate-based method may be provided to dynamically update the hovering effect based on changes in the view state of a UI element. In one or more implementations, a delegate-based method may be provided to dynamically customize the behavior of the hovering effect. In one or more implementations, additional customized definitions of remote effects may be provided (e.g., to provide the ability to match effects on multiple views). In one or more implementations, multiple hovering effects may consist of or be added to a single view (e.g., a single UI element). In one or more implementations, an automatic default shape may be provided for some or all effects, such as for hovering effects, focus effects, drag-and-drop effects, context menu previews, etc. In one or more implementations, a shape definition may define a shape as a clip shape (e.g., to automatically set appropriate properties on a given layer, such as automatically setting corner radii for a rounded rectangle or a mask for a custom shape). In one or more implementations, the shape of an effect may be resolved based on one or more device properties (e.g., based on the corner radius of the device display) or based on the container shape (e.g., the shape of the superview on which the shape is framed). In one or more implementations, a hovering effect may be defined based on the type of content associated with the UI element (e.g., image or video) and / or the input type of user input (e.g., gaze, hand, or pointer style). In one or more implementations, an effect may be defined to reveal additional UI information and / or content (e.g., depending on the gaze at a particular location), to extend a UI element to display more content, and / or to scale up or transform a UI element.In various implementations, out-of-process (e.g., remote) effects and / or application-managed effects can be defined and / or implemented using application programming interfaces (APIs) and / or block-based programming.

[0064] In one or more implementations, an application process (e.g., app process 202) can identify application-managed user interface (UI) effects and remote UI effects for a first UI element (e.g., in box 552) and provide a system process (e.g., system process 204) with a definition of the remote UI effect and identification of the first UI element. The application process can receive user input corresponding to the first UI element and render the application-managed UI effect on the first UI element. In one or more embodiments, the user input may include a preliminary user input corresponding to the first UI element (e.g., preliminary user input to the first UI element, as determined by a hit test) followed by a confirmed user input (e.g., confirmed user input to the first UI element, as determined by a hit test) that has been confirmed by the system process and on which the remote UI effect has been rendered by the system process. For example, the confirmed user input may include a confirmed user input to a second UI element generated by the system process in response to a preliminary user input to the first UI element. In one or more use cases, the first UI element may include a list selection UI that expands to include selectable options in response to preliminary user input, and at least one of the selectable options includes the second UI element. In one or more other use cases, the first UI element may represent an option within the list selection UI, a remote UI effect may correspond to a preliminary selection of an option within the list selection UI (e.g., it may be rendered and indicated in response to its detection), and an application management UI effect may correspond to a completed selection of an option within the list selection UI (e.g., it may be rendered and indicated in response to its detection).

[0065] In the various examples described herein, examples of remote UI effects include remote effects (e.g., glow, highlight, lift, jitter, buzz, etc.) that are generated in response to identifying preliminary user input that matches a user interface element. In another exemplary use case, one or more remote effects may be generated by a system process (e.g., system process 300 described herein) depending on environmental conditions and / or characteristics, without providing the application with any indication that a remote effect is being generated.

[0066] For example, the system process 300 may generate lighting effects for virtual content, such as rendered three-dimensional (3D) models and / or UI elements, in response to identifying lighting conditions in the physical environment of the computing device 120. For example, the system process 300 may receive a request from application 260 (for example, in relation to box 554) for lighting effects for virtual content managed by application 260 (e.g., rendered 3D models and / or user interface (UI) elements). The request for lighting effects may be a system-defined request for lighting effects, or it may include a definition of lighting effects defined by application 260 (e.g., a declarative definition). For example, the lighting effects may include brightening, darkening, or otherwise modifying the appearance of different parts of a UI element based on lighting conditions in the physical environment (e.g., the direction and / or brightness of one or more light sources). For example, the system process 300 may identify the lighting conditions of the physical environment while the virtual content is being displayed by the computing device 120. Identifying lighting conditions may include receiving sensor data and / or images from one or more cameras and / or sensors (for example, in box 304, with or in place of user input). The system process 300 can then render lighting effects on the virtual content based on the identified lighting conditions without providing the lighting conditions to the application 260.

[0067] As another example, system process 300 may generate a mirroring effect on virtual content (e.g., rendered 3D models and / or user interface (UI) elements) that mirrors one or more parts of the physical environment of computing device 120. For example, system process 300 may receive a request from application 260 (e.g., in relation to box 554) for a mirroring effect on virtual content managed by application 260. The request for a mirroring effect may be a request for a system-defined mirroring effect, or it may include a definition of a mirroring effect (e.g., a declarative definition) as defined by application 260. For example, the mirroring effect may include displaying an image of a part of the physical environment so that it appears to be reflected from a part of the (e.g., rendered 3D models and / or user interface (UI) elements) (e.g., a reflective surface or texture on the rendered 3D models and / or UI elements). For example, while user interface elements are displayed by the computing device 120, the system process 300 may identify one or more features of the physical environment of the computing device 120 (for example, by capturing one or more images of one or more parts of the physical environment). Identifying one or more features may include receiving sensor data and / or images from one or more cameras and / or sensors (for example, in box 304, together with or instead of user input) and identifying one or more features using the sensor data and / or images (for example, in box 306). The system process 300 can then render a reflection of one or more features of the physical environment on the UI elements based on the identified features, without providing the one or more features of the physical environment to the application.Rendering a reflection of one or more features on virtual content may include, for example, obtaining an image of a portion of the physical environment, modifying the image based on the position and / or shape of the virtual content and / or the reflectivity of the surface of the virtual content (e.g., rotating, scaling, warping, distorting), and rendering the modified image on part or all of the virtual content. For example, virtual content in the form of a reflective sphere may be provided by application 260, and system process 300 may display a distorted mirror image of the physical environment on the surface of the reflective sphere without providing application 260 with any information about the physical environment.

[0068] In one or more implementations, a system process of an electronic device, such as a computing device 120, can provide one or more other out-of-process services to an application, such as an application 260 (for example, without providing the application with information about the services, which can help protect the privacy of the electronic device user regarding the application). For example, in one or more implementations, a system process 300 can provide out-of-process anchoring for one or more applications on the computing device 120.

[0069] For example, in one or more embodiments, the system process 300 may receive a request (for example, in relation to box 554) from an application such as application 260 to anchor the application's virtual content (e.g., rendered 3D models and / or UI elements, such as UI elements, which may also provide one or more out-of-process effects) to a physical object in the physical environment of the computing device 120. The system process 300 may then, as requested, anchor the virtual content to the physical object (for example, as part of rendering in box 308) without providing application 260 with any information associated with the physical object or physical environment. For example, a physical object that may be requested for anchoring the application's virtual content may include a table, wall, chair, floor, horizontal plane, vertical plane, hand, finger, or any other physical object that can be identified by the computing device 120. For example, anchoring may be provided by the system process without providing application with any transformations, images, sensor data, or any other information describing the location of the physical object or physical environment or a physical object in the physical environment. For example, the system process may only provide the requesting application with an indication of the success or failure of the anchoring.

[0070] In one or more implementations, anchoring virtual content to a physical object may include the system process 300 identifying the physical object in the physical environment (e.g., using sensor data, one or more cameras, and / or computer vision operations). The system process 300 may also include receiving rendering information for the virtual content from the application 260, and using the rendering information, rendering the virtual content on the device's display to a location corresponding to the location of the physical object in the physical environment (e.g., a location on display 115 that makes the virtual content appear to be at the location of the physical object in the physical environment when the display 115 is viewed by a user). In various implementations, the rendering information received from the application 260 may include a rendered 3D model, a rendered UI element, or instructions to render a 3D model or UI element (e.g., a layer tree). Anchoring virtual content to a physical object may also include the system process 300 detecting the motion of the physical object relative to the computing device 120 (for example, using sensors and / or cameras on the computing device 120), and the system process 300 modifying the rendering of the virtual content to track the motion of the physical object relative to the computing device 120 without providing information about the motion of the physical object to the application.

[0071] In one or more implementations, the system process 300 can provide the application 260 with confirmation that virtual content is anchored to a physical object without providing the application with information associated with the physical object or physical environment.

[0072] In one or more implementations, virtual content anchored to a physical object by the system process 300 in out-of-process anchoring operation may be non-interactive virtual content containing contextual information about the physical object. In one or more implementations, the physical object may include images in the physical environment. For example, the image may include a logo (e.g., a company, team, or product logo). In one or more implementations, virtual content anchored to a physical object by the system process 300 in out-of-process anchoring operation may include contextual information about the physical object, such as contextual information about the logo (e.g., company information or product information).

[0073] For example, in one or more use cases, a request from an application 260 to a system process 300 for anchoring virtual content to physical objects in the physical environment of a computing device 120 may include an electronic version of an image. In response to a request including an electronic version of an image, the system process 300 may use the electronic version of the image to monitor the physical environment of the image while the device is operating during the period prior to displaying the virtual content. For example, in a use case where the image includes a logo, the electronic version of the image may include an electronic version of the logo, and the system process 300 may monitor the physical environment for an image that matches the electronic version of the image (e.g., within a threshold) (e.g., using computer vision operations on camera frames captured by a camera(s) 117).

[0074] The system process 300 can also display virtual content in response to detecting images in the physical environment during monitoring. For example, in a use case where the image includes a logo, the system process 300 may detect products (or advertisements or other physical objects) in the physical environment where the logo is printed or otherwise displayed (e.g., using computer vision operations), and the virtual content displayed in response may include information about the company or other product to which the logo corresponds. In various implementations, information about the company or other product may be stored locally on the computing device 120 before detecting the logo in the physical environment (e.g., to protect user privacy and prevent information about the detection of the logo in the user's environment from being transmitted from the device), or (e.g., with user permission) before display in response to the detection of the logo in the physical environment. The system process 300 may then continuously update the location of the displayed virtual content to track the location of the physical object (e.g., by anchoring the virtual content to the physical object outside the application process) while the physical object remains within the field of view of the computing device 120 (e.g., one or more cameras of the computing device 120).

[0075] In one or more implementations, multiple applications on the computing device 120 can provide anchoring requests to the system process 300. The system process 300 can hold multiple anchoring requests while monitoring the physical environment of the requested physical objects during the operation of the computing device 120, and can render virtual content for multiple applications (e.g., one or more anchored rendered 3D models or anchored UI elements) to display when individual requested physical objects for individual applications are detected.

[0076] Figure 4 illustrates an exemplary use case in which one or more UI elements 406 are displayed in a visible display area 400 of a computing device 120. For example, the display 115 of the computing device 120 (see, for example, Figure 1) may be operable to display one or more user interface windows 402 in the visible display area 400 of the display 115, each corresponding to an underlying application (e.g., app 260) running on the computing device 120. As shown, each of the user interface windows 402 may contain one or more user interface elements 406. For example, the user interface elements 406 may include virtual buttons, virtual switches, virtual lists (e.g., drop-down lists), tabs, scroll bars, application icons, and / or other interactable elements. As shown in Figure 4, in one or more implementations, the user interface elements 406 may be displayed separately from the user interface windows 402. As described herein, the system process 300 may render one or more effects on the user interface element 406 when user input to the computing device 120 corresponds to the UI element 406 (for example, when the user's gaze location 114 is within or within the boundaries of the UI element 406, and / or when the user's hand or finger (or other pointer) hovers over a location within or within the boundaries of the UI element 406 for a predetermined time or so). These effects may be out-of-process effects defined by the underlying application of the UI element 406 and rendered without providing user information (e.g., the location of user input) to the application.

[0077] As described herein, out-of-process effects associated with UI elements such as UI element 406 may include audio effects. For example, when UI element 406 is implemented as a virtual button, the user's hand and / or line of sight to the button or nearby may cause one or more speakers of the electronic device (e.g., a speaker array including speaker 117) to output a declaratively defined sound, such as a click, lifting, buzzing, bouncing, or other sound, so that the sound is perceived by the user of the electronic device as originating from the location of UI element 406. These audio effects may be provided by the system process 300 using or in combination with visual effects, such as highlight or glow effects.

[0078] In addition to protecting user privacy with respect to the underlying application of the UI element 406, providing these audio effects as out-of-process effects, as described herein, can be beneficial, for example, in reducing latency between user actions associated with the UI element 406 and the output of the corresponding audio effect. For example, if the audio effect is generated by the application on which the UI element 406 is displayed, delays may occur due to the time used to send user input information to the application, the time used by the application to generate the sound, and the time used for the sound to be output from the speaker(s). Furthermore, in the examples described herein where the audio output is generated so that it is perceived as emanating from the location of the UI element 406, the location of the UI element 406 on which the user input is occurring is used to generate the audio effect. Preparing the audio output by the application can reduce some of this latency, but the application running on the device may not have access to the location information for the user input, and therefore, in some implementations, it may not be possible to pre-prepare the sound for its own UI element(s).

[0079] Even when providing out-of-process audio effects generated by system process 300, the location of the UI element from which the audio effect is generated is unknown until a user action occurs at or near that location. Therefore, the dependency of the audio output on the location of the UI element can result in latency between the user action and the output resulting from the audio effect.

[0080] One option to reduce such latency is for a system process to pre-prepare an individual audio stream for each of the displayed UI elements 406. However, this can be undesirably costly in terms of processing power and / or memory usage, especially when there are multiple (e.g., many) UI elements 406 displayed simultaneously. Another option is to start an audio stream each time the user's gaze, hand, or finger intersects with the location of a UI element. However, this option still involves latency and can result in the generation of a large number of unused audio streams, especially in use cases where multiple (e.g., many) UI elements are displayed and the user is looking around the display area or making hand gestures without intending to interact with the UI elements. This can also be undesirably costly in terms of processing power and / or memory usage, as well as potentially costly.

[0081] According to an embodiment of this technology, an audio stream in progress can be continuously generated such that any non-zero audio content in the audio stream is perceived as being at (e.g., emanating from) a location 408 corresponding to a user action (e.g., the location of the user's hand or fingers and / or a gaze location 114 that the user's eyes are fixated on at any given time), and the audio stream in progress can follow location 408 as the user action location 408 moves around. As shown in Figure 4, if there are no UI elements at the user action location 408, the audio content of the continuously running audio stream may be null audio content 410 containing only zeros (e.g., silence may be output by the speaker 117 in correspondence, as perceived at the user action location 408). As shown in Figure 5, when a user action location 408 corresponds to a location of a UI element 406 (for example, in relation to box 306 in Figure 3A, according to a hit test as described herein), the audio content 500 (e.g., non-zero audio content) of a specific sound of that UI element 406 can be inserted into a running audio stream that is being generated to be perceived at that location for output (e.g., replacing null audio content 410 in the audio stream). For example, the audio content 500 may be declaratively defined by the application process 350, and the declarative definition of the audio content 500 may be provided to the system process 300 before the output of the audio content 500, as described herein. In this way, the audio effect on the UI element 406 (defined in the audio content 500) can be immediately rendered (e.g., box 308 in Figure 3A) for user perception at the location of the user interface element 406 when the user action location 408 coincides with the location of the user interface element 406, and can be output by the speaker 117.

[0082] In one or more implementations, the computing device 120 may include a buffer corresponding to a continuously operating audio stream, and spatialization processing may be applied to the audio content in the buffer (e.g., null audio content 410 or audio content 500) to generate an audio output that includes the content of the buffer as perceived at the user action location 408 (e.g., the user's gaze location 114 and / or the location of the user's hand or fingers), based on the user action location 408. For example, spatialization processing may generate multiple audio signals provided to each of the multiple speakers 117 of the computing device 120, causing the multiple speakers 117 to cooperate in outputting the audio content in the buffer, such that the audio output from the multiple speakers 117 is perceived by the user of the computing device 120 as originating from the user action location 408.

[0083] In the example in Figure 4, when the location 408 of the user action does not intersect with the UI element 406 (as determined, for example, by using hit testing in other ways as described herein), the audio content in the buffer may be zero (e.g., null audio content 410). In this use case, spatialization processing may be applied to zero in the buffer, and spatialization processing is applied to null audio content 410, but an audio signal is provided to the speaker to cause the null audio content to be output for perception at location 408, but because the null audio content is zero, the user will not receive any audio output from speaker 117. In the example in Figure 5, when the location 408 of a user action intersects with the UI element 406 (as determined, for example, using hit testing or other methods as described herein), the buffer may be filled with audio content 500 (e.g., a non-zero audio content value) corresponding to the sound of the UI element 406, spatialization processing may be applied to the audio content 500, and the resulting audio signal to the speaker may cause the audio content 500 to be perceived as being output at location 408.

[0084] In this way, when user 110 gazes at a virtual button, for example, the sound corresponding to the virtual button can be inserted into an already active audio stream that is already generating spatial audio output for perception at approximately the location of the button (for example, at location 408, which corresponds to the location of UI element 406). In this example, the location input for spatialization processing may be switched from the location of the user action 408 (e.g., gaze location) to the location of the virtual button (e.g., the center of the button) when or immediately before the audio content for the button is inserted into the audio stream.

[0085] For example, Figure 6 shows an example where the location 408 of a user action is within the boundary 600 of a UI element 406 (e.g., a visible boundary or a hidden boundary not visible to the user). As shown in Figure 6, the location input to the spatialization process (to make the sound generated according to the audio content 500 perceived as originating from a specific spatial location in the user's physical environment) can be switched from the location 408 of the user action to the location 602 of the UI element 406. In the example in Figure 6, the location 602 of the UI element 406 is shown as being at the center of a rectangular UI element, but this is merely illustrative, and the shape and / or size of the UI element 406 may be any appropriate shape and / or size, and the location of the UI element may be any location in or near the UI element 406 (e.g., within the boundary 600 of the UI element 406). In this example, the jump from location 408 to location 602 is small (for example, smaller than the visible size of UI element 406, as the jump does not begin until location 408 matches at least a portion of UI element 406), so the location jump may be imperceptible to the user even if the location jump occurs while sound corresponding to the audio content 500 for UI element 406 is being output (for example, by speaker 117).

[0086] In one or more implementations, audio content for a specific UI element may be inserted into an audio stream (e.g., loaded into a buffer for that audio stream), and once the location of that audio stream is set to the location of that specific UI element, a new continuous audio stream may be generated that moves along with the user action at location 408 (for example, a virtual button is clicked, and after the click sound of the virtual button is output using the previous audio stream snapped to location 602, a new continuous audio stream may be generated that moves along with the user's gaze as the user's gaze moves away from the virtual button).

[0087] Figure 7 shows an exemplary process 700 that provides an out-of-process audio effect in one or more implementation forms. For illustrative purposes, process 700 is primarily described herein with reference to the computing device 120 in Figure 1. However, process 700 is not limited to the computing device 120 in Figure 1, and one or more blocks (or operations) of process 700 may be performed by one or more other components of other suitable devices. For further illustrative purposes, some blocks of process 700 are described herein as occurring sequentially or linearly. However, multiple blocks of process 700 may occur in parallel. In addition, the blocks of process 700 do not need to be executed in the order shown, and / or one or more blocks of process 700 do not need to be executed and / or can be replaced by other operations.

[0088] In the example of Figure 7, in block 702, an electronic device (e.g., computing device 120) can display user interface elements (e.g., UI elements 406) (e.g., using display 115). For example, user interface elements may be displayed within a user interface window for an application (e.g., app 260) (e.g., user interface window 402 as in the examples of Figures 4 and 5), or they may be displayed without surrounding or associated UI windows. User interface elements may be managed by the underlying application (e.g., app 260) running on the electronic device.

[0089] In block 704, the electronic device may track the location of a user action (e.g., location 408) (e.g., while a user interface element is displayed). For example, the location of a user action may include a gaze location (e.g., gaze location 114). As another example, the location of a user action may include the location of at least a portion of the user's hands on the electronic device. Tracking the location of a user action may include tracking the location of a user action using one or more cameras and / or other sensors on the electronic device (e.g., one or more cameras that capture images of the user's eyes(s) and / or one or more cameras that capture images of the user's hands(s)).

[0090] In block 706, while tracking the location of a user action, the electronic device may generate an audio stream corresponding to the tracked location of the user action, the audio stream including null audio content (e.g., zeros in null audio content 410). For example, generating an audio stream may include applying a spatialization process to the null audio content based on the tracked location. In one or more implementations, generating an audio stream may include loading audio content (e.g., null audio content 410) into a buffer, providing the location of the user action to the spatialization process, applying the spatialization process to the audio content in the buffer to generate one or more audio output streams for one or more speakers, activating one or more audio components (e.g., amplifiers, filters, etc.), and providing one or more audio streams through one or more audio components (e.g., including audio content to which the spatialization process has been applied) for output by one or more speakers.

[0091] In one or more implementations, an electronic device may determine whether or not to generate an audio stream before generating it. For example, an electronic device may determine that an audio stream should be generated when certain conditions of the electronic device are met. For example, conditions may include: the electronic device is being held, worn, or otherwise engaged by a user; other audio content has already been generated; the location of a user action has recently been touched or passed over within a given range of a UI element having an audio effect (e.g., freshness determined using a keep-alive timer, such as a timer that counts down from a given value to zero and is started by the touching or passing over the location of the user action); at least one user interface element having an audio effect is visible; and / or the electronic device is not thermally limited (e.g., when the temperature of the electronic device or a part thereof falls below a temperature threshold). In one or more implementations, an electronic device may stop generating an audio stream if one or more of the above conditions are no longer met while the audio stream is being generated.

[0092] In block 708, in response to the determination that the location of a user action corresponds to the location of a user interface element (e.g., location 602), the electronic device may replace the null audio content in the audio stream with audio content corresponding to an audio effect on the user interface element (e.g., audio content 500). For example, determining that the location of a user action corresponds to the location of a user interface element may include performing a hit test (e.g., in relation to box 306 in Figure 3A) as described herein to determine whether the location of the user action is within the boundary of the user interface element (e.g., boundary 600 in Figure 6) (e.g., for at least a threshold time).

[0093] In block 710, an electronic device may use one or more speakers of the electronic device (e.g., speaker 117) to generate an audio output corresponding to an audio stream containing audio content corresponding to an audio effect. For example, generating an audio output may include applying a spatialization process (e.g., a spatialization process applied to null audio content) to an audio stream containing audio content corresponding to an audio effect, based on the location of a user interface element, and operating multiple speakers of the electronic device to generate an audio output according to an audio stream containing audio content corresponding to an audio effect and to which the spatialization process based on the location of a user interface element has been applied. For example, an audio effect to which the spatialization process based on the location of a user interface element has been applied may be configured so that it is perceived by the user of the electronic device (e.g., user 110) as emanating from the location of the user interface element. In one or more implementations, applying a spatialization process based on the location of a user interface element may include replacing the tracked location of a user action with the location of the user interface element in the spatialization process (e.g., then applying the spatialization process to the audio content) in response to a determination that the location of a user action corresponds to the location of a user interface element. In one or more other implementations, the spatialization process may continue to use the location of a user action, even while the location of the user action corresponds to the location of a user interface element.

[0094] In one or more implementations, while generating an audio output corresponding to an audio stream containing audio content corresponding to an audio effect, the electronic device may generate a new audio stream corresponding to a tracked location of a user action (e.g., location 408) and containing null audio content (e.g., null audio content 410, such as all zeros). For example, the new audio stream containing null audio content may then follow location 408 of a user action that is far removed from the location of a user interface element (e.g., by the continued application of spatialization processing).

[0095] In one or more embodiments, process 700 may also include, in a system process of the electronic device (e.g., system process 300), receiving definitions of audio effects for user interface elements (e.g., in effect definition 252) from an application running on the electronic device (e.g., app process 350) (e.g., as described herein in relation to box 302 in Figure 3A) (e.g., before displaying user interface elements). For example, user interface elements may be managed by the application. For example, generating an audio output may include generating an audio output (e.g., according to the definition) without providing the application with the location of a user action.

[0096] In one or more implementations, process 700 may also include, by a system process (for example, after or during the generation of an audio output), identifying the user's intent to interact with the application (for example, as described herein in relation to box 312 in Figure 3A) based on additional user input, and, once the user's intent has been identified, providing additional user input to the application (for example, as described herein in relation to box 314 in Figure 3A).

[0097] In one or more implementations, the definition of an audio effect may include the identification of one or more remote states of the audio effect, including an idle state, and / or one or more properties and corresponding sets of values ​​for each of the one or more remote states.

[0098] In one or more embodiments, the definition of an audio effect is included in a set of effect definitions for a user interface element, the set of definitions includes the definition of the audio effect and the definition of an animation for the user interface element associated with the transition between two or more different states of one or more remote states of the effect.

[0099] In one or more implementations, the definition of an audio effect may be a declarative definition, which may include the identification of a user interface element, the identification of a user action that triggers it, and the identification of an audio effect (e.g., audio content 500) that is rendered when the triggering user action corresponds to the user interface element.

[0100] In the various examples described herein, out-of-process effects are described as being generated by a system process when a user's gaze or hand hovers over or near a UI element displayed in the XR environment. However, it is also understood that out-of-process effects may be provided by electronic devices that do not detect gaze and / or hand (e.g., gesture) input, and / or by electronic devices that receive user input via a trackpad, touchpad, mouse, touchscreen, stylus, or other pointer-style device or component. For example, an application programming interface (API) provided on an XR-enabled device may include subclasses that can be used to provide out-of-process effects as described herein (e.g., a set of pointer-specific effects, which is a subset of the effects available on an XR-enabled device) based on input using a pointer-style device. In one or more implementations, one or more subclasses may map to one or more respective legacy effect APIs for effects of pointer-style devices or components. In this way, an API (or other cross-platform compatible definition of an effect applicable to various electronic devices with different hardware configurations) may be provided that works automatically across various input types and / or various platforms, and operates within the privacy constraints presented by XR systems that utilize user-specific inputs such as gaze, hand, and / or gesture input.

[0101] In one or more implementations, subclasses can be used to define hidden styles that do not produce a hovering effect for any input type, and system styles that allow a system process to automatically select an effect. For example, system or auto styles may be useful for representing a default effect for a UI element that may have one or more other built-in effects (e.g., to select an appropriate effect for a given context, such as based on the background property, to allow internal resolution by the view). In one or more implementations, out-of-process or remote effects may be defined to apply to some input types (e.g., gaze or manual input) but not to others (e.g., not to pointer-style input). In one or more implementations, different out-of-process or remote effects may be defined to apply to different input types (e.g., a glow effect for gaze input, a lift effect for manual input, and / or a shape (morphing) effect for pointer-style input).

[0102] In the various examples described herein, effects such as glow effects, highlight effects, lift effects, custom effects, effects including shape definitions (e.g., static and dynamic shape definitions), full UI element effects, and hovering effects implemented as sub-element effects are described as being applied as out-of-process or remote effects. However, it is also understood that any or all of the various effects and / or features described herein may also be provided by an application (e.g., as an application-generated effect, for example, in use cases where the effect trigger is not a user-specific trigger, for example, in use cases where the trigger corresponds to a pointer-style device or component).

[0103] As described above, one aspect of the technology is the collection and use of data available from certain legitimate sources to provide out-of-process audio to an electronic device. This disclosure assumes that, in some cases, such collected data may include personal data that uniquely identifies or can be used to identify a particular person. Such personal data may include audio data, voice data, demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, encrypted information, data or records relating to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other personal information.

[0104] This disclosure acknowledges that the use of personal data in this technology may be for the benefit of the user. For example, personal data may be used to provide out-of-process audio to electronic devices.

[0105] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transferring, storing, or otherwise using such personal data will adhere to well-established privacy policies and / or privacy practices. Specifically, such entities are expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding industry or government requirements for maintaining user privacy. Such information regarding the use of personal data should be conspicuously and readily accessible to users and should be updated as data collection and / or use changes. Personal data from users should be collected only for legitimate use. Furthermore, such collection / sharing should be done after obtaining user consent or on other legitimate grounds specified in applicable law. In addition, such entities should consider taking all necessary steps to protect and secure access to such personal data and to ensure that others with access to personal data faithfully adhere to those privacy policies and procedures. Furthermore, such entities may undergo third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be tailored to the specific types of personal data collected and / or accessed, and should conform to applicable laws and standards, including jurisdiction-specific considerations that may play a role in imposing higher standards. For example, in the United States, the collection or access to certain health data may be subject to federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA). Health data in other countries, on the other hand, may be subject to other regulations and policies and should be addressed accordingly.

[0106] Notwithstanding the foregoing, the Disclosure also envisions embodiments that allow a user to selectively prevent the use of or access to personal data. That is, the Disclosure intends that hardware and / or software elements may be provided to prevent or prevent access to such personal data. For example, in the example of providing out-of-process audio to an electronic device, the technology may be configured to allow a user to choose to “opt in” or “opt out” of participating in the collection and / or sharing of personal data during or at any time thereafter of registering for the service. In addition to providing “opt-in” and “opt-out” options, the Disclosure intends to provide notices regarding access to or use of personal data. For example, a user may be notified when downloading an app that will access their personal data, and then again immediately before the app accesses the personal data.

[0107] Furthermore, the intent of this disclosure is that personal data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data when it is no longer needed. In addition, where applicable in certain health-related applications, data anonymization can be used to protect user privacy. Anonymization may be facilitated, where appropriate, by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level or a scale insufficient for facial recognition), controlling how data is stored (e.g., aggregating data across users), and / or by other means such as differential privacy.

[0108] Therefore, although this disclosure extensively covers the use of personal data to implement one or more different disclosed embodiments, it is also conceivable that these different embodiments can be implemented without requiring access to such personal data. In other words, the different embodiments of the Technology are not rendered inoperable by the absence of all or part of such personal data.

[0109] Figure 8 shows an exemplary computing device 800 in which aspects of the present technology may be realized according to one or more implementation forms. The computing device 800 may be, and / or be part thereof any computing device or server for generating the features and processes described above, including, but not limited to, laptop computers, smartphones, tablet devices, wearable devices such as goggles or glasses. The computing device 800 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The computing device 800 includes a persistent storage device 802, system memory 804 (and / or buffers), input device interface 806, output device interface 808, bus 810, ROM 812, one or more processing units 814, one or more network interfaces 816, and / or subsets and variations thereof.

[0110] Bus 810 collectively represents all system buses, peripheral buses, and chipset buses that communicate with a number of internal devices of the computing device 800. In one or more implementations, bus 810 communicates with one or more processing units 814 to ROM 812, system memory 804, and persistent storage device 802. From these various memory units, one or more processing units 814 retrieve instructions to execute and data to process in order to perform the processes of this disclosure. One or more processing units 814 may be a single processor or a multi-core processor in different implementations.

[0111] ROM 812 stores static data and instructions required by one or more processing units 814 and other modules of the computing device 800. On the other hand, persistent storage device 802 may be a read / write memory device. Persistent storage device 802 may be a non-volatile memory unit that stores instructions and data even when the computing device 800 is off. In one or more implementations, a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) may be used as persistent storage device 802.

[0112] In one or more implementations, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) may be used as the persistent storage device 802. Similar to the persistent storage device 802, the system memory 804 may be a read-write memory device. However, unlike the persistent storage device 802, the system memory 804 may be a volatile read-write memory, such as random-access memory. The system memory 804 can store any instructions and data that one or more processing units 814 may need at runtime. In one or more implementations, the processes of this disclosure are stored in the system memory 804, the persistent storage device 802, and / or the ROM 812. From these various memory units, one or more processing units 814 retrieve the instructions to be executed and the data to be processed in order to execute the processes of one or more implementations.

[0113] Bus 810 also connects to input and output device interfaces 806 and 808. Input device interface 806 allows a user to communicate information to computing device 800 and select commands. Input devices that may be used with input device interface 806 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a “cursor control device”). Output device interface 808 may, for example, enable the display of images generated by computing device 800. Output devices that may be used with output device interface 808 may include, for example, printers and display devices such as liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, flexible displays, flat panel displays, solid-state displays, projectors, or any other devices for outputting information.

[0114] One or more implementations may include a device that functions as both an input and output device, such as a touchscreen. In these implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic input, voice input, or haptic input.

[0115] Finally, as shown in Figure 8, the bus 810 also connects the computing device 800 to one or more networks and / or one or more network nodes via one or more network interfaces 816. In this way, the computing device 800 can be part of a network of networks, such as a computer network (LAN, wide area network ("WAN"), or intranet), or the Internet. Any or all components of the computing device 800 can be used with this disclosure.

[0116] Implementations within the scope of this disclosure can be partially or completely realized using tangible computer-readable storage media (or multiple tangible computer-readable storage media of one or more types) that encode one or more instructions. The tangible computer-readable storage media may also be, in fact, non-transient.

[0117] A computer-readable storage medium can be any storage medium that can be read, written to, or otherwise accessed by a general-purpose or dedicated computing device, including any processing electronic equipment and / or processing circuitry capable of executing instructions. For example, but not limited to, a computer-readable medium can include any volatile semiconductor memory such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. A computer-readable medium can also include any non-volatile semiconductor memory such as ROM, PROM, EPROM, EEPROM, NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, Racetrack memory, FJG, and Millipede memory.

[0118] Furthermore, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage devices, magnetic disk storage devices, magnetic tapes, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more implementations, the tangible computer-readable storage medium may be directly coupled to a computing device, while in other implementations, the tangible computer-readable storage medium may be indirectly coupled to a computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0119] Instructions can be made directly executable or used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or as instructions in a high-level language that can be compiled to produce executable or non-executable machine code. Furthermore, instructions can also be implemented as data or contain data. Computer executable instructions can also be structured in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As will be recognized by those skilled in the art, details including, but not limited to, the number, structure, order, and structuring of instructions can be changed considerably without altering the basic logic, function, processing, and output.

[0120] The above discussion primarily refers to microprocessors or multicore processors that run software, but one or more implementations are performed by one or more integrated circuits, such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions stored within the circuit itself.

[0121] It is well understood that the use of personally identifiable information should be governed by privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for maintaining user privacy. In particular, personally identifiable information data should be managed and handled in a manner that minimizes the risk of unintended or unauthorized access or use, and the nature of authorized use should be clearly indicated to the user.

[0122] Those skilled in the art will understand that the various exemplary blocks, modules, elements, components, methods, and algorithms described herein can be implemented as electronic hardware, computer software, or a combination of both. Above, to demonstrate this hardware-software compatibility, the various exemplary blocks, modules, elements, components, methods, and algorithms have been generally described in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the design constraints imposed on the overall system and the specific application. Those skilled in the art will be able to perform the described functionality in various ways for each specific application. The various components and blocks may be arranged differently (for example, in a different order or divided in a different way) without departing entirely from the scope of the art of this application.

[0123] Any particular order or hierarchy of blocks in the disclosed process should be understood as an example of an exemplary approach. Based on design preferences, any particular order or hierarchy of blocks in the process may be rearranged, or all of the exemplary blocks may be executed. Any of the blocks may be executed simultaneously. Multitasking and parallel processing may be advantageous in one or more implementations. Furthermore, the separation of various system components in the implementations described above should not be understood as a requirement for all implementations, and the described program components (e.g., computer program products) and systems may generally be integrated into a single software product or packaged into multiple software products.

[0124] As used herein and in the claims, the terms “base station,” “receiver,” “computer,” “server,” “processor,” and “memory” all refer to electronic or other technical devices. These terms exclude persons or groups of persons. For the purposes of this specification, the terms “display” or “displaying” mean displaying on an electronic device.

[0125] When used herein, the phrase “at least one” preceding a set of items, along with the terms “and” or “or” separating any of the items, qualifies the list as a whole, rather than each element of the list (i.e., each item). The phrase “at least one” does not require the selection of at least one of each item listed; rather, it allows for meanings including at least one of any one of the items, and / or at least one of any combination of the items, and / or at least one of each of the items. For example, the phrases “at least one of A, B, and C” or “at least one of A, B, or C” refer, respectively, to A only, B only, or C only, any combination of A, B, and C, and / or at least one of each of A, B, and C.

[0126] The predicates “configured to,” “operable to,” and “programmed to” are not intended to imply any specific tangible or intangible modification of the object, but rather to be interchangeable. In one or more implementations, a processor configured to monitor and control operations or components may also mean that the processor is programmed to monitor and control operations, or that the processor is operable to monitor and control operations. Similarly, a processor configured to execute code may be interpreted as a processor that is programmed to execute code, or operable to execute code.

[0127] The phrases "one aspect," "that aspect," "another aspect," "several aspects," "one or more aspects," "one implementation," "that implementation," "another implementation," "several implementations," "one or more implementations," "one embodiment," "that embodiment," "another embodiment," "several implementations," "one or more implementations," "one configuration," "that configuration," "another configuration," "several configurations," "one or more configurations," "the technology of the Application," "disclosure," "this disclosure," "other variations thereof," and similar phrases are for convenience only and do not imply that disclosures relating to such phrases (singular or plural) are essential to the technology of the Application or that such disclosures apply to all configurations of the technology of the Application. Disclosures relating to such phrases (singular or plural) may apply to all configurations or one or more configurations. Disclosures relating to such phrases (singular or plural) may provide one or more examples. Phrases such as "aspect" or "several aspects" may refer to one or more aspects, and vice versa, as with the other aforementioned phrases.

[0128] The word “exemplary” is used herein to mean “to serve as an example, case, or illustration.” Any embodiment described herein as “exemplary” or “example” should not necessarily be construed as being preferable or advantageous over other forms of implementation. Furthermore, to the extent that terms such as “include” and “have” are used in the specification or claims, such terms are intended to be comprehensive in the same manner as the term “comprise,” as “comprise” is construed as when “comprise” is used as a transitional term in the claims.

[0129] All structural and functional equivalents of the elements of various aspects described herein, whether known to those skilled in the art or to become known thereafter, are expressly incorporated herein by reference and are intended to be included in the claims. Furthermore, nothing disclosed herein is to be made public, whether such disclosure is expressly enumerated in the claims. No element of any claim should be construed under Section 112(f) of the United States Patent Act unless the element is expressly enumerated using the phrase “means for” or, in the case of a method claim, the element is enumerated using the phrase “step for”.

[0130] The foregoing descriptions are provided to enable those skilled in the art to realize the various embodiments described herein. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein can also be applied to other embodiments. Therefore, the claims are not intended to limit themselves to the embodiments shown herein, but rather to encompass the entire scope consistent with the literal claims, and references to elements in the singular are not intended to mean "one and only one" unless otherwise specified, but rather "one or more." Unless otherwise specified, the term "some" refers to one or more things. Masculine pronouns (e.g., he) include feminine and neuter genders (e.g., she and her), and vice versa. Headings and subheadings, if any, are used for convenience only and do not limit the disclosure of this application.

Claims

1. It is a method, In the system process of an electronic device, the system receives definitions of effects on a first user interface (UI) element managed by an application running on the electronic device, While the first user interface element is displayed by the electronic device without the effect being applied to the first user interface element, the system process receives user input. A method comprising: determining that the user input corresponds to the first user interface element displayed without the effect, the system process rendering the effect on the first UI element in accordance with the definition without providing the user input to the application.

2. The method according to claim 1, wherein the system process is an operating system process separate from the application process on which the application is executed.

3. The system process determines that the user input corresponds to the first user interface element displayed without the effects. The method according to claim 1, further comprising determining by the system process that the user's line of sight of the electronic device intersects with the first user interface element displayed without the effect.

4. The system process determines that the user input corresponds to the first user interface element displayed without the effects. The method according to claim 1, further comprising determining by the system process identifying a hand gesture at a location within the range of the first user interface element displayed without the effect.

5. The system process determines that the user input corresponds to the first user interface element displayed without the effects. The method according to claim 1, further comprising determining by the system process that the user's line of sight of the electronic device intersects with the first user interface element for a period longer than a dwell time threshold.

6. The system process determines that the user input corresponds to the first user interface element displayed without the effects. Identifying multiple UI elements that have potential interactions with the user input, The method according to claim 1, further comprising determining that the plurality of identified UI elements include the first UI element.

7. The system process identifies the user's intent to interact with the application based on additional user input, The method according to claim 1, further comprising providing the additional user input to the application when the user's intent is identified.

8. The method of claim 7, wherein identifying the user intent includes identifying the user intent based on the user input and the additional user input, and the method further includes providing the user input to the application after the first user interface has been displayed using the effect, in response to the identification of the user intent.

9. The above definition of the effect is, One or more remote states of the effect, including the idle state, The method according to claim 1, comprising identifying one or more properties and corresponding sets of values ​​for each of the one or more remote states.

10. The above definition of the effect is, The method according to claim 9, further comprising identifying the animation of the first UI element associated with the transition between two or more different states among the one or more remote states of the effect.

11. The method according to claim 1, wherein the definition of the effect further includes identifying the shape of the effect.

12. The method according to claim 11, wherein the shape includes a dynamic shape.

13. The method according to claim 11, wherein the identification of the shape includes selecting the shape from a plurality of predefined shapes.

14. The method according to claim 11, wherein the identification of the shape includes an instruction to render a custom shape for the application.

15. The method according to claim 11, wherein the shape is inserted into the first UI element.

16. The method according to claim 11, wherein the shape is outset relative to the first UI element.

17. The method according to claim 1, wherein the definition of the effect includes the definition of a single effect applied to multiple sub-elements of the first UI element.

18. The method according to claim 1, wherein the definition of the effect includes definitions of a plurality of effects applied to each of the plurality of sub-elements of the first UI element.

19. The method according to claim 1, wherein the user input includes pointer-style user input.

20. The method according to claim 1, wherein rendering the effect on the first UI element according to the definition comprises rendering the effect according to the definition and based on background content in a location corresponding to the first UI element.

21. The method according to claim 1, wherein the definition of the effect includes a cross-platform compatible definition applicable to the electronic device and at least one other electronic device having a different hardware configuration from the electronic device.

22. The first UI element is the first entry in the list having the current selection of the second entry in the list, The method according to claim 1, wherein, in response to the determination that the user input corresponds to the first entry in the list, the system process renders the effect for the first entry in the list and the alternative effect for the second entry in the list without providing the user input to the application.

23. The method according to claim 1, wherein the effect includes an audio effect.

24. The method according to claim 1, wherein the effect is included in a group of effects, and when the effect is rendered, all effects in the group are rendered.

25. The system process renders the effect on the first UI element according to the definition without providing the user input to the application, The system process receives a description of the user interface of the application, the description of which includes a description of the first UI element. The system process renders the user interface in accordance with the description of the user interface, The method according to claim 1, comprising rendering the first UI element within the user interface in accordance with the description and definition of the first UI element by the system process.

26. The method according to claim 1, wherein the definition of the effect is a declarative definition, and the definition of the effect includes identifying the first UI element, identifying a triggering user input, and identifying the effect that is rendered when the triggering user input corresponds to the first UI element.

27. In the aforementioned system process, the system receives a request from the application for lighting effects for virtual content managed by the application, While the virtual content is being displayed by the electronic device, the system process identifies the lighting conditions of the physical environment of the electronic device. The method according to claim 1, further comprising rendering the lighting effect on the virtual content by the system process based on the identified lighting conditions, without providing the lighting conditions to the application.

28. In the aforementioned system process, the system receives a request from the application for a mirroring effect on the virtual content managed by the application, While the virtual content is being displayed by the electronic device, the system process identifies one or more features of the physical environment of the electronic device. The method according to claim 1, further comprising the system process rendering a reflection of the one or more identified features of the physical environment onto the virtual content, without providing the one or more features of the physical environment to the application, based on the one or more identified features.

29. In the aforementioned system process, the system receives a request from the application to anchor virtual content to a physical object within the physical environment of the electronic device, The method according to claim 1, further comprising, in response to the request, the system process anchoring the virtual content to the physical object without providing the application with information associated with the physical object or the physical environment.

30. The method according to claim 29, further comprising providing the application with confirmation that the virtual content is anchored to the physical object, without providing the application with the information associated with the physical object or the physical environment, by the system process.

31. The method according to claim 29, wherein the virtual content is non-interactive virtual content that includes contextual information relating to the physical object.

32. Anchoring the virtual content to the physical object is The system process identifies the physical objects within the physical environment, The system process receives rendering information for the virtual content from the application, The method according to claim 29, further comprising rendering the virtual content on the display of the electronic device to a location corresponding to the location of the physical object in the physical environment by the system process.

33. The system process detects the motion of the physical object relative to the electronic device, The method of claim 32, further comprising modifying the rendering of the virtual content to track the motion of the physical object relative to the electronic device without providing the application with information regarding the motion of the physical object.

34. The method according to claim 29, wherein the physical object includes an image within the physical environment.

35. The method according to claim 34, wherein the image includes a logo, and the virtual content includes contextual information of the logo.

36. The method of claim 34, wherein the request to anchor the virtual content to the physical object in the physical environment of the electronic device includes an electronic version of the image.

37. The system process, in response to the request including the electronic version of the image, Using the electronic version of the aforementioned image, the physical environment of the image is monitored while the electronic device is operating during the period prior to displaying the virtual content. The method according to claim 36, further comprising displaying the virtual content in response to detecting the image in the physical environment during the monitoring.

38. The method according to claim 29, wherein the virtual content includes the first UI element.

39. A non-temporary computer-readable medium for storing instructions, wherein the instructions, when executed by a processor, In the system process of an electronic device, the system receives definitions of effects for a first user interface (UI) element managed by an application running on the electronic device. While the first user interface element is displayed by the electronic device without the effect being applied to the first user interface element, the system process receives user input. A non-temporary computer-readable medium, in which, upon determination that the user input corresponds to the first user interface element displayed without the effect, the system process causes the effect to be rendered on the first UI element in accordance with the definition without providing the user input to the application.

40. The previous instruction further, The system process identifies the user's intent to interact with the application based on additional user input. The non-temporary computer-readable medium according to claim 39, which causes the application to provide the additional user input when the user's intent is identified.

41. The non-temporary computer-readable medium according to claim 39, wherein the definition of the effect is a declarative definition, and the definition of the effect includes identifying the first UI element, identifying a triggering user input, and identifying the effect to be rendered when the triggering user input corresponds to the first UI element.

42. It is a system, Processor and A storage device for storing instructions, wherein when an instruction is executed by the processor, it is stored in the system. In the system process of an electronic device, the system receives definitions of effects for a first user interface (UI) element managed by an application running on the electronic device. While the first user interface element is displayed by the electronic device without the effect being applied to the first user interface element, the system process receives user input. A system in which, upon determining that the user input corresponds to the first user interface element displayed without the effect, the system process causes the effect to be rendered on the first UI element according to the definition without providing the user input to the application.

43. The aforementioned instruction further instructs the system, The system process identifies the user's intent to interact with the application based on additional user input. The system according to claim 42, wherein when the user's intent is identified, the system causes the application to provide the additional user input.

44. The system according to claim 42, wherein the definition of the effect is a declarative definition, and the definition of the effect includes identifying the first UI element, identifying a triggering user input, and identifying the effect to be rendered when the triggering user input corresponds to the first UI element.

45. It is a method, In the application process, the application management user interface (UI) effect and remote UI effect are identified for the first UI element. The application process provides the system process with the definition of the remote UI effect and the identification of the first UI element, The application process receives user input corresponding to the first UI element, A method comprising rendering the application management UI effect on the first UI element by the application process.

46. The method of claim 45, wherein the user input includes confirmed user input, which is confirmed by the system process following a preliminary user input corresponding to the first UI element, and the remote UI effect is rendered by the system process.

47. The method according to claim 46, wherein the confirmed user input includes confirmed user input to a second UI element generated by the system process in response to the preliminary user input to the first UI element.

48. The method according to claim 47, wherein the first UI element includes a list selection UI that expands to include selectable options in response to the preliminary user input, and at least one of the selectable options includes the second UI element.

49. The method according to claim 46, wherein the first UI element represents an option in a list selection UI, the remote UI effect corresponds to a preliminary selection of the option in the list selection UI, and the application management UI effect corresponds to a completed selection of the option in the list selection UI.

Images