Out-of-process hit testing for electronic devices

Abstract

The invention relates to out-of-process hit testing for electronic devices. Out-of-process hit testing may provide enhanced privacy and improved efficiency in a computer user input system. In an aspect, an application specifies a control style for a UI window for separate management from the application, such as by a system process running outside the application process. When a user input is received at a location corresponding to a portion of the application UI separate from the UI window that has specified the control style, the user input may be redirected to the UI window that has specified the control style.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 470,955, filed on June 4, 2023, entitled “OUT-OF-PROCESS HIT-TESTING FOR ELECTRONIC DEVICES,” the disclosure of which is hereby incorporated herein in its entirety. Technical Field

[0003] The present disclosure relates to human-computer interface technology and computer privacy, including, for example, out-of-process hit testing for electronic devices. Background Art

[0004] Operating system software typically provides an abstraction layer between the user interface hardware and applications running on top of the operating system. Multiple applications with corresponding user interface windows can be presented on a display managed by the operating system. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Certain features of the subject technology are set forth in the appended claims. However, for illustrative purposes, several specific implementations of the subject technology are set forth in the following figures.

[0006] Figure 1 An exemplary environment for practicing aspects of the present disclosure is illustrated.

[0007] Figure 2 An exemplary system for providing out-of-process hit testing in accordance with aspects of the subject disclosure is illustrated.

[0008] Figure 3A and Figure 3B An exemplary method for providing out-of-process UI services in accordance with aspects of the subject disclosure is illustrated.

[0009] Figure 4 An exemplary use case of simultaneously displaying multiple UI windows of multiple applications according to aspects of the present subject disclosure is illustrated.

[0010] Figure 5 An exemplary use case of displaying a foreground UI window in accordance with aspects of the subject disclosure is illustrated.

[0011] Figure 6 An exemplary use case in which UI windows including a foreground UI window are displayed to appear to be distributed in a three-dimensional manner according to aspects of the subject disclosure is illustrated.

[0012] Figure 7An exemplary use case for receiving user input at a UI element while a foreground window is displayed in accordance with aspects of the subject disclosure is illustrated.

[0013] Figure 8 An exemplary use case for receiving user input at a location remote from application UI elements while a foreground UI window is displayed in accordance with aspects of the subject disclosure is illustrated.

[0014] Fig. 9 A flow diagram illustrating illustrative operations that may be performed to provide out-of-process hit testing in accordance with aspects of the subject disclosure is illustrated.

[0015] Fig.10 An exemplary computing device is illustrated upon which aspects of the present disclosure may be implemented. DETAILED DESCRIPTION

[0016] The detailed description set forth below is intended to be a description of various configurations of the subject technology and is not intended to represent the only configuration by which the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, but can be practiced using one or more other specific implementations. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concepts of the subject technology.

[0017] Aspects of the present disclosure include techniques for providing enhanced privacy and / or improved efficiency in computer user input systems. While any computer input system may benefit from these techniques, systems capable of capturing personally identifiable user data (such as extended reality systems that track a user's gestures and / or eye gaze location) may particularly benefit from these techniques.

[0018] In one aspect of the present disclosure, a user input and rendering system may receive and render user input, and may filter the user input for use in a computer software application (or "app" herein). User input filtering enables user privacy for an application by at least limiting the application's access to the user's filtered input data. In some aspects, a computer user input and rendering system may be more trusted than an application running on the same computer because, for example, the user input and rendering system is provided by a trusted operating system vendor, while the application is provided by a less trusted third-party application vendor. In addition, a user may explicitly provide authorization to access user input information to the operating system, and may not provide the same authorization to an application, such as a third-party application.

[0019] In one aspect, a user may receive rendered feedback of a user's preliminary interaction with a user interface element (such as a button or scroll bar) without making the preliminary interaction available to the application itself. The preliminary interaction with the UI element may include an intentional initial interaction of the user with the user interface, such as exploration of the interface of the application (e.g., discovering that a rectangle with a square inside is actually a scroll bar), and the preliminary interaction may also include an unintentional and even unconscious interaction with the UI element of the application (e.g., when the user's eyes move over the user interface while the user is not paying attention to the user interface).

[0020] Aspects of the present disclosure provide techniques for effectively retaining the privacy of a user's preliminary UI interaction to an application while still providing the rendered feedback of the preliminary interaction to the user. The rendered feedback of the preliminary interaction with a UI element may include, for example, a rendered audio prompt when a gesture occurs near a UI element, or when the user's gaze position is near a UI element or hovers near a UI element, the UI element may begin to emit light. On the one hand, when it is determined that the user intends to interact with the application (or the UI element of the application), the user input may be considered no longer preliminary, and some user input may be provided to the application. On the one hand, the application may make a declaration or definition of the rendered feedback effect of the preliminary interaction with the UI element before the preliminary interaction occurs, thereby giving the application control of the nature of the preliminary interaction feedback effect, even when the application has never been informed of the preliminary interaction with its UI element. The application may provide the declaration or definition of the effect to the operating system or another software component for managing the rendering of the preliminary interaction. When the rendering of the effect is managed in an operating system process separated from the application that provides the declaration or definition of the effect, the effect is referred to as a remote effect or "out-of-process effect" in this article. As described herein in conjunction with various examples and / or use cases, an operating system process separate from an application can provide other out-of-process services on behalf of the application without providing information about the out-of-process services to the application (e.g., including out-of-process hit testing, which may include out-of-process hit test redirection, as described in further detail below).

[0021] As described above, various aspects of the present disclosure may be applied to extended reality systems. A physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, user input may include tracking a person's physical movements, and in response, the XR system may render adjustments to one or more characteristics of one or more virtual objects simulated in the XR in a manner that complies with at least one physical law. For example, if a user's gesture is rendered in the XR system at a location close to but not touching a rendered application UI element, the UI element may be rendered with a glow or shake or otherwise indicate a preliminary interaction with the UI element without notifying the application of the preliminary user interaction. In an alternative example, if the user's gesture touches or grabs the UI element, it may be determined that the user intentionally intended to interact with the UI element, and user input indicating the touch or grab of the UI element may then be provided to the application.

[0022] Many different types of electronic user input and rendering systems can enable human users to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some specific implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0023] Various aspects of the present disclosure provide rendered confirmation of user engagement with an application user interface to the user before providing the user input to the application. A user interface (UI) may be displayed as appearing in a physical environment at a location away from a user input device and / or away from a display device displaying the user interface (e.g., in a three-dimensional XR display environment). For example, the physical movement of a mouse input device may be rendered as movement of a mouse cursor at a physical location (perhaps only a few inches) different from the physical mouse on the display. As another example, a gaze cursor may indicate the location of the user's gaze, which is away from one or more eye-facing cameras used to determine the gaze location. Similarly, gestures may be rendered as hand objects in a virtual space and presented to the user. Thus, a rendering effect of a user input may be rendered or presented to the user at a location physically separated from the physical location of the sensor that obtains the sensed user input. In a virtual space or other extended reality environment, user input may also appear at a location away from a display component that displays a representation of the application UI and / or the user's hand to make the user appear to be located away from the display component.

[0024] In addition, a user may move their hands and / or direct their gaze at or near a rendered application user interface without intending to interact with the application UI (e.g., due to normal movement of the user around the physical environment, such as during a conversation with another person not associated with the electronic device), and / or may perform gestures to interact with one displayed application UI without intending to provide those gestures to an application associated with another displayed application UI.

[0025] Aspects of the present disclosure include receiving, at a system process of an electronic device, from an application running on the electronic device, a definition of an effect for a first user interface (UI) element managed by the application. The system process may receive user input when the first user interface element is displayed by the electronic device without the effect applied to the first user interface element. In response to determining that the user input corresponds to the first user interface element displayed without the effect, the system process renders the effect on the first UI element according to the definition without providing the user input to the application.

[0026] Figure 1 An exemplary environment 100 for practicing various aspects of the present disclosure is illustrated. In the exemplary environment 100, a user 110 interacts with a computing device 120 having a display 115. As shown, the computing device 120 may include one or more speakers, such as a speaker 117 (e.g., a plurality of speakers operable to generate spatialized audio that is perceived by the user 110 as emanating from locations in the user's 110 physical environment that are separate from the location of the speaker 117). In one or more examples described herein, the user 110 may gaze at the computing device 120, and the device may sense the user's gaze location 114. In one or more implementations, the user's gaze location may constitute a user input to the device.

[0027] The computing device 120 may be a smart phone, a tablet device, or a wearable device (such as a smart watch or a head-mounted portable system) that includes a display system capable of presenting a visualization of an extended reality environment to the user 110. The computing device 120 may be powered by a battery and / or any other power source incorporated into the computing device 120 and / or coupled to the computing device 120 (e.g., via a cable). In one example, the display system of the computing device 120 provides a stereoscopic presentation of the extended reality environment to the user, thereby achieving a three-dimensional visual display of a rendering of a particular scene. In one or more specific implementations, as an alternative or in addition to using the computing device 120 to access the extended reality environment.

[0028] The computing device 120 may include one or more cameras (e.g., visible light cameras, infrared cameras, etc.). In addition, the computing device 120 may include various sensors that can detect user input, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near field communication sensors, etc.). In addition, the computing device 120 may include hardware elements that can receive user input, such as hardware buttons or switches. The user input detected by such sensors and / or hardware elements corresponds to various input modalities for interacting with virtual content displayed within a given extended reality environment. For example, such input modalities may include but are not limited to facial tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breathing, temperature, electroencephalogram, smell), recognition of voice or audio (e.g., specific hot words), and activation of buttons or switches, etc. The computing device 120 may also detect and / or classify physical objects in the physical environment of the computing device 120.

[0029] In one or more specific implementations, computing device 120 may be communicatively coupled to an infrastructure device. Generally, such an infrastructure device may include more computing resources and / or available power than computing device 120. In one example, computing device 120 may operate in various modes. For example, computing device 120 may operate in an independent mode independent of any infrastructure device.

[0030] Computing device 120 may also operate in a wirelessly connected mode (e.g., connected to a base device via a wireless connection) to work in conjunction with a given base device. Computing device 120 may also operate in a connected mode in which computing device 120 is physically connected to a base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., where the base device charges computing device 120 and / or provides power to computing device 120 when physically connected).

[0031] When the computing device 120 is operating in a wireless connection mode or a connected mode, at least a portion of processing user input and / or rendering an extended reality environment can be offloaded to the base device, thereby reducing the processing burden on the computing device 120. For example, in one specific implementation, the computing device 120 works in conjunction with the base device to generate an extended reality environment that includes physical objects and / or virtual objects that enable different forms of interaction (e.g., visual, auditory, and / or physical or tactile interaction) between the user and the extended reality environment in real time. In one example, the computing device 120 provides a rendering of a scene corresponding to the extended reality environment that can be perceived by the user and interacted with in real time. In addition, as part of presenting the rendered scene, the computing device 120 may provide sound and / or tactile or haptic feedback to the user. The content of a given rendered scene may depend on available processing power, network availability and capacity, available battery power, and current system workload.

[0032] Computing device 120 may also detect events that have occurred within the scene of the augmented reality environment. Examples of such events include detecting the presence of a creature (such as a person or pet, a specific person, entity, or object) in the scene.

[0033] Figure 2 An exemplary system 200 for providing out-of-process effects is illustrated. System 200 may be implemented, for example, on computing device 120. System 200 includes an application (app) process 202 and a system process 204, which corresponds to application (app) 260, and the system process includes a rendering system 270 (e.g., a rendering process) and an effects component 280 (e.g., an effects process). In some aspects, processes 202 and 204 may be separate processes running on a common device (such as computing device 120) or on separate devices. Various aspects of system process 204 (e.g., including rendering system 270, effects component 280, and / or hit test 282) may be implemented in hardware, software, or a combination thereof.

[0034] like Figure 2 As depicted, system process 204 includes both rendering system 270 and effects component 280. However, the present disclosure is not limited in this regard. For example, rendering system 270 may run in a separate process from effects component 280, rendering system 270 and effects component 280 may be implemented in a common process, or rendering system 270 and effects component 280 may be further distributed to run in more than two processes, all of which may be separate from application process 202. Similarly, in other aspects, application 260 may be distributed to run in more than one application process. Processes (such as application process 202 and system process 204) may be distributed to run in a common process (such as Figure 1 The computing device 120 or Fig.10260 ). In other aspects, the system process 204, the rendering system 270, and the effects component 280 may include one or more operating system drivers that do not run in a virtual memory space.

[0035] The operation of the system 200 may include receiving user input and outputting rendering effects as feedback of the user input, and this may be performed outside the application process 202 and without the knowledge of the application 260. The application 260 may provide a description of its UI element 250 to the rendering system 270, and the application 260 may provide its effect definition 252 to be rendered in response to future user input. The application 260 may also provide a definition of a control style for one or more parts of the UI of the application 260. When the effects component 280 receives a user input corresponding to the definition of the effect received from the application 260, the effects component 280 may cause the effect 256 to be rendered as user output by the rendering system 270. When the effects component 280 receives a user input at a particular location while displaying a UI window or UI element having a control style defined by the application 260, the effects component 280 may redirect the user input associated with another part of the UI to a UI window or UI element having the defined control style (e.g., a foreground style).

[0036] In an optional aspect of the system 200, the effects component 280 may learn the position of the UI element (e.g., the position of a button or scroll bar), for example, directly from the application 260 or via an optional UI element 258 message from the rendering system 270. In one or more specific implementations, the effects component 280 may then perform an optional hit test 282 between the current position of the UI element and the current position of the user input. In one or more other specific implementations, the hit test 282 may be performed by another system process that is separate from the effects component 280 and performs hit tests for a variety of different purposes (e.g., for the effects component 280 and other components and / or processes). When a specific user input is identified as being associated with a specific UI element by a hit test 282 between the user input and a specific UI element of the application 260 (e.g., by the effects component 280 or another system process of the computing device 120), the hit test 282 may identify a preliminary interaction with the specific UI element of the application 260. When the preliminary interaction is identified, the effect 256 may be rendered as a user output without notifying the application 260 of the user input or the identified interaction. Alternatively, when the replacement user input is identified as a confirmed interaction of the user with the UI element, effects component 280 or another system process may notify application 260 of the confirmed interaction as filtered user input message 254 .

[0037] Effect definition 252 can describe various effects that application indicates system process 204 renders on one or more UI elements. Each effect defined in effect definition 252 can correspond to a specific type of user input interaction with a specific UI element. In various aspects, effect definition 252 can describe multiple effects to be rendered in response to a single user input interaction with a single UI element. For example, effect definition 252 can indicate that when the user's gaze is within a specific range of a specific button UI element, the button will begin to glow to a specific brightness level and / or make a sound, and when the gaze is within a distance range closer to the button, the UI element can glow at a brighter level and also swing and / or generate a buzzing sound, and when the gaze is within the third closest range (possibly directly gazing at the button and and / or directly gazing at the button for at least a predefined dwell time, such as one second, half a second, one tenth of a second, or a smaller portion of a second), the user interaction can be identified as a confirmed interaction.

[0038] In an alternative optional aspect, application 260 may provide information describing UI elements of application 260 directly to effects component 280 (e.g., rather than effects component 280 receiving UI information from rendering system 270). Additionally, instead of location-based hit testing, hit testing 282 may more generally determine that a particular user input corresponds to an interaction with a particular user interface element. For example, a user's verbal audio input saying "red button" may cause hit testing 282 to associate the audio input with the red button UI element.

[0039] In one aspect, effect definition 252 can be a declarative definition. In this aspect, application 260 can provide all information necessary for software components (such as effect component 280) outside application process 202, so that the desired effect of the application is rendered without the application knowing or further participating. The declarative definition of the effect may include, for example, an identification of a UI element provided by a user interface framework or an operating system, an identification of a triggering user input, and an identification of the effect to be rendered when the triggering user input corresponds to the first UI element. In one or more specific implementations, UI elements may be provided from one or more files (such as a universal scene description (USDZ) file). In another specific implementation, the UI elements referenced in the declarative definition of the application may be provided by the operating system. In one or more specific implementations, effect definition 252 may include one or more definitions of control styles for one or more corresponding UI windows and / or UI elements. For example, the declarative definition of a control style may include an identifier of a UI window and / or UI element to which the control style is applied, an identifier of one or more triggering user inputs that may appear away from the UI window and / or UI element to which the control style is applied, and a redirection instruction for redirecting the user input to the UI window and / or UI element to which the control style is applied when the triggering user input is received. In one or more other specific implementations, the control style may be a control style flag or other value or parameter that indicates that any user input with other parts of the application UI should be redirected to the UI window and / or UI element to which the control style is applied whenever the UI window and / or UI element to which the control style is applied is displayed.

[0040] Figure 3A An exemplary method for rendering UI effects is illustrated. As shown in the figure, Figure 3A The method may include in system process 300 (eg, Figure 2 A definition of an effect for a UI element of an application is received at a specific implementation of system process 204 (e.g., a specific implementation of application process 204) (block 302). One or more definitions of one or more control styles may also be received at block 302. User input may be received by system process 300 (block 304), and when it is determined (e.g., by system process 300, such as by performing hit test 282) that the received user input corresponds to a UI element of the received effect definition (block 306), the effect may be rendered (e.g., by system process 300) on the UI element according to the effect definition (block 308). In one aspect, the effect may be executed in system process 300, which is separate from application process 350 (e.g., a specific implementation of application process 204) in which the application is running. Figure 3A operation.

[0041] Figure 3AThe method may include optional additional operations, such as the system process 300 receiving additional user input different from the user input received in block 304 (block 310). When the user's intent to interact with an application or a UI element of an application is identified by the system process 300 (e.g., by performing hit test 282) based on the original user input and / or the additional user input (block 312), some or all of the user input identifying the user's intent may be optionally provided from the system process 300 to the application process 350 (or otherwise made available to the application process) (block 314). In one or more use cases, the user input may be provided to the application process 350 in conjunction with a UI element of an application for which the user's intent to interact with the application or UI element of the application is identified. In one or more other use cases, a foreground UI window or foreground UI element may be displayed, foreground control style definitions may be provided to the system process 300 for the foreground UI window or the foreground UI element (e.g., at box 302), and some or all of the user input identified as being intended for user intent may be provided from the system process 300 to the application process 350 (or otherwise made available to the application process) in conjunction with the foreground UI window or the foreground UI element (e.g., even if the user input is identified as intended to interact with another UI element or UI window) (box 314).

[0042] Run through Figure 3A and Figure 3B The referenced UI element may be part of the user interface of the application. For example, the UI element may be defined by the application, and the purpose of the UI element may be to enable the user to interact with the application (after identifying the user's intention to interact with the application). In one aspect, the UI element may include one or more attributes, such as the position, size, orientation, color, transparency or brightness (as an example), and control style of the UI element. In one or more specific implementations, the effect definition may include a modification of one or more attributes (such as the position, size, orientation, color, transparency or brightness (as an example) of the UI element). For example, the effect definition of a button UI element may include a lifting effect that raises the position attribute of the button and a brightening effect that increases the brightness attribute of the button. In addition, the application UI element may include a predefined "remote state", where the remote state corresponds to a predefined value of the attribute of the UI element. The remote state of the UI element of the application may be considered remote, because although the application may define the UI element, its attributes, and its remote state, only system components in a separate system process (such as the rendering system 270 and the effects component 280) can manage or understand the remote state of the UI element. This may allow a user's initial interaction with a UI element to be rendered while maintaining privacy to the application by preventing the application from discovering which remote state the UI element is in at any particular time.

[0043] In one aspect, the runtime can also be obtained directly from the application or from an alternative source (such as the UI framework and file formats described above) or from the operating system. Figure 3A and Figure 3B These UI elements provided by alternative sources can serve as the user interface of the application (such as Figure 2 The UI elements from the alternate source may have associated properties and "remote states" defined by the alternate source and / or by the application using it. For example, even though the UI element may originate from an alternate source, such UI element may still be managed by the application using it to cause the system process to render the remote effect while maintaining privacy to the managing application.

[0044] In one aspect, the UI element may be specified as a layer tree or a render tree. For example, application 260 may provide the layer tree or render tree of the UI element to rendering system 270, and application 260 may provide effect definition 252 including the layer tree or render tree of the UI element to an operating system or other component that manages out-of-process UI effects.

[0045] The effect definition (box 302) can be received from a specific application running in the application process 350, and can describe the effect to be rendered on the UI element belonging to the specific application. However, the specific implementation of the present disclosure is not limited to this. For example, the effect definition received in box 302 can be received indirectly from other sources, the effect definition can be a predetermined effect, or the UI element can be a predetermined UI element. In one example, a button UI element can be predefined, and the "lift" effect can be predefined to occur when a gaze or hovering user input is determined to correspond to a predefined button effect. In the case of predefined effects and / or predefined UI elements, the received effect definition may include a reference to a predefined element without fully defining such an element.

[0046] In one aspect, the effect definition may identify the effect to be rendered for the UI element by identifying the remote state (with predefined rendering property values) to be used when the user input is determined to correspond to the UI element. In another aspect, the effect definition may identify the animation to be used when transitioning between predefined remote states.

[0047] User input (such as received in blocks 304 or 310 of FIG. 3 ) may include many types of input from a user. For example, in addition to user input received from a user input device (such as a mouse, keyboard, and / or audio input device), received user input may also include position and / or orientation tracking of a user's body or body part (such as a fingertip or hand), gesture recognition (such as hand shape, hand movement, and / or facial expression), or the location of the user's optical gaze (e.g., gaze location 114). User input may include time or velocity measurements, such as a gaze dwell duration that measures the duration that the user's gaze remains on a particular location or UI element or remains within a particular distance of a particular location or UI element, or a hover dwell that measures the duration that a user's hand or other body part remains within a particular two-dimensional or three-dimensional distance of a particular location or UI element.

[0048] On the one hand, UI effects (such as defined in box 302 and rendered in box 308 of Figure 3) may include any rendered output that a user can sense, including audio, images, and / or tactile effects. Examples of audio effects on UI elements may include audible sounds designed to be perceived as emanating from the position of the UI element, or distortion of audio that has been emitted from the UI element before rendering the audio effect. As an example of an image effect, the UI element may be displayed to the user or otherwise visually presented before rendering the effect, and the rendering effect may change the appearance or position of the UI element. For example, a luminous effect may make an existing UI element glow. A highlight effect may add a new visual cue, such as a circle or a check mark, near the position of the UI element. On the one hand, an effect definition may specify a "reverse" rendering effect, wherein the specified effect is generally rendered unless the user input corresponds to the UI element.

[0049] On the one hand, identifying the correspondence between the user input and the first UI element (box 306) may include identifying multiple user interface elements with which the UI element may intersect, and then determining that the multiple identified UI elements include the first UI element. For example, the user input may be a gaze position. The system process may perform a hit test process (e.g., hit test 282), which includes: such as by identifying all UI elements from all applications within a threshold distance of the gaze position and / or identifying all UI elements with which the gaze direction intersects to identify a list of potential UI elements that the user may be trying to interact with. If the effect definition includes an application UI element in the list, the corresponding effect may be rendered on the application UI element. In a use case where the gaze direction intersects with multiple UI elements, each of which has an effect definition, the system process may render the effect on the following UI elements: a UI element that is displayed as appearing closest to the user among the multiple UI elements and / or a UI element that the gaze position intersects with most centrally among the multiple UI elements.

[0050] In one aspect, a single effect may be defined for a group of UI elements. For example, when determining (306) user input correspondence for any of the UI elements in the group, the defined effect may be rendered on all UI elements in the group. In another example, the user input correspondence in box 306 may be determined only by a subset of the UI elements in the group. The definition of an effect for a group of UI elements may include an indication of which subset of the UI elements in the group are to be hit tested. If the user input position (such as the position of the user's gaze or gesture) is within a specific proximity range (e.g., a predefined distance defined by a system process or in an effect definition) of any element in the indicated subset of UI elements, the effect may be rendered on all UI elements in the group (e.g., unless a foreground UI window or foreground UI element with a foreground control style has been defined and displayed).

[0051] In one aspect, a single effect definition may define a set of rendering effects. When determining the correspondence with a UI element (block 306), the UI element may be rendered using a plurality of different effects (block 308). For example, an effect definition for a button may include a combination of a lift effect (e.g., movement of the button in a direction opposite to pressing the button), a glow effect (e.g., brightening), and an audio cue (e.g., an audio tone emitted from the button).

[0052] In one aspect, an effect definition may combine multiple aspects described above. For example, a single effect definition may include multiple UI elements, multiple effects to be rendered, and / or multiple types of user input that may trigger the rendering of one or more effects.

[0053] In one aspect, even after identifying the user's intent to interact with the application (block 312), only a summary or subset of the user input may be provided to the application (block 314). For example, once the user's gaze dwell time on an application button is greater than a threshold, the intent to interact with the application may be identified. However, instead of providing the user's gaze location or gaze dwell time to the application, an indication that the user intends to press the application button may be provided to the application without providing the application with any knowledge of the user's gaze.

[0054] A list selection UI may be used to illustrate a more complex example of an out-of-process effect that combines the various aspects described above. A common list selection UI task may include presenting a list of several options to a user, including an indication of the current selection. Examples of list selection UIs include a "combo box" in MacOS or a "drop-down list box" in Windows. Each of the options in the list may be rendered, for example, as a visual box including text describing the option, and the list may be presented as a row of text boxes, none of which are highlighted except for the current selection, which is highlighted with a first color.

[0055] In an exemplary out-of-process implementation of a list selection UI, application 260 may provide a definition of a UI element 250 including a row of text boxes to rendering system 270. The definition of each text box may include three predefined remote states, including: 1) an unselected (idle) remote state with a highlight attribute set to a first color (or no highlight); 2) a currently selected remote state with a highlight attribute set to a second color; 3) a preliminary selected remote state with a highlight attribute set to a third color. Application 260 may also provide an indication that the first text box of the list is currently selected, and application 260 may provide effect definition 252 indicating how preliminary user input should affect the remote state of the text box to effect component 280. Rendering system 270 may render a list of text boxes, each of which has its corresponding remote state, and rendering system 270 may provide the definition of the text boxes to effect component 280 via UI element 258 messages along with the current position of each text box. Effect component 280 may use these definitions to manage the remote states of the text boxes. When effects component 280 receives user input including a location (e.g., a gaze location or a location of a gesture), effects component 280 may perform a hit test of the user input location for the location of the text box to determine the correspondence between the user input and one of the rendered text boxes. When the user input location corresponds to a second text box in the list that is different from the currently selected first text box, effects component 280 may cause rendering system 270 to render effect 256 including a change in a highlight attribute from an unselected remote state to a preliminarily selected remote state without notifying the application of the change in the user input location or remote state and highlight attribute of the second text box.

[0056] As an alternative or in addition to the above preliminary selection, when the user input indicates the user's final selection, the application may be notified of the user's intention to change the current selection in the list. For example, after rendering the above preliminary selection and in response to receiving additional user input (block 310) identifying the user's intention to change the current selection to the second box (e.g., when a gaze dwell duration threshold is exceeded on the second box or a button press gesture occurs on the second box), the application may be notified of the user's intention to change the current selection in the list.

[0057] In one aspect, the remote effect definition may specify an animation to be rendered when transitioning between remote states. For example, in the list selection UI example above, the remote state transition animation may be defined from the unselected remote state to the preliminarily selected remote state to include a gradient transition from a first highlight color (no highlight for the idle state) to a third highlight color (for the preliminarily selected state).

[0058] Figure 3B An exemplary method for rendering UI effects according to various aspects of the subject disclosure is illustrated. As shown in the figure, Figure 3BThe method may include identifying, by the application process 350, an application-managed rendering effect and a remote rendering effect corresponding to the UI element (block 552). The definition of the identified remote rendering effect may be provided by the application process 350 (block 554) to a remote effect rendering service provider (such as Figure 2 270 and / or effects component 280). For UI elements with corresponding application management effects, user input may be received by application process 350 (box 556). Application management UI effects may be rendered by application process 350 on their corresponding UI elements (box 558). In one or more specific implementations, application management UI effects and / or one or more application functions that may be performed by the application in response to user input may be performed by application process 350 based on which parts of the application's UI (e.g., which UI element) the user input interacts with. In one or more specific implementations, when a foreground control style is defined by application process 350 and provided (554) to system process 300, the user input received at box 556 may be received in conjunction with a UI window or UI element that is different from the UI window or UI element in which system process 300 detected the user input (e.g., a foreground UI window with the defined foreground control style).

[0059] In one aspect, the application process 350 may be executed in a separate application process 350 from the system process 300 that manages remote / out-of-process UI effects. Figure 3B In another aspect, Figure 3A The method of may be an exemplary method for providing a remote effect rendering service, and the effect definition provided in block 554 may be received at block 302. In yet another aspect, the additional user input provided in block 314 may be received at block 556. In one or more specific implementations, the effect definition provided in block 554 and received at block 556 may include a control style for a portion of a UI, such as a specific UI window (e.g., a foreground window) for an application UI.

[0060] In other aspects, the effect of the preliminary user interaction may be identified as a remote UI effect, while the effect of the confirmed user input may be identified as an application management UI effect (box 552). In one or more specific implementations, the application management UI effect based on the confirmed user input may be managed by the application based on the location of the user input (e.g., as provided in box 314). In one or more specific implementations described in further detail below, the system process 300 may provide redirected user input received at one location on the application UI to an application associated with a different location on the application UI (e.g., on a foreground UI window that has provided a foreground control style). In the above list selection UI example, the preliminary selection rendering effect may be identified as a remote UI effect, while the final selection rendering effect may be identified as an application management UI effect.

[0061] Figure 4 An exemplary use case is illustrated in which one or more UI windows 402 of an application (such as application 260) are displayed in a visible display area 400 of a computing device 120 (e.g., a visible area of ​​a display 115 of the computing device 120). For example, the display 115 of the computing device 120 (e.g., see Figure 1 ) is operable to display one or more user interface windows 402 corresponding to underlying applications (e.g., application 260) running at computing device 120 in a visible display area 400 of display 115. As shown, computing device 120 may also display a UI window 422 of another underlying application running at computing device 120.

[0062] As shown, user interface window 402 of application 260 may include one or more user interface elements 406. As shown, user interface window 422 may also include one or more user interface elements 426. As examples, user interface element 406 and / or user interface element 426 may include virtual buttons, virtual switches, virtual lists (e.g., drop-down lists), tabs, scroll bars, application icons, and / or other interactive elements. Figure 4 As shown, in one or more specific implementations, the user interface element 406 of the application 260 can be displayed separately from the user interface window 402. For example, in Figure 4 , user input is received at a location 408 corresponding to a user interface element 406 within one of UI windows 402 of application 260 .

[0063] As described herein, when user input to computing device 120 is received at location 408 of UI element 406 (e.g., when user's gaze location 114 falls within or within the boundaries of UI element 406 and / or when a user's hand or finger hovers over a location within or within the boundaries of UI element 406, such as for a predetermined amount of time), system process 300 may render one or more effects for user interface element 406. As described herein, these effects may be out-of-process effects predefined by an underlying application of UI element 406, which are rendered without providing user information (e.g., the location of the user input) to the application.

[0064] In one or more specific implementations, user input to user interface element 406 and / or another interaction with the application UI can cause the foreground window of the application UI to be presented. Figure 5An exemplary use case is illustrated in which a foreground window 402F of an application 260 is displayed in a visible display area 400 (e.g., along with other portions of an application UI, such as other UI windows 402 and / or UI elements 406 managed by the same application 260). For example, the foreground window 402F may have been opened in response to a request to Figure 4 408). As illustrative examples, foreground window 402F may include a share sheet for sharing (e.g., sending) application-related content to a remote device, a toolbar or palette for editing text displayed by application 260, a purchase interface for making in-app purchases via application 260, or any other pop-up or pop-up window that may be provided by the application. When foreground window 402F is displayed, foreground window 402F may have foreground functionality relative to other UI windows 402 and UI elements 406 of application 260. For example, foreground window 402F may have foreground functionality defined by a foreground control style that specifies or otherwise indicates priority for receiving user input over other UI windows 402 and UI elements 406 of application 260.

[0065] like Figure 5 As indicated, a hit test (e.g., hit test 282) performed by system process 300 may determine that location 508 of the user input is a location corresponding to foreground user interface element 406F of foreground window 402F. For example, the user input at location 508 may be a preliminary user input, such as a gaze or hover of a hand or input device (e.g., for less than a predetermined amount of time, such as a dwell time). In this example, the user input at location 508 may cause system process 300 (e.g., without providing any information about the user input or location 508 to the application) to provide an out-of-process or remote effect (e.g., a visual effect or an audio effect) on foreground UI element 406F, as described herein. In another example, the user input at location 508 may be a confirmed user input (e.g., including a detected pinch or click gesture, a pinch or click gesture combined with a gaze, or a gaze or hover for at least a predetermined amount of time (such as a dwell time)), and may cause system process 300 to provide an indication of the user input to application 260 at foreground UI element 406F within foreground window 402F. In this other example, application 260 can determine how to handle the user input (e.g., by closing the foreground window and / or performing one or more application operations corresponding to foreground UI element 406F, such as initiating playback of a song or other audio content, changing the appearance of text displayed by the application, making a purchase, or performing any other application functionality).

[0066] exist Figure 5In the example of , foreground window 402F is displayed, and user input is received at location 508 within foreground window 402F. However, in other use cases, user input may be received at locations other than within foreground window 402F when foreground window 402F is displayed. However, in some use cases and / or systems, it may be challenging to avoid providing user input to other UI windows 402 and / or UI elements 406 of application 260 when foreground window 402F is displayed.

[0067] For example, in a smartphone, tablet, laptop, or desktop computer implementation of computing device 120 where the visible display area is limited to the physical area of ​​the display itself, the computing device may generate an invisible display layer over the entire visible display area between foreground window 402F and all other displayed content, the invisible display layer being configured to receive any user input outside of foreground window 402F. However, in systems with large displays and / or systems that display a three-dimensional environment in which the displayed content appears to the user to be at a location far from the display itself, displaying such a virtual layer over the entire visible display area may be undesirable, inefficient, or even ineffective.

[0068] For example, in Figure 5 In the example of , foreground window 402F is at least partially displayed in front of one of UI windows 402. Figure 5 In the example of , the foreground window 402F is spatially separated from the other UI window 402. Figure 5 In the example of , the foreground window 402F is spatially separated from the independent UI element 406 of the application 260. Figure 5 In the example of , foreground window 402F is spatially separated from UI window 422 of another application. With these and other arrangements of UI windows and / or elements, it may be challenging to determine how to handle user input at locations other than within foreground window 402F when foreground window 402F is displayed. For example, this may be particularly challenging in a use case where UI window 402, UI element 406, foreground window 402F, and / or foreground UI element 406F are displayed so as to appear in three dimensions at a corresponding location that is remote from computing device 120 (e.g., remote from display 115 of computing device 120).

[0069] For example, Figure 6Illustrated is how, in one or more use cases, a foreground UI element 406F may be displayed as appearing at least partially in front of one UI window 402 and spatially separated from another UI window 402, and (in some examples) the other UI window 402 may still appear to be closer to the user, computing device 120, and / or display 115 than foreground window 402F, even when foreground window 402F has foreground functionality relative to all other UI windows 402 and / or UI elements 406 of an application 260.

[0070] Various aspects of the subject technology provide out-of-process hit testing that can determine how to handle user input (e.g., including preliminary user input and / or confirmed user input as described herein) when foreground window 402F is displayed with one or more other UI windows and / or UI elements 406 of an underlying application (such as application 260).

[0071] For example, Figure 7 A use case is illustrated in which, while a foreground window 402F is displayed, a user input is received at a location 708 corresponding to another UI window 402 of the same application (e.g., as determined by the system process 300 based on hit testing 282, as described herein). In this example, the system process 300 can detect the user input at the location 708. The system process 300 can then redirect the user input to the foreground window 402F (e.g., by providing an indication of the user input to the application 260 in conjunction with the foreground window 402F), rather than providing a remote effect on the UI element 406 at the location 708 (e.g., if the user input is preliminary user input) or rather than providing an indication of the user input to the application 260 below the UI window 402 in conjunction with the UI element 406 at the location 708 (e.g., if the user input is confirmed user input).

[0072] In the use case where user input received while foreground window 402F is displayed (e.g., as determined by system process 300) is preliminary user input, system process 300 may avoid providing (e.g., may not provide) a previously defined remote effect on UI element 406 that received the user input.

[0073] In the use case where the user input received when the foreground window 402F is displayed (e.g., as determined by the system process 300) is a confirmed user input, the application that receives the redirection indication of the user input in conjunction with the foreground window 402F can then determine whether to take any action and / or what action to take in response to the user input. For example, the application 260 can close the foreground window 402F in response to the user input at the position 708, or the application 260 can take no action with respect to the foreground window 402F. To provide this foreground functionality for the foreground window 402F, the application 260 can provide the system process 300 with a definition of a control style (e.g., a foreground control style) that defines the foreground functionality (e.g., in the remote definition of block 554).

[0074] Figure 8 Another exemplary use case of receiving user input (comment declaration language) at a location separate from any of the UI windows and / or elements of application 260 is illustrated. For example, the user may be looking at or gesturing at a location in a free (e.g., empty) area between various application UIs and / or elements. In this exemplary use case, regardless of whether the user input at location 808 is determined (e.g., by system process 300) as preliminary user input or confirmed as confirmed user input, system process 300 can prevent any information about the user input from being provided to application 260.

[0075] Fig. 9 An exemplary process 900 for providing out-of-process hit testing according to one or more specific implementations is illustrated. For the purpose of explanation, this document mainly refers to Figure 1 The process 900 is described with reference to the computing device 120 of FIG. However, the process 900 is not limited to Figure 1 The computing device 120 of the present invention may be used, and one or more blocks (or operations) of the process 900 may be performed by one or more other components of other suitable devices. Further for the purpose of explanation, some blocks of the process 900 are described herein as occurring sequentially or linearly. However, multiple blocks of the process 900 may occur in parallel. In addition, the blocks of the process 900 do not have to be executed in the order shown, and / or one or more blocks of the process 900 do not have to be executed and / or may be replaced by other operations.

[0076] exist Fig. 9In an example, at box 902, an electronic device (e.g., computing device 120) may receive a definition of a control style (e.g., a foreground control style) of a first user interface window (e.g., a foreground UI window, such as foreground UI window 402F) of a user interface (UI) managed by the application (e.g., application 260) from a system process (e.g., system process 300) of the electronic device from an application (e.g., application 260) running on the electronic device. In one or more specific implementations, the definition of the control style may be a declarative definition that includes an identification of the first UI window (and / or its UI element) to which the control style is applied, an identification of the user input, and a redirection instruction for redirecting the user input to the first UI window (and / or its UI element) to which the control style is applied when the user input is received (e.g., at the location of other UI windows and / or elements of the same application). In one or more other specific implementations, the control style may be a control style flag or other value or parameter that indicates to the system process that whenever the first UI window (and / or its UI element) is displayed, user input associated with other parts of the application UI of the application should be redirected to the first UI window (and / or its UI element) to which the control style is applied.

[0077] At box 904, the system process can receive user input at a location (e.g., location 708) associated with the second UI window when the first user interface window and the second user interface window (e.g., UI window 402 or UI element 406) managed by the application are displayed by the electronic device. For example, the location associated with the second UI window can be a location that has been determined by the system process (e.g., using a hit test operation, such as hit test 282 described herein) to intersect the second UI window and / or its UI elements.

[0078] In one or more specific implementations, the first user interface window may be displayed as appearing at least partially in front of the second user interface window. In one or more specific implementations, the first user interface window may be spatially separated from the second user interface window. In one or more specific implementations, the first user interface window and the second user interface window may be displayed as appearing at a first corresponding position and a second corresponding position away from the electronic device, and the second corresponding position of the second user interface window may be closer to the electronic device than the first corresponding position of the first user interface window (for example, as in Figure 6 ).

[0079] At box 906, the system process may redirect the user input at the location associated with the second UI window to the first UI window based on the definition of the control style. For example, redirecting the user input at the location associated with the second UI window to the first UI window may include: the system process provides an indication of the user input associated with a different location (e.g., different from the location where the user input was received) to the application, and the different location corresponds to the first UI window. As another example, redirecting the user input at the location associated with the second UI window to the first UI window may include: the system process provides an indication of the user input and an indication for processing the user input associated with the first UI window to the application (e.g., without providing the location associated with the second UI window to the application).

[0080] In one or more specific implementations, process 900 may also include (for example, before or after receiving the user input that is redirected to the first UI window, such as before the first UI window is opened or after the first UI window is closed): when the second UI window is displayed instead of the first UI window, receiving another user input at the location associated with the second UI window (for example, the same location 708); and providing the other user input to the second UI window (for example, by providing an indication of receiving the other user input at the UI element at the location 708 to the application 260, such as without providing the location 708 to the application).

[0081] In one or more specific implementations, process 900 may also include (for example, before or after receiving the user input that is redirected to the first UI window, such as before the first UI window is opened or after the first UI window is closed): when the second UI window is displayed instead of the first UI window, receiving another user input (for example, preliminary user input, such as user gaze) at the location associated with the second UI window (for example, the same location 708); and when the second UI window is displayed instead of the first UI window, the system process generates a visual or audio effect (for example, a remote effect as described herein) for an element of the second UI window (for example, UI element 406 that coincides with the location 708) in response to the another user input at the location associated with the second UI window, the visual or audio effect being based on another definition previously provided to the system process by the application (for example, at box 554). In one or more specific implementations, when the first user interface window and the second user interface window managed by the application are displayed by the electronic device, the system process may also avoid providing the visual or audio effect for the element of the second UI window in response to the user input received at the location associated with the second UI window.

[0082] In one or more specific implementations, the electronic device (e.g., application 260) may close the first user interface window in response to the first user interface window receiving the redirected user input. In one or more specific implementations, the electronic device (e.g., application 260) may ignore the user input in response to the first user interface window receiving the redirected user input (e.g., application 260 may not take action in response to the user input).

[0083] In one or more specific implementations, the system process may receive another user input at another location (e.g., location 808) that is not associated with the first user interface window or the second user interface window when the first user interface window and the second user interface window managed by the application are displayed by the electronic device; and avoid providing the other user input to the application (e.g., as described herein in conjunction with Figure 8 described).

[0084] As described above, one aspect of the technology of the present invention is to collect and use data available from specific and legitimate sources to provide out-of-process hit testing for electronic devices. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include audio data, voice data, demographic data, location-based data, online identifiers, phone numbers, email addresses, home addresses, encrypted information, data or records related to the user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

[0085] The present disclosure recognizes that the use of personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to provide out-of-process hit testing for electronic devices.

[0086] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities will be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding the requirements of the industry or government that maintain user privacy. Such information about the use of personal data should be prominently and easily accessible to users, and should be updated as the collection and / or use of data changes. The user's personal information should be collected only for legitimate use. In addition, such collection / sharing should only occur after receiving user consent or other legal basis specified in applicable law. In addition, such entities should consider taking any necessary steps to defend and safeguard access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate that they comply with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted for specific types of personal information data collected and / or accessed, and applied to applicable laws and standards, including specific considerations that are exclusive to jurisdictions that can be used to impose higher standards. For example, in the United States, collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0087] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, in an example of providing an out-of-process hit test for an electronic device, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" to participate in the collection and / or sharing of personal information data during registration for a service or at any time thereafter. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0088] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than at an address level or at a level that is insufficient for facial recognition), controlling how data is stored (e.g., aggregating data across users), and / or other methods such as differential privacy, when appropriate.

[0089] Therefore, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or part of such personal information data.

[0090] Fig.10 An exemplary computing device 1000 that can be used to implement various aspects of the subject technology according to one or more specific implementations is illustrated. The computing device 1000 can be any computing device or server used to generate the above-mentioned features and processes and / or can be a part of any computing device or server used to generate the above-mentioned features and processes, including but not limited to laptop computers, smart phones, tablet devices, wearable devices such as goggles or glasses, etc. The computing device 1000 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The computing device 1000 includes a permanent storage device 1002, a system memory 1004 (and / or buffer), an input device interface 1006, an output device interface 1008, a bus 1010, a ROM 1012, one or more processing units 1014, one or more network interfaces 1016, and / or their subsets and variations.

[0091] The bus 1010 generally represents all system, peripheral, and chipset buses that communicatively connect the many internal devices of the computing device 1000. In one or more implementations, the bus 1010 communicatively connects the one or more processing units 1014 with the ROM 1012, the system memory 1004, and the permanent storage device 1002. The one or more processing units 1014 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in the present subject matter. In different implementations, the one or more processing units 1014 can be a single processor or a multi-core processor.

[0092] ROM 1012 stores static data and instructions needed by one or more processing units 1014 and other modules of computing device 1000. On the other hand, permanent storage device 1002 can be a read-write memory device. Permanent storage device 1002 can be a non-volatile memory unit that stores instructions and data even when computing device 1000 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) can be used as permanent storage device 1002.

[0093] In one or more implementations, a removable storage device (such as a floppy disk, a flash drive, and its corresponding disk drive) may be used as the permanent storage device 1002. Like the permanent storage device 1002, the system memory 1004 may be a read-write memory device. However, unlike the permanent storage device 1002, the system memory 1004 may be a volatile read-write memory, such as a random access memory. The system memory 1004 may store any of the instructions and data that one or more processing units 1014 may need at runtime. In one or more implementations, the processes disclosed in the subject matter are stored in the system memory 1004, the permanent storage device 1002, and / or the ROM 1012. The one or more processing units 1014 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more implementations.

[0094] The bus 1010 is also connected to an input device interface 1006 and an output device interface 1008. The input device interface 1006 enables a user to transmit information and select commands to the computing device 1000. Input devices that can be used with the input device interface 1006 can include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 1008 can, for example, enable the display of images generated by the computing device 1000. Output devices that can be used with the output device interface 1008 can include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information.

[0095] One or more implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form, including acoustic, voice, or tactile input.

[0096] Finally, if Fig.10As shown, bus 1010 also couples computing device 1000 to one or more networks and / or to one or more network nodes via one or more network interfaces 1016. In this manner, computing device 1000 may be part of a computer network, such as a LAN, a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet. Any or all components of computing device 1000 may be used with the subject disclosure.

[0097] Implementations within the scope of the present disclosure may be implemented in part or in whole using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) programmed with one or more instructions. Tangible computer-readable storage media may also be non-transitory in nature.

[0098] Computer-readable storage media can be any storage media that can be read, written, or otherwise accessed by a general or special computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0099] In addition, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more specific implementations, the tangible computer-readable storage medium may be directly coupled to the computing device, while in other specific implementations, the tangible computer-readable storage medium may be indirectly coupled to the computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0100] Instructions can be directly executable, or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or can be implemented as high-level language instructions that can be compiled to produce executable or non-executable machine code. In addition, instructions can also be implemented as data, or can include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those skilled in the art, the details including but not limited to the number, structure, sequence and organization of instructions can be significantly different without changing the underlying logic, function, processing and output.

[0101] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0102] It is understood that the use of personally identifiable information should be subject to privacy policies and practices that are generally recognized to meet or exceed industry or government requirements for maintaining user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of the authorized use should be clearly stated to users.

[0103] Those skilled in the art will recognize that the various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways) without departing from the scope of the subject technology.

[0104] It should be understood that the specific order or the hierarchical structure of the frame in the process disclosed by the present invention are the illustrations of exemplary methods. Based on the design preferred requirements, it should be understood that the specific order or the hierarchical structure of the frame in the process can be rearranged or all the illustrated frames are executed. Any frame in these frames can be executed simultaneously. In one or more specific implementations, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the specific implementation described above should not be understood as requiring such division in all specific implementations, and it should be understood that the program component (e.g., computer program product) and the system can be generally integrated together in a single software product or encapsulated in multiple software products.

[0105] As used in this specification and any claims of this patent application, the terms "base station", "receiver", "computer", "server", "processor" and "memory" all refer to electronic devices or other technical devices. These terms exclude people or groups of people. For the purpose of this specification, the term "display" or "displaying" means displaying on an electronic device.

[0106] As used herein, the phrase "at least one of" following a list of items, any of which is separated by the terms "and" or "or", modifies the list as a whole, rather than each member of the list (i.e., each item). The phrase "at least one of" does not require selection of at least one of each item listed; rather, the phrase allows for a meaning that includes at least one of any one item and / or at least one of any combination of items and / or at least one of each item. For example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" each refers to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0107] The predicate words "configured to," "operable to," and "programmed to" do not imply any particular tangible or intangible modification of a subject matter but are intended to be used interchangeably. In one or more specific implementations, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Likewise, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0108] Phrases such as aspect, this aspect, on the other hand, some aspects, one or more aspects, a specific implementation, this specific implementation, another specific implementation, some specific implementations, one or more specific implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, configuration, this configuration, other configurations, some configurations, one or more configurations, subject technology, disclosure, the present disclosure, other variations thereof, etc. are for convenience and do not mean that the disclosure involving such one or more phrases is essential to the subject technology, nor does it mean that such disclosure applies to all configurations of the subject technology. Disclosures involving such one or more phrases may apply to all configurations or one or more configurations. Disclosures involving such one or more phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to the other aforementioned phrases.

[0109] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive, in a manner similar to the way the term "comprising" is interpreted when used as a transitional word in a claim.

[0110] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be made available to the public regardless of whether the disclosure is explicitly stated in the claims. No claim element shall be interpreted under the provisions of 35 U.S.C. §112(f) unless the element is explicitly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “step for…”

[0111] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications of these aspects are apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein reference to elements in singular values ​​is not intended to mean "only one", but refers to "one or more", unless specifically noted. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used only for convenience and do not limit the subject disclosure.

Claims

1. A method, comprising: receiving, at a system process of an electronic device from an application running on the electronic device, a definition of a control style for a first user interface window of a user interface (UI) managed by the application; The system process receives user input at a location associated with the second UI window when the first UI window and the second UI window managed by the application are displayed by the electronic device; as well as The system process redirects the user input at the location associated with the second UI window to the first UI window based on the definition of the control style.

2. The method according to claim 1, further comprising: receiving another user input at the location associated with the second UI window while the second UI window is displayed and the first UI window is not displayed; as well as The other user input is provided to the second UI window.

3. The method according to claim 1, further comprising: receiving another user input at the location associated with the second UI window while the second UI window is displayed and the first UI window is not displayed; as well as When the second UI window is displayed instead of the first UI window, the system process generates a visual or audio effect for an element of the second UI window in response to the other user input at the location associated with the second UI window, the visual or audio effect being based on another definition previously provided to the system process by the application.

4. The method according to claim 3, further comprising: When the first user interface window and the second user interface window managed by the application are displayed by the electronic device, in response to the user input received at the location associated with the second UI window, avoiding providing the visual or audio effect for the elements of the second UI window. The method of claim 1 , wherein the first user interface window is displayed to appear at least partially in front of the second user interface window. The method of claim 1 , wherein the first user interface window is spatially separated from the second user interface window.

7. A method according to claim 1, wherein the first user interface window and the second user interface window are displayed as appearing at a first corresponding position and a second corresponding position away from the electronic device, and wherein the second corresponding position of the second user interface window is closer to the electronic device than the first corresponding position of the first user interface window.

8. The method according to claim 1, further comprising: In response to the first user interface window receiving the redirected user input, the first user interface window is closed.

9. The method according to claim 1, further comprising: In response to the first user interface window receiving the redirected user input, ignoring the user input.

10. The method according to claim 1, further comprising: receiving, by the system process, another user input at another location not associated with the first user interface window or the second user interface window while the first user interface window and the second user interface window managed by the application are displayed by the electronic device; as well as Providing the other user input to the application is avoided.

11. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving, at a system process of an electronic device from an application running on the electronic device, a definition of a control style for a first user interface window of a user interface (UI) managed by the application; The system process receives user input at a location associated with the second UI window when the first UI window and the second UI window managed by the application are displayed by the electronic device; as well as The system process redirects the user input at the location associated with the second UI window to the first UI window based on the definition of the control style.

12. The non-transitory computer readable medium of claim 11, the operations further comprising: receiving another user input at the location associated with the second UI window while the second UI window is displayed and the first UI window is not displayed; as well as The user input is provided to the second UI window.

13. The non-transitory computer readable medium of claim 11, the operations further comprising: receiving another user input at the location associated with the second UI window while the second UI window is displayed and the first UI window is not displayed; as well as When the second UI window is displayed instead of the first UI window, the system process generates a visual or audio effect for an element of the second UI window in response to the user input at the location associated with the second UI window, the visual or audio effect being based on another definition previously provided to the system process by the application.

14. The non-transitory computer readable medium of claim 13, the operations further comprising: When the first user interface window and the second user interface window managed by the application are displayed by the electronic device, in response to the user input received at the location associated with the second UI window, avoiding providing the visual or audio effect for the elements of the second UI window. 15 . The non-transitory computer-readable medium of claim 11 , wherein the first user interface window is displayed to appear at least partially in front of the second user interface window. 16 . The non-transitory computer-readable medium of claim 11 , wherein the first user interface window is spatially separated from the second user interface window.

17. A non-transitory computer-readable medium according to claim 11, wherein the first user interface window and the second user interface window are displayed as appearing at a first corresponding position and a second corresponding position away from the electronic device, and wherein the second corresponding position of the second user interface window is closer to the electronic device than the first corresponding position of the first user interface window.

18. The non-transitory computer readable medium of claim 11, the operations further comprising: In response to the first user interface window receiving the redirected user input, the first user interface window is closed or the user input is ignored.

19. An electronic device, comprising: Memory; and One or more processors configured to: receiving, at a system process of the electronic device, from an application running on the electronic device, a definition of a control style for a first user interface window of a user interface (UI) managed by the application; The system process receives user input at a location associated with the second UI window when the first UI window and the second UI window managed by the application are displayed by the electronic device; as well as The system process redirects the user input at the location associated with the second UI window to the first UI window based on the definition of the control style.

20. The electronic device of claim 19, wherein redirecting the user input at the location associated with the second UI window to the first UI window comprises: The system process provides an indication of the user input associated with a different location, the different location corresponding to the first UI window.

Images