Out-of-process pointer styles and resizing effects for electronic devices

Abstract

Out-of-process hit testing can provide privacy in a computer user input system. In an aspect, an application specifies a pointer pattern for a pointer element according to a position of the pointer element relative to a bounded region of a UI window. In another aspect, an application specifies a resizing effect for a UI element that, when triggered by an out-of-process hit test, causes the UI element to change from a first state to a second state.

Description

Technical Field

[0001] This disclosure relates to human-computer interface technologies and computer privacy, including, for example, out-of-process pointer styles and resizing effects for electronic devices. Background Technology

[0002] Operating system software typically provides an abstraction layer between user interface hardware and applications running on top of the operating system. Multiple applications, each with its own user interface window, can be displayed on a monitor managed by the operating system. Attached Figure Description

[0003] Specific features of this subject matter are set forth in the appended claims. However, for illustrative purposes, several specific embodiments of this subject matter are illustrated in the following figures.

[0004] Figure 1 Example environments for practicing various aspects of this disclosure are illustrated.

[0005] Figure 2 Example systems for providing out-of-process hit tests are illustrated based on various aspects disclosed in this topic.

[0006] Figure 3A and Figure 3B Example methods for providing out-of-process UI services are illustrated based on various aspects disclosed in this topic.

[0007] Figure 4 Example use cases are shown where pointer elements have a pointer style that is applied outside the process.

[0008] Figure 5A and Figure 5B This section provides example use cases of UI elements transitioning from a first state to a second state, based on various aspects disclosed in this topic.

[0009] Figure 6 A flowchart illustrating exemplary operations for providing out-of-process hit tests is provided, based on various aspects disclosed in this subject.

[0010] Figure 7 A flowchart illustrating exemplary operations for providing out-of-process hit tests is provided, based on various aspects disclosed in this subject.

[0011] Figure 8A Example computing devices are illustrated thereon on which various aspects of this disclosure can be implemented. Figure 8B and Figure 8C A flowchart illustrating the application process of implementing various aspects of this disclosure according to some specific implementations is provided. Figure 8D and Figure 8E Examples of equipment and system diagrams according to some specific implementations are shown, each of which implements aspects of this disclosure. Figure 8F and 8G The process of using API call modules and API interfaces based on some specific implementations is illustrated. Detailed Implementation

[0012] The detailed description listed below is intended to describe various configurations of the subject matter and is not intended to represent the only configuration in which the subject matter can be practiced. The accompanying drawings are incorporated herein and form part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject matter. However, the subject matter is not limited to the specific details listed herein, but can be practiced in one or more other specific embodiments. In one or more specific embodiments, structures and components are shown in block diagram form to avoid obscuring the concepts of the subject matter.

[0013] Various aspects of this disclosure include techniques for providing enhanced privacy and / or improved efficiency in computer user input systems. While any computer input system can benefit from these techniques, systems capable of capturing personally identifiable user data (such as extended reality systems that track user gestures and / or eye gaze positions) may particularly benefit from them.

[0014] In one aspect of this disclosure, a user input and rendering system can receive and render user input, and can filter user input that can be used by a computer software application (or "app" as used herein). User input filtering achieves user privacy for the application at least by restricting the application's access to the user's filtered input data. In some aspects, the computer user input and rendering system may be more trustworthy than an application running on the same computer because, for example, the user input and rendering system is provided by a trusted operating system vendor, while the application is provided by a less trustworthy third-party application vendor. Furthermore, the user can explicitly grant authorization to access user input information to the operating system, and may not grant the same authorization to applications, such as third-party applications.

[0015] On the one hand, users can receive rendered feedback from their initial interactions with user interface elements (such as buttons or scrollbars) without making that initial interaction available to the application itself. Initial interactions with UI elements can include intentional initial interactions between the user and the user interface, such as exploring the application's interface (e.g., discovering that a rectangle with a square inside is actually a scrollbar), and initial interactions can also include unintentional and even unconscious interactions with the application's UI elements (e.g., when the user's eyes move across the user interface without the user paying attention to it).

[0016] Various aspects of this disclosure provide techniques for effectively preserving the privacy of an application through initial user UI interactions while still providing the user with rendered feedback from those initial interactions. Rendered feedback from initial interactions with UI elements may include, for example, pointer style changes as the user positions their pointer over different parts of the application interface (such as various UI elements within the application interface). In another example, rendered feedback from initial interactions with UI elements may include hover effects on UI elements that cause the shape of the UI element to change from an initial state to a secondary state when the user's gaze or pointer is near or hovers over the UI element.

[0017] On one hand, when it is determined that a user intends to interact with the application (or its UI elements), user input can be considered no longer preliminary, and some user input can be provided to the application. User input may include, for example, coordinates corresponding to the gaze position or pointer position and an element identifier (id) corresponding to the UI element. On the other hand, the application may provide a declaration or definition of the rendered feedback effect of the initial interaction before the initial interaction with the UI element occurs, thus giving application control over the nature of the initial interaction feedback effect, even if the application is never informed of the initial interaction with its UI element. The application may provide the declaration or definition of the effect to the operating system or another software component that manages the rendering of the initial interaction. When the rendering of the effect is managed in an operating system process separate from the application that provides the declaration or definition of the effect, the effect is referred to herein as a remote effect or “out-of-process effect.” As described herein in conjunction with various examples and / or use cases, an operating system process separate from the application may provide other out-of-process services on behalf of the application without providing information about the out-of-process services to the application (e.g., out-of-process pointer style changes and out-of-process resizing effects, as described in further detail below).

[0018] As described above, aspects of this disclosure can be applied to extended reality systems. A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment may include physical features, such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment, such as through sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, user input may include tracking the physical movement of a person, and in response, the XR system may render modulation of one or more properties of one or more virtual objects simulated in the XR in a manner consistent with at least one physical law. For example, if a user's gesture in an XR system is rendered near but not touching a rendered application UI element, that UI element can be rendered with glowing, shaking, or resizing effects, or a pointer style change can be rendered on top of the UI element or a combination thereof, to otherwise indicate an initial interaction with that UI element without notifying the application of that initial user interaction. In an alternative example, if the user's gesture touches or grasps a UI element, it can be determined that the user intentionally intends to interact with the UI element, and then the user input indicating the touch or grasp can be provided to the application. Furthermore, in the case of a resizing effect, the element ID of the UI element can be provided to the application.

[0019] Many different types of electronic user input and rendering systems enable human users to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to accept an external opaque display (e.g., a smartphone). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0020] Various aspects of this disclosure provide the user with rendered confirmation of user engagement with the application's user interface before providing user input to the application. User input is protected from application influence until confirmation from the user is provided for interaction with the application. The user interface (UI) can be displayed as a location in the physical environment remote from the user input device and / or remote from the display device displaying the user interface (e.g., in a 3D XR display environment). For example, physical movement of a mouse input device can be rendered as movement of a mouse cursor at a physical location on the display different from the physical mouse (perhaps only a few inches). As another example, a gaze cursor can indicate the location of the user's gaze, which is remote from one or more eye-facing cameras used to determine the gaze location. As another example, hand movements can be used in conjunction with the user's gaze to move the cursor relative to the application interface and UI elements within the interface. Similarly, gestures can be rendered as hand objects in virtual space and presented to the user. Therefore, the rendering of user input can be presented to the user at a location physically separate from the physical location of the sensor that obtained the sensed user input. In virtual spaces or other extended reality environments, user input may also appear in a location remote from the display component, which displays the application UI and / or a representation of the user's hand to make the user appear to be located in a location remote from the display component.

[0021] In addition, users may move their hands and / or direct their gaze toward or near the rendered application UI without intending to interact with the application UI (e.g., due to normal movement of the user in the physical environment, such as during a conversation with another person unrelated to the same electronic device), and / or perform gestures to interact with one of the displayed application UIs without intending to provide those gestures to an application associated with another displayed application UI.

[0022] This disclosure includes, at a system process of an electronic device, receiving from an application running on the electronic device a definition of one or more pointer styles for a pointer element managed by the application. A pointer element can be understood as a system-provided control element that a user can manipulate to move around a display to interact with various applications displayed on the display. The system can apply different styles to pointer elements based on their position or usage. Different styles applied to pointer elements alter their appearance and can provide different interactive experiences for the user. In the aspects described herein, the system can apply different pointer styles defined and provided by the application to pointer elements, such that the system can apply pointer styles without revealing the position of the pointer element to the application. The pointer style definition includes a pointer style or a source of pointer styles, and corresponding boundary information related to the application interface, which defines when the pointer style should be applied. In initial interactions, the system can determine that the user is guiding a cursor (such as a mouse pointer, touch pointer, gaze pointer, or gesture input) to interact with various parts of the application. As the cursor moves from one predefined bounded area to another, the pointer style can be rendered based on the cursor's position relative to the bounded area according to the pointer style definition. When a pointer element is displayed by an electronic device, the system process can receive intentional user input, such as pinching or clicking. Upon receiving user input, the system process can provide the application with the user input or a portion of the user input, such as the pointer's position.

[0023] This disclosure includes, at a system process of an electronic device, receiving from an application running on the electronic device a definition of a resizing effect for a first user interface (UI) element managed by that application. In an initial interaction, the system may determine that the user is directing user input (such as gaze input or a cursor, such as a mouse pointer, touch pointer, gaze pointer, or gesture input) onto or near the first UI element. In response, the system may use the defined resizing effect to change the rendered appearance of the first UI element from a first state to a second state. When the first UI element is displayed by the electronic device in the second state, the system process may receive intentional user input (e.g., pinch, click, dwell time trigger, etc.) while the cursor is over the first UI element in the second state. In response to the determination that the user input corresponds to the first UI element in the second state, the location of the user input, along with an indication that the first UI element is in the second state (such as an element ID corresponding to the first UI element), is provided to the application. For example, the initial interaction may cause the first UI element to change (e.g., expand) from the first state to the second state. The application is unaware of the initial interaction and the corresponding change to the second state. Therefore, if the cursor is moved to a location where it intersects with a first UI element in the second state but not in the first state, the application will not understand that the user input was directed to the first UI element without still receiving the indication that the first UI element is in the second state as part of the user input. In one implementation, this indication may include the element ID of the first UI element corresponding to the location of the user input. The application can use the element ID to determine that the first UI element is in the second state, and the application can respond accordingly.

[0024] Figure 1 An example environment 100 for practicing various aspects of this disclosure is illustrated. In example environment 100, user 110 interacts with computing device 120 having display 115. As shown, computing device 120 may include one or more speakers, such as speaker 117 (e.g., a plurality of speakers operable to generate spatialized audio perceived by user 110 as emanating from a location separate from the location of speaker 117 in user 110's physical environment). In one or more examples described herein, user 110 may gaze (e.g., gaze 130) at computing device 120, and the device may sense the user's gaze position 114. In one or more examples described herein, user 110 may point (e.g., point 140) at computing device 120, and the device may sense the pointer position 119. In one or more embodiments, the user's gaze position and / or pointing position may constitute user input to the device.

[0025] Computing device 120 may be a smartphone, tablet, or wearable device (such as a smartwatch or head-mounted portable system) that includes a display system capable of presenting a visualization of an extended reality environment to user 110. The computing device 120 may be powered by a battery and / or any other power source incorporated in and / or (e.g., via cable) coupled to the computing device 120. In one example, the display system of computing device 120 provides a stereoscopic representation of the extended reality environment to the user, thereby enabling a three-dimensional visual display of a rendered scene. In one or more embodiments, this may be an alternative to or supplement to utilizing computing device 120 to access the extended reality environment.

[0026] The computing device 120 may include one or more cameras (e.g., visible light cameras, infrared cameras, etc.). Furthermore, the computing device 120 may include various sensors capable of detecting user input, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near-field communication sensors, etc. Additionally, the computing device 120 may include hardware elements capable of receiving user input, such as hardware buttons or switches. The user input detected by such sensors and / or hardware elements corresponds to various input modalities for interacting with virtual content displayed within a given extended reality environment. For example, such input modalities may include, but are not limited to, face tracking, eye tracking (e.g., gaze direction), hand tracking (e.g., pointing direction), posture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, respiration, temperature, electroencephalogram, olfaction), speech or audio recognition (e.g., specific hot words), and activation of buttons or switches. The computing device 120 may also detect and / or classify physical objects in the physical environment of the computing device 120.

[0027] In one or more embodiments, computing device 120 may be communicatively coupled to infrastructure. Generally, such infrastructure may include more computing resources and / or available power compared to computing device 120. In one example, computing device 120 may operate in various modes. For example, computing device 120 may operate in stand-alone mode independent of any infrastructure.

[0028] The computing device 120 can also operate in a wireless connection mode (e.g., connected to a base device via a wireless connection), thereby working in conjunction with a given base device. The computing device 120 can also operate in a connection mode where the computing device 120 is physically connected to the base device (e.g., via a cable or some other physical connector), and can utilize the power resources provided by the base device (e.g., in cases where the base device charges and / or supplies power to the computing device 120 when physically connected).

[0029] When computing device 120 operates in wireless or connected mode, at least a portion of processing user input and / or rendering the extended reality environment can be offloaded to the base device, thereby reducing the processing burden on computing device 120. For example, in one implementation, computing device 120 works in conjunction with the base device to generate an extended reality environment comprising physical and / or virtual objects that enable various forms of interaction (e.g., visual, auditory, and / or physical or tactile interaction) between the user and the extended reality environment in real time. In one example, computing device 120 provides rendering of a scene corresponding to the extended reality environment, which can be perceived and interacted with by the user in real time. Additionally, as part of presenting the rendered scene, computing device 120 may provide sound and / or haptic or touch feedback to the user. The content of a given rendered scene may depend on available processing power, network availability and capacity, available battery power, and the current system workload.

[0030] The computing device 120 can also detect events that have occurred within the scene of the extended reality environment. Examples of such events include detecting the presence of creatures (such as people or pets, specific people, entities, or objects) in the scene.

[0031] Figure 2 An example system 200 for providing out-of-process effects is illustrated. System 200 may be implemented, for example, on computing device 120. System 200 includes an application process 202 and a system process 204, which corresponds to application 260, and includes a rendering system 270 (e.g., a rendering process) and effects components 280 (e.g., an effects process). In some aspects, processes 202 and 204 may be separate processes running on a common device (such as computing device 120) or on a separate device. Aspects of system process 204 (e.g., including rendering system 270, effects components 280, and / or hit test 282) may be implemented in hardware, software, or a combination thereof.

[0032] like Figure 2 As depicted, system process 204 includes both rendering system 270 and effects component 280. However, this disclosure is not limited thereto. For example, rendering system 270 may run in a separate process from effects component 280, rendering system 270 and effects component 280 may be implemented in a common process, or rendering system 270 and effects component 280 may be further distributed to run in more than two processes, all of which may be separate from application process 202. Similarly, in other respects, application 260 may be distributed to run in more than one application process. Processes such as application process 202 and system process 204, when on a common device (such as...) Figure 1 Computing device 120 or Figure 8A Running on a computing device can provide security and / or privacy between processes. For example, application process 202 and system process 204 can operate with different virtual memory spaces, such as those controlled by the operating system and executed by the processor's memory controller hardware. Separate application and system processes can help prevent application 260 from accessing data within system process 204. In other respects, system process 204, rendering system 270, and effects component 280 may include one or more operating system drivers that do not run in virtual memory space.

[0033] The operation of system 200 may include receiving user input and outputting rendered effects as feedback to the user input, and this may be performed outside of application process 202 and without the knowledge of application 260. User input may correspond to, for example, gaze input and / or pointing input. The rendered effects may include effects applied to UI elements and / or changes to pointer styles applied to pointer elements. Application 260 may provide a description and pointer style of its UI element 250 to rendering system 270, and application 260 may provide definitions of pointer styles and / or effects 256 that it will render in response to future user input. When effect component 280 receives user input corresponding to the effect definition received from application 260, effect component 280 may cause pointer styles and / or effects 256 to be rendered by rendering system 270 as output to be displayed to the user. When effect component 280 receives user input at a specific location when a UI window or UI element with pointer styles defined by application 260 is displayed, effect component 280 may apply pointer style definitions to pointer elements (e.g., border areas or UI elements) corresponding to the specific location.

[0034] In an optional aspect of system 200, effects component 280 may learn the position of a UI element (e.g., the position of a button or scrollbar) directly from application 260 or via optional UI element 258 messages from rendering system 270. In one or more embodiments, effects component 280 may then perform an optional hit test 282 between the current position of the UI element and the current position of the user input. In one or more other embodiments, hit test 282 may be performed by another system process separate from effects component 280 and for a variety of different purposes (e.g., for effects component 280 and other components and / or processes). When a particular user input is identified as associated with a particular UI element through hit test 282 between the user input and a particular UI element of application 260 (e.g., through effects component 280 or another system process of computing device 120), hit test 282 may identify an initial interaction with the particular UI element of application 260. When an initial interaction is identified, pointer style and / or effect 256 may be rendered as user output without notifying application 260 of the user input or the identified interaction. Alternatively, when an alternative user input is identified as a confirmed interaction between the user and a UI element, effect component 280 or another system process may notify application 260 of the confirmed interaction as filtered user input message 254.

[0035] Referring to the preceding paragraphs, instead of performing an optional hit test 282 between the current position of the UI element and the current position of the user input, the optional hit test 282 can be performed between the set of boundaries defined for the UI window and the current position of the user input. The hit test 282 can identify an initial interaction with a specific bounded area of ​​the UI window of application 260 when a particular user input is identified as associated with a specific predefined bounded area of ​​the UI window via a hit test 282 between the user input and the UI window of application 260 (e.g., via effects component 280 or another system process of computing device 120). When an initial interaction is identified, pointer styles and / or effects 256 can be rendered as user output without notifying application 260 of the user input or the identified interaction.

[0036] Effect definition 252 can describe multiple effects or pointer styles rendered by application instruction system process 204 on one or more UI elements and / or pointer elements. Each effect or pointer style defined in effect definition 252 can correspond to a specific type of user input interaction with a specific bounded area of ​​a particular UI element and / or UI window. In various aspects, effect definition 252 can describe multiple pointer styles to be rendered in response to one or more user input interactions with a bounded area of ​​a UI element and / or UI window. For example, effect definition 252 can indicate that when the user's gaze is within a certain threshold distance of a specific area of ​​the UI window or within a certain threshold distance of the UI element and the user's hand is pointing within a certain threshold distance of the specific area of ​​the UI window or within a certain threshold distance of the UI element, the pointer style of the pointer element will change from one style to another. The threshold distance can be, for example, a certain number of pixels, a distance, etc., and can include zero and can be negative (such that the threshold distance is embedded within the UI element). In various aspects, effect definition 252 can describe multiple effects to be rendered in response to a single user input interaction with a single UI element. For example, effect definition 252 can indicate that when the user's gaze is within a specific area of ​​a particular button UI element, the button will animate and transition from a first state to a second state, such as a state larger than the first state.

[0037] Alternatively, application 260 may directly provide information describing UI elements of application 260 to effect component 280 (e.g., instead of effect component 280 receiving UI information from rendering system 270). Furthermore, instead of location-based hit testing, hit test 282 can more generally determine the correspondence between a specific user input and an interaction with a specific user interface element. For example, a user's spoken audio input of "red button" could cause hit test 282 to associate that audio input with the red button UI element.

[0038] In one respect, pointer style and / or effect definition 252 can be a declarative definition. In this respect, application 260 can provide all the information necessary for software components outside application process 202, such as effect component 280, to render the application's desired pointer style and / or effect without the application's knowledge or further involvement. For pointer styles, the declarative definition of the pointer style may include identifiers of UI elements and / or boundary definitions for the position of UI windows (e.g., where UI elements and / or UI windows are provided by a user interface framework or operating system), identifiers that trigger user input, and identifiers of the pointer style to be rendered when the user input corresponds to the UI element and / or boundary definition. The identifier of the pointer style may correspond to a description of the pointer style (e.g., defining one or more parameters and / or variables of one or more of size, shape, dimension, color, etc.), an image of the pointer style, or the source location of the pointer style (e.g., the output of the application that is redirected to become a pointer element).

[0039] For resizing effects, the declarative definition of the effect may include an identifier for a UI element, state information for the UI element's default or first state, state information for the UI element's second or triggered state, transition or animation information for the transition from the first to the second state, and an identifier that triggers user input, wherein the effect to be rendered when the triggered user input corresponds to the first UI element corresponds to transitioning the UI element from the first state to the second state. In one or more embodiments, the second state of the UI element may include multiple additional UI elements, each of which is enabled for further out-of-process hit testing according to the procedures described herein. In one or more embodiments, the UI elements and / or defined bounded regions may be provided from one or more files such as a Universal Scene Description (USDZ) file. In another embodiment, the UI elements and / or bounded regions referenced in the application's declarative definition may be provided by the operating system. In one or more embodiments, the declarative definition for the resizing effect may include state information for any number of states (such as for X states greater than the current state and / or Y states smaller than the current state).

[0040] Figure 3A This example demonstrates a method for rendering UI effects. As shown in the figure, Figure 3A The method may include in system process 300 (e.g., Figure 2The system process 300 receives definitions of effects for UI elements of the application (box 302) at system process 204. One or more definitions of one or more control styles may also be received at box 302. User input may be received by system process 300 (box 304), and when (e.g., by system process 300, such as by performing hit test 282) it is determined that the received user input corresponds to a bounded position within the UI element and / or UI window corresponding to the received pointer style definition (box 306), the system process 300 may (e.g., by the system process 300) render the effect and / or pointer style on the UI element or pointer element according to the effect definition and / or pointer definition (box 308). In one aspect, it may execute in system process 300, separate from the application process 350 in which the application runs (e.g., an implementation of application process 204). Figure 3A The operation.

[0041] Figure 3A The method may include optional additional operations, such as system process 300 receiving additional user input that differs from the user input received in box 304 (box 310). When the user's intent to interact with an application or an application's UI element is identified by system process 300 (e.g., by performing hit test 282) based on the original user input and / or additional user input (box 312), some or all of the user input identifying the user's intent may optionally be provided from system process 300 to application process 350 (or otherwise made available to that application process) (box 314). In one or more use cases, user input may be provided to application process 350 in conjunction with an application's UI element, for which the user's intent to interact with the application or the application's UI element is identified. In one or more other use cases, a foreground UI window or foreground UI element may be displayed, for which a foreground control style definition is provided to system process 300 (e.g., at box 302), and some or all of the user inputs identifying user intent may be provided from system process 300 to application process 350 (or otherwise made available to application process) (box 314) in conjunction with the foreground UI window or foreground UI element (e.g., even if user input is identified as intended to interact with another UI element or UI window).

[0042] Through Figure 3A and Figure 3BThe referenced UI elements can be part of the application's user interface. For example, UI elements can be defined by the application, and their purpose can be to enable users to interact with the application (after identifying the user's intention to interact with the application). In one aspect, UI elements can include one or more attributes, such as the UI element's position, size, orientation, color, transparency or brightness (as an example), and control style. In one or more implementations, effect definitions can include modifications to one or more attributes (such as the UI element's position, size, orientation, color, transparency or brightness (as an example)). For example, an effect definition for a button UI element can include a boosting effect to raise the button's position attribute and a brightening effect to increase the button's brightness attribute. In another example, an effect definition for a button UI element can include a resizing effect to change the button UI element from a smaller size to a larger size, and can include animation definitions for performing the transition. Furthermore, application UI elements can include predefined "remote states," where remote states correspond to predefined values ​​of UI element attributes. The remote state of an application's UI elements can be considered remote because, while an application can define UI elements, their properties, and their remote states, only system components within a single system process (such as the rendering system 270 and effects component 280) can manage or know the remote state of the UI elements. This allows rendering of the user's initial interaction with the UI elements to be performed while maintaining the application's privacy by preventing the application from discovering at any given time which remote state the UI element is in. For example, for a resizing definition, the remote state could include initial state information of the UI element and one or more second or triggered state information of the UI element, and the system could implement a change from the first state to the second state based on the user's initial interaction without providing input to the application.

[0043] When an application provides customization options to the system for applying pointer styles to pointer elements, throughout Figure 3A and Figure 3BThe referenced pointer element can be part of the application's user interface. For example, the pointer style for a pointer element can be defined by the application, and the purpose of the pointer style can be to enable the user to perform an initial interaction with the application using a pointer style that is context-sensitive to the application while still protecting user input. Custom pointer styles can more effectively allow the intended use of user input to the application after identifying the user's intent to interact with the application. For example, a default pointer style may not adequately represent the action to be performed, while a custom pointer style can provide a more useful initial interaction with the UI elements of the application's interface. In one aspect, a pointer style can include one or more attributes or source definitions, such as size, orientation, color, transparency, shape, dimensions, brightness, image, icon, or a redirected application-provided style (as an example). In one or more implementations, a pointer style definition can include modifications to one or more attributes of another pointer style. For example, a painting application can provide different brushes and erasers. Pointer styles can define a brush pointer style when the brush tool is selected and an eraser pointer style when the eraser tool is selected. However, when user input moves the pointer element to a portion that is no longer on the canvas of the drawing application, the pointer style can change to another pointer style, so the user is not confused as to why the pointer of the drawing pen style would be on a portion of the drawing application that is not on the canvas. Furthermore, if the pointer element is directed to move outside the UI window provided by the application (e.g., outside the application's boundaries) and into the system-provided portion of the display, the pointer style reverts to the default pointer style or the system-provided pointer style. The movement of the pointer element can be an initial interaction where no user input is provided to the application (until the user, for example, pinches or clicks, to indicate that user input should be sent). In this case, user input can include gaze input (e.g., gaze at 130) and hand input (e.g., pointing to 140), where gaze input indicates the application to which the interaction is expected, and hand input controls the pointer element. Additionally, the application pointer style can include a predefined "remote state," where the remote state corresponds to a predefined value of the pointer style attribute. The remote state of an application's pointer style can be considered remote because, although an application can define pointer styles, their properties, and their remote states, only system components within a single system process (such as the rendering system 270 and the effects component 280) can manage or know the remote state of the pointer style. This allows the rendering of the user's initial interaction with the UI via pointer elements to utilize the corresponding pointer style, while maintaining the application's privacy by preventing the application from discovering the remote state of UI elements at any given time.

[0044] On the one hand, it is also possible to obtain it directly from the application or from alternative sources (such as the UI framework and file format mentioned above) or from the operating system. Figure 3A and Figure 3BThe referenced UI elements and / or pointer styles. These UI elements, provided by alternative sources, can serve as the application's user interface (such as...). Figure 2 The UI elements and / or pointer styles from the alternative source may have associated attributes and "remote states" defined by the alternative source and / or by the application using them. For example, even if the UI elements and / or pointer styles may originate from the alternative source, they can still be managed by the application using them to enable the system process to render remote effects while maintaining the privacy of the managing application.

[0045] On one hand, UI elements and / or pointer styles can be specified as part or all of a layer tree or render tree. For example, application 260 may provide a layer tree or render tree of UI (including, for example, pointer styles) and UI elements to rendering system 270, and application 260 may provide effect definitions 252, including the layer tree or render tree of UI elements, to an operating system or other component that manages UI effects outside the process.

[0046] Effect and / or pointer style definitions (box 302) may be received from a specific application running in application process 350 and may describe effects (such as resizing effects) to be rendered on UI elements belonging to that specific application, and / or may describe pointer styles to be rendered at pointer elements when the pointer element is located in certain specified areas or above certain specified UI elements. However, specific implementations of this disclosure are not limited thereto. For example, the effect and / or pointer style definitions received in box 302 may be received indirectly from other sources, and the effect and / or pointer style definitions may be predetermined effects, or the UI elements may be predetermined UI elements. In one example, a button UI element may be predefined, and a "lift" effect may be predefined to occur when a gaze or hover user input is determined to correspond to a predefined button effect. In another example, a UI element may have a first state and a second state to transition to when a gaze or hover user input is determined to correspond to a predefined UI element. In the case of predefined effects and / or predefined UI elements, the received effect definition may include a reference to the predefined element without fully defining such an element.

[0047] On one hand, effect definitions can identify the effects to be rendered for a UI element by identifying the remote state (with predefined rendering property values) to be used when user input is determined to correspond to a UI element. On the other hand, effect definitions can identify the animations to be used when transitioning between predefined remote states.

[0048] User input (such as that received in boxes 304 or 310 of Figure 3) may include one or more different types of input from the user. For example, in addition to user input received from user input devices such as a mouse, keyboard, and / or audio input devices, received user input may also include position and / or orientation tracking of the user's body or body parts (such as fingertips or hands), gesture recognition (such as hand shape, hand movements, and / or facial expressions), or the position of the user's optical gaze (e.g., gaze position 114). User input may include time or velocity measurements, such as gaze duration, which measures the duration for which the user's gaze remains on or within a specific distance from a particular position or UI element, or hovering duration, which measures the duration for which the user's hand or other body parts remain within a specific two-dimensional or three-dimensional distance from a particular position or UI element.

[0049] On one hand, UI effects (such as those defined in box 302 and rendered in box 308 of FIG. 3) may include any rendered output that is perceptible to the user, including audio, image, and / or haptic effects. Examples of audio effects on UI elements may include audible sounds designed to be perceived as emanating from the location of the UI element, or distortion of audio that has already been emitted from the UI element before the audio effect is rendered. As examples of image effects, UI elements may be displayed to the user or otherwise visually presented in a way that alters the appearance or position of the UI element before the effect is rendered, and the rendering effect may change the appearance or position of the UI element. For example, a glowing effect may cause an existing UI element to glow. For example, UI effects may also include a resizing effect that changes the apparent size of a UI element from a first state to a second state, where the second state may include additional UI elements. A highlighting effect may add new visual cues, such as circles or checkmarks, near the location of a UI element. On one hand, the effect definition may specify a “reverse” rendering effect, where the specified effect is rendered generally unless the user input corresponds to a UI element.

[0050] On one hand, identifying the correspondence between user input and a first UI element or pointer style (box 306) may include identifying multiple user interface elements that the first UI element may intersect with, and then determining that the multiple identified UI elements include the first UI element; or it may include identifying a predefined bounded region of the user interface. For example, user input may be a gaze position. The system process may execute a hit test process (e.g., hit test 282) that includes identifying a list of potential UI elements and / or bounded regions that the user may be trying to interact with, such as by identifying all UI elements and / or bounded regions from all applications located within a threshold distance of the gaze position and / or identifying all UI elements that intersect with the gaze direction. Similarly, for example, user input may be a hand gesture or finger positioning in a pointer position. If an effect or pointer style definition is included in the list of applied UI elements, a corresponding effect and / or a pointer element that can be modified to correspond to the pointer style corresponding to the applied UI element may be rendered on the applied UI element. If a pointer style definition includes a bounded position of the applied UI in the list, a corresponding pointer style may be rendered on the pointer element. In use cases where the gaze position or hand pointer position intersects with multiple UI elements, each with its own effect definition, the system process can render effects or select pointer styles on the UI element that appears closest to the user among the multiple UI elements and / or on the UI element where the gaze position or hand pointer position intersects most centrally among the multiple UI elements.

[0051] On one hand, a single effect can be defined for a group of UI elements. For example, when a user input correspondence (306) is determined for any UI element in the group, the defined effect can be rendered on all UI elements in the group. For example, if the defined effect is a resizing effect, each corresponding UI element in the group may have a defined second state applied during the application of the resizing effect. In another example, the user input correspondence in box 306 may be determined only by a subset of the UI elements in the group. The definition of an effect for a group of UI elements may include an indication of which subset of the UI elements in the group to be tested. If the user input location (such as the location of the user's gaze or gesture) is within a specific proximity range (e.g., a distance defined by a system process or a predefined distance in the effect definition) of any element in the indicated subset of UI elements, the effect can be rendered on all UI elements in the group (e.g., unless a foreground UI window or foreground UI element with a foreground control style has been defined and displayed). In another example, if a UI element is in a second state because a resizing effect is being applied, the second state of the UI element may contain one or more UI child elements, where, by definition, user input can be determined to correspond to one of these one or more UI child elements to provide further effects and / or pointer style changes.

[0052] On one hand, a single effect definition can define a set of rendering effects. When the correspondence with a UI element is determined (box 306), multiple different effects can be used to render the UI element (box 308). For example, an effect definition for a button may include a combination of boost effects (e.g., the button moves in the opposite direction to when it is pressed), glow effects (e.g., brightening), audio cues (e.g., the tone of an audio emanating from the button), and resizing effects.

[0053] On one hand, effect definitions can combine multiple aspects mentioned above. For example, a single effect definition may include multiple UI elements, multiple effects to be rendered, and / or various types of user input that can trigger the rendering of one or more effects. Furthermore, effect definitions can be combined with pointer style definitions.

[0054] On the one hand, even after identifying the user's intent to interact with the application (box 312), only an overview or subset of the user's input can be provided to the application (box 314). For example, an intent to interact with the application can be identified once the user's gaze lingers on an application button for more than a threshold. However, instead of providing the application with the user's gaze location or gaze duration, the application can be given an indication that the user intends to press the application button, without providing the application with any knowledge about the user's gaze.

[0055] On one hand, remote effect definitions can specify the animations used to render when transitioning between remote states. For example, when the effect is a resizing effect, a remote state transition animation can be defined for the transition from the first state of a UI element to its second state. Furthermore, in some aspects, the second state of a UI element can include continuous animations of the UI element. If the second state of a UI element includes UI child elements and is animated, then the user input sent to the application when the interaction is confirmed (box 312) can include the current position of the UI element relative to the global coordinate system or relative to the application interface, and the position of the user input relative to the application interface.

[0056] On one hand, additional user input can be received (box 310). Additional user input may include physical indications of the user's intent to interact with the application. For example, a user may use their hands to make pinching, swiping, or flicking gestures. Other gestures may be made using the legs, arms, or head. Gestures may include hand gestures. Additional user input may also be, or alternatively, mouse clicks, sound signals, taps, etc. Additional user input may be received by a system process, and the system process may use the additional user input to identify (box 312) the user's intent to interact with a UI element or application. When the system identifies the user's intent to interact with a UI element or application, it may provide the user input to the application (box 314). The provided user input may be an abstraction of the user input. For example, instead of providing eye-gazing input to the application, the location of input without source type information may be provided to the application.

[0057] On one hand, for example when the effect definition is for a resizing effect, an initial interaction can trigger the resizing effect of a first UI element to change from its first state to a second state. The second state can be larger in size than the first state. The application is unaware of the state of the first UI element. Therefore, when user input is sent to the application, the position corresponding to the user input can be directed to the first UI element in the second state, but if the application only receives the user input position data, it may assume that the first UI element is in the first state, and thus the input may lead to unexpected results. However, when the user expects to interact with the first UI element in the second state, because the second state is unavailable to the application, the application directs the input to perhaps another UI element, or perhaps to a non-interactive feature of the application. Aspects of this subject matter send the ID of the UI element in the second state corresponding to the user input position data along with the position information. The application can receive this ID information and determine that because the user input position data corresponds to a position not aligned with the first state of the first UI element, the first UI element must be in the second state, and can therefore respond accordingly.

[0058] On one hand, pointer styles can correspond to pointer styles that are sized or patterned to affect a large area of ​​the application interface. For example, in a painting application, brush-style pointers can have different sizes so that their use affects a large area of ​​the application interface. Pattern-style pointers can correspond to zebra stripes or some other pattern that affects multiple areas of the application interface. When user input is provided to the application (box 314), the user input may include the position of the pointer element when additional user input is provided. In such cases, the additional user input may include user input position data. The application is aware of its own state, so when it receives the user input position data, it can determine which pointer style was applied when the additional user input was provided and modify the application interface accordingly.

[0059] In one or more aspects, the system can apply pointer styles to pointer elements that interact with virtual objects. Virtual objects can be generated by an application based on visual sensor input from the physical environment, replacing physical objects in the environment. In some implementations, after the virtual object is generated, it can be made invisible, making the physical object appear to be in the user's field of vision. The user can interact with the virtual object, which now appears to be a physical object, thus allowing pointer styles to be applied to interactions with what appears to be a physical object. Furthermore, effects can be applied in a similar manner, for example, after the virtual object is generated, such as applying surface textures, labels, logos, or color effects to the virtual object (which is now invisible), without providing the system with user input data related to the initial user interaction with the virtual object.

[0060] Figure 3B Example methods for rendering UI effects, based on various aspects disclosed in this topic, are illustrated. As shown in the figure, Figure 3B The method may include application-managed rendering effects and / or pointer styles, as well as remote rendering effects and / or pointer styles, identified by application process 350 for corresponding UI elements and / or pointer elements (box 352). The definition of the identified remote rendering effects may be provided by application process 350 to a remote effects rendering service provider (such as...). Figure 2(Rendering system 270 and / or effects component 280). For UI elements with corresponding application-managed effects, user input can be received by application process 350 (box 356). Application-managed UI effects can be rendered by application process 350 on their corresponding UI elements (box 358). In one or more embodiments, application-managed UI effects and / or one or more application functions that can be performed by the application in response to user input can be performed by application process 350 based on which parts (e.g., which UI elements) of the application's UI the user input interacts with. In one or more embodiments, when the foreground control style is defined by application process 350 and provided (354) to system process 300, the user input received at box 356 can be received in conjunction with a UI window or UI element different from the UI window or UI element where system process 300 detects user input (e.g., a foreground UI window with the defined foreground control style).

[0061] On the one hand, it can be executed in application process 350, which is separate from system process 300, which manages remote / out-of-process UI effects and pointer styles. Figure 3B The operation of the method. On the other hand, Figure 3A The method could be an example method for providing remote effects and pointer style rendering services, and the effect definition and pointer style definition provided in box 354 could be received at box 302. In another aspect, additional user input provided in box 314 could be received at box 356. In one or more embodiments, the pointer style definition provided in box 354 and received at box 302 may include those provided by the application and... Figure 3A The method redirects the rendering pointer to the position of the pointer element.

[0062] In other respects, the effect of initial user interaction can be identified as a remote UI effect, while the effect of confirmed user input can be identified as an application-managed UI effect and / or pointer style (box 352). In one or more embodiments, the application-managed UI effect and / or pointer style based on confirmed user input can be managed by the application based on the location of the user input (e.g., as provided in box 314) and optionally the element ID corresponding to the UI element to which the effect has been applied. For example, if the effect applied to the first UI element is a resizing effect, the user input can be provided at a location that does not correspond to the first UI element in a first state but only to the first UI element in a second state. The application process 350 is unaware that the effect of the first UI element has been applied, and therefore the operation at box 314 can include the element ID of the first UI element, causing the application to receive an indication that the first UI element is in a second state. The application process 350 can then determine, based on the user input location and known information about the second state of the first UI element, which part of the first UI element in the second state has received the user input. In one or more embodiments described further in detail below, the system process 300 can redirect the rendered pointer style from the application and apply it to the pointer element.

[0063] Figure 4 An example use case is illustrated where one or more UI windows 402 of an application (such as application 260) are displayed in the visible display area 400 of computing device 120 (e.g., the visible area of ​​display 115 of computing device 120). For example, the display 115 of computing device 120 (e.g., see...) Figure 1 It is operable to display one or more user interface windows 402 corresponding to an underlying application (e.g., application 260) running on computing device 120 in the visible display area 400 of display 115. As shown, computing device 120 may also display a UI window 422 of another underlying application running on computing device 120.

[0064] As shown in the figure, the user interface window 402 of application 260 may include one or more user interface elements 406. As shown in the figure, the user interface window 422 may also include one or more UI elements 426. As an example, user interface elements 406 and / or UI 426 may include virtual buttons, virtual switches, virtual lists (e.g., dropdown lists), tabs, scroll bars, application icons, and / or other interactive elements. Figure 4As shown, for illustrative purposes only and not as a limitation, some of the user interface elements in user interface element 406 correspond to various tools that can be selected for use in application 260 of the painting application, such as move tool 406A, brush tool 406B, pencil tool 406C, fill tool 406D, zoom tool 406E, text tool 406F, and eraser tool 406G.

[0065] As described herein, when user input to computing device 120 is received within the bounded region 410 of UI window 402 (e.g., directing a system pointer element to the computing device), or when user input to computing device 120 is received at position 408 (which intersects with UI element 406), system process 300 may render a pointer style for the pointer element for use in user interface element 406. In other words, the system can apply pointer styles to pointer elements through two different mechanisms, which can each be implemented simultaneously. In the example of the first mechanism, the initial interaction with UI window 402 occurs within a predefined bounded region or region 410.

[0066] exist Figure 4 In the illustrated example, when a specific tool is selected in a previous user input to application 260, application 260 can provide pointer style information for that tool to the system. Then, when the initial interaction with UI window 402 controls the system pointer element 428 to be at position 408B (within the large bounded area 410), the pointer style corresponding to the specific tool can be applied to pointer element 428B. In the example of the second mechanism, when pointer element 428 is moved to position 408C, the initial interaction provides user input (in this case, the pointer element position) that intersects with the UI element corresponding to pencil tool 406C. When the pointer element is determined to intersect with the UI element corresponding to pencil tool 406C, the pointer style can be changed to another pointer style, such as the pointer style depicted at pointer element 428A. In some aspects, UI elements may not be used for pointer style definition, and only defined bounded areas may be used, such as a smaller bounded area 410 surrounding tool UI element 406, to define pointer elements to apply the specific pointer element style shown at pointer element 428A. In some respects, when pointer element 428 is within UI window 402 but outside a predefined bounded region (such as a large predefined bounded region 410), the pointer style can be pre-selected, or the default pointer style can be used. In other words, developers may only need to specify the bounded region of interest where the pointer style can be changed and anywhere else the pointer style can be defined by default.

[0067] In the immediate example, selecting the brush tool 406B and initial interaction allows the user to input control of the pointer element at position 408B. Because this position is within a predefined bounded area 410 as part of a pointer style definition from application 260, the system can utilize the pointer style corresponding to the brush tool 406B. Another tool, such as the pencil tool 406C, can be selected by moving the pointer element to the UI element corresponding to it. For example, the user's gaze and / or finger movement can manipulate the pointer element to participate in various initial interactions with the UI window 402 of application 260 to reposition the pointer element. When the pointer element is at position 408C, within a smaller bounded area 410, and / or intersects with the pencil tool 406C UI element, the pointer style can change to a selection pointer (as illustrated at pointer element 428A).

[0068] Furthermore, as described herein, when user input to computing device 120 is received at position 408 of UI element 406 (e.g., when the user's gaze position 114 falls within or within the boundaries of UI element 406, and / or when the user's hand or finger hovers over or within the boundaries of UI element 406 for a predetermined amount of time), system process 300 may render one or more effects of user interface element 406. As described herein, these effects may be out-of-process effects predefined by the underlying application of UI element 406, rendered without providing user information (e.g., the location of user input) to the application.

[0069] In one or more specific implementations, user input to UI element 426 and / or another interaction with the application UI may cause the system to apply a resizing effect to UI element 426 to change UI element 426 from a first state to a second state, and apply a transition animation to UI element 426, all according to the effect definition.

[0070] Figure 5A and Figure 5B Examples include Figure 4 Example use case for UI window 422, and the initial user interaction with UI element 426B changes UI element 426B from its first state 426B' to its second state 426B". For example Figure 5AAs indicated, a hit test (e.g., hit test 282) performed by system process 300 can determine that the location 408 of the user input is the location where it intersects with the UI element 426B in the initial interaction. For example, the user input at location 408 can be initial user input, such as a hand or gaze or hovering over an input device (e.g., for more than a predetermined amount of time, such as dwell time). In this example, the user input at location 408 can cause system process 300 (e.g., without providing any information about the user input or location 408 to the application) to provide out-of-process or remote effects (e.g., visual or audio effects) to cause UI element 426B to resize and change from a first state 426B' to a second state 426B" according to the effect definition of UI element 426B.

[0071] like Figure 5B As indicated, when the resizing process is completed by animing the change from the first state 426B' to the second state 426B", the second state 426B" may include additional UI elements, such as UI element 426B"-1 (e.g., a slider control), UI element 426B"-2 (e.g., a button), and UI element 426B"-3 (e.g., a button). It should be understood that these elements are examples, and any suitable UI element type may be used as needed. Furthermore, each of these UI elements may have an effect definition provided to the system process 300 by the application process 350, and a hit test (e.g., hit test 282) may be performed to apply the effect to each of these UI elements.

[0072] It should be noted that the second state 426B" of UI element 426 has been extended to cover the UI element in the first state 426B', and the second state 426B" of UI element 426 has been extended to cover a portion of UI element 426C. Therefore, if a confirmed user input is about to occur (e.g., including a detected pinch or click gesture, a pinch or click gesture combined with a gaze, or a gaze or hover for at least a predetermined amount of time, such as a dwell time), such that the user input is provided to the application, for example as coordinates, if the user input interacts with one of the UI elements 426B"-1, 426B"-2, or 426B"-3, the application receiving the user input will not understand that the user input was directed to UI element 426 in the second state 426B", and the user input may be misunderstood. For example, the application may understand that the user input sent to the location of UI element 426B"-3 is intended for UI element 426C because they intersect. In this case, in addition to the location corresponding to the user input, the system process 300 may also send an indication to the application process 350 of which UI element to interact with. This indication may be, for example, the element ID of the UI element, the name of the UI element, an indication of the state information of the UI element, etc. The application process 350 may then adjust its internal understanding of the application state to internally replace the first state 426B' with the second state 426B" and then determine it based on, for example, the location of the user input that interacts with UI element 426B"-3.

[0073] It should also be understood that, regarding Figure 4 The pointer style discussed can be related to... Figure 5A and Figure 5B The discussion covers combinations of out-of-process resizing effects, which can also be combined with other out-of-process effects discussed in this paper, such as glow and hover effects.

[0074] Figure 6 Example procedure 600, based on one or more specific implementations, is illustrated for providing out-of-process hit tests and applying pointer-style methods. For illustrative purposes, this document primarily refers to... Figure 1 The computing device 120 is used to describe process 600. However, process 600 is not limited to... Figure 1 The computing device 120, and one or more blocks (or operations) of process 600 may be performed by one or more other components of other suitable devices. Further, for illustrative purposes, some blocks of process 600 are described herein as occurring sequentially or linearly. However, multiple blocks of process 600 may occur in parallel. Moreover, the blocks of process 600 need not be executed in the order shown, and / or one or more blocks of process 600 need not be executed and / or may be replaced by other operations.

[0075] like Figure 6As shown, at box 602, process 600 may include: receiving, at a system process of the electronic device, a definition of a first pointer style managed by an application running on the electronic device for at least a first portion of a first user interface window for a user interface (UI). User input may correspond to an initial interaction. The pointer element may be displayed by the system process without providing the application with an indication of the pointer element in the first portion of the first user interface window.

[0076] The definition of a first pointer style may include a description of the first pointer style, which includes one or more of a size parameter, a color parameter, an offset parameter, or a shape parameter. The definition of the first pointer style includes the location of the applied rendering output, and displaying the pointer element according to the definition of the first pointer style includes redirecting the applied rendering output to the location corresponding to the pointer element. The definition of the first pointer style may include an indication of a bounded region corresponding to a first portion of the first user interface window. The definition of the first pointer style may include an indication of a first user interface element of the first user interface window.

[0077] In some respects, it may accept one or more definitions of one or more additional pointer styles, where each pointer style indicates the part of the UI to which the pointer style is applied.

[0078] At box 604, process 600 may further include: when the user inputs an indication pointer element in the first part of the first user interface window, the system process displays the pointer element according to the definition of the first pointer style.

[0079] At box 606, the process may further include: when the user inputs an indication pointer element in a portion of the first user interface window other than the first portion of the first user interface window, the system process displays the pointer element in a style other than the first pointer style. Styles other than the first pointer style may also be provided to the system process by the application in a definition declaration.

[0080] In some aspects, process 600 may also include: receiving initial user input to move the pointer element out of a first portion of the first user interface window; and changing the pointer style of the pointer element to a style other than the first pointer style.

[0081] In some respects, initial user input includes at least one of gaze input indicating interaction with the first user interface window or finger gestures controlling pointer elements.

[0082] In some aspects, process 600 further includes: receiving active input from a user by a system process to provide the active input to an application; providing the active input to the application, the active input including the position of a pointer element in a first user interface window; in response to the active input, receiving an updated definition of a first pointer style for a first portion of the first user interface window by the system process; and when the user inputs an indication that the pointer element is in the first portion of the first user interface window, displaying the pointer element by the system process according to the updated definition of the first pointer style.

[0083] In some respects, process 600 also includes: when the system process displays the pointer element according to the definition of the first pointer style, applying a resizing effect to a first UI element that intersects with the pointer element, the resizing effect causing the display of the first UI element to change from a first state to a second state.

[0084] Figure 7 Example process 700, based on one or more specific implementations, is illustrated for providing out-of-process hit testing and resizing effects. For illustrative purposes, this document primarily refers to... Figure 1 The computing device 120 is used to describe process 700. However, process 700 is not limited to... Figure 1 The computing device 120, and one or more blocks (or operations) of process 700 may be performed by one or more other components of other suitable devices. Further, for illustrative purposes, some blocks of process 700 are described herein as occurring sequentially or linearly. However, multiple blocks of process 700 may occur in parallel. Furthermore, the blocks of process 700 need not be executed in the order shown, and / or one or more blocks of process 700 need not be executed and / or may be replaced by other operations.

[0085] At box 702, process 700 may include: receiving, at a system process of the electronic device, a definition of a resizing effect managed by an application running on the electronic device for a first user interface (UI) element, the definition of the resizing effect including a configuration of a second state for the first UI element. The definition of the resizing effect may include a configuration of a first state for the first UI element.

[0086] At box 704, process 700 may further include: when the user inputs an instruction to interact with the first UI element while it is in a first state, the system process changes the display of the first UI element from the first state to a second state of the first UI element according to the definition of a resizing effect. The interaction may be a resizing interaction. The display size of the second state may be larger than the display size of the first state.

[0087] At box 706, process 700 may include: determining by the system process the intent to provide user input information to the first UI element when in the second state.

[0088] At box 708, process 700 may include: providing user input information to the application, the user input information indicating the location of the user input and the element ID of a first UI element. The location of the user input may be outside the area of ​​a first state of the first UI element and within the area of ​​a second state of the first UI element.

[0089] In some aspects, the second state of the first UI element includes one or more additional UI elements, and process 700 may further include: when user input in the first UI element indicates further indication regarding a second UI element among the one or more additional UI elements, applying an effect to the second UI element, wherein the effect is applied according to an effect definition received from the application. When the first UI element is in the first state, one or more additional UI elements may be unavailable.

[0090] In some aspects, for example, when the user detaches from the first UI element, the first UI element can return from the second state to the first state. Process 700 may also include, for example: receiving user input by a system process while the first UI element is in the second state, the user input indicating that the position corresponding to the user input is outside the intersection with the first UI element; and the system process changing the display of the first UI element from the second state to the first state. In some aspects, process 700 may also include: providing a pointer style for a pointer element based on the position of the user input; and updating the pointer style for the pointer element when the position of the user input changes from a first redefined region to a second predefined region.

[0091] although Figure 7 An example block for process 700 is shown, but in some specific implementations, it differs from... Figure 7 Compared to the boxes depicted, process 700 may include additional boxes, fewer boxes, different boxes, or boxes with different arrangements. Additionally or alternatively, two or more boxes in process 700 may be executed in parallel.

[0092] As described above, one aspect of the present invention is the collection and use of data available from specific and legitimate sources to provide out-of-process hit testing for electronic devices. This disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include audio data, voice data, demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, encrypted information, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other personal information.

[0093] This disclosure recognizes that the use of personal information data in the techniques of this invention can benefit users. For example, personal information data can be used to provide out-of-process hit testing for electronic devices.

[0094] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, it is expected that such entities will implement and consistently apply privacy practices generally recognized as meeting or exceeding industry or governmental requirements for protecting user privacy. Such information regarding the use of personal data should be prominently displayed and easily accessible to users, and should be updated as the collection and / or use of data changes. Users' personal information should be collected only for lawful use. Furthermore, such collection / sharing should only occur after receiving user consent or other lawful grounds provided for in applicable law. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that other entities authorized to access personal information data comply with the privacy policies and procedures of other entities. Furthermore, such entities may be subject to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. Moreover, policies and practices should be tailored to the specific types of personal information data collected and / or accessed, and made applicable to applicable laws and standards, including specific jurisdictional considerations that may be used to impose higher standards. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0095] Regardless of the foregoing, this disclosure also anticipates implementation schemes for users to selectively block the use or access to personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. For example, in an example of providing out-of-process hit testing for electronic devices, the inventive technology can be configured to allow users to opt-in or opt-out to participate in the collection and / or sharing of personal information data during or at any time after registering for the service. In addition to providing opt-in and opt-out options, this disclosure also anticipates providing notifications related to access to or use of personal information. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0096] Furthermore, the intent of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by restricting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level or at a level insufficient for facial recognition), controlling how data is stored (e.g., aggregating data among users), and / or other methods such as differentiated privacy.

[0097] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not become inoperable due to the absence of all or part of such personal information data.

[0098] Figure 8AExample computing devices 800 are illustrated according to one or more specific embodiments that can be used to implement aspects of the subject matter. The computing device 800 may be any computing device or server for generating the features and processes described above and / or may be part of any computing device or server for generating the features and processes described above, including but not limited to laptop computers, smartphones, tablet devices, wearable devices such as goggles or glasses. The computing device 800 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The computing device 800 includes a persistent storage device 802, a system memory 804 (and / or buffers), an input device interface 806, an output device interface 808, a bus 810, a ROM 812, one or more processing units 814, one or more network interfaces 816, and / or subsets and variations thereof.

[0099] Bus 810 generally represents all system, peripheral, and chipset buses that communicatively connect numerous internal devices of computing device 800. In one or more embodiments, bus 810 communicatively connects one or more processing units 814 to ROM 812, system memory 804, and persistent storage device 802. One or more processing units 814 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in this subject matter. In different embodiments, one or more processing units 814 may be a single processor or a multi-core processor.

[0100] ROM 812 stores static data and instructions required by one or more processing units 814 and other modules of computing device 800. On the other hand, persistent storage device 802 can be a read-write memory device. Persistent storage device 802 can be a non-volatile memory cell that stores instructions and data even when computing device 800 is turned off. In one or more embodiments, mass storage devices (such as disks or optical discs and their corresponding disk drives) can be used as persistent storage device 802.

[0101] In one or more embodiments, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) may be used as persistent storage device 802. Like persistent storage device 802, system memory 804 may be a read-write memory device. However, unlike persistent storage device 802, system memory 804 may be volatile read-write memory, such as random access memory. System memory 804 may store any instructions and data that one or more processing units 814 may need during operation. In one or more embodiments, the processes disclosed in this subject matter are stored in system memory 804, persistent storage device 802, and / or ROM 812. One or more processing units 814 retrieve instructions to be executed and data to be processed from these various memory units to execute the processes of one or more embodiments.

[0102] Bus 810 is also connected to input device interface 806 and output device interface 808. Input device interface 806 enables a user to communicate information and select commands to computing device 800. Input devices that can be used with input device interface 806 may include, for example, an alphanumeric keypad and pointing devices (also referred to as "cursor control devices"). Output device interface 808 may, for example, enable the display of images generated by computing device 800. Output devices that can be used with output device interface 808 may include, for example, printers and display devices such as liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, flexible displays, flat panel displays, solid-state displays, projectors, or any other device for outputting information.

[0103] One or more embodiments may include a device that functions as both an input and an output device, such as a touchscreen. In these embodiments, the feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0104] Finally, as Figure 8A As shown, bus 810 also couples computing device 800 to one or more networks and / or one or more network nodes via one or more network interfaces 816. In this way, computing device 800 may be part of a computer network (such as a LAN, wide area network (“WAN”), or intranet), or may be part of a network of networks (such as the Internet). Any or all components of computing device 800 may be used in conjunction with the subject matter disclosed herein.

[0105] Specific embodiments within the scope of this disclosure may be implemented, in part or in whole, using a tangible computer-readable storage medium (or a plurality of tangible computer-readable storage media of one or more types) that encodes one or more instructions. The tangible computer-readable storage medium may also be substantially non-transitory.

[0106] Computer-readable storage media can be any storage medium that can be read, written, or otherwise accessed by general-purpose or special-purpose computing devices, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, track memory, FJG, and Millipede memory.

[0107] Furthermore, computer-readable storage media may include any non-semiconductor memory, such as optical disc storage devices, magnetic disk storage devices, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more embodiments, the tangible computer-readable storage medium may be directly coupled to a computing device, while in other embodiments, the tangible computer-readable storage medium may be indirectly coupled to a computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0108] Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or as high-level language instructions that can be compiled to produce executable or non-executable machine code. Furthermore, instructions can also be implemented as data, or may include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As those skilled in the art will recognize, details including, but not limited to, the number, structure, sequence, and organization of instructions can vary significantly without altering the underlying logic, functionality, processing, and output.

[0109] While the above discussion primarily concerns microprocessors or multi-core processors that execute software, one or more specific implementations are executed by one or more integrated circuits such as ASICs or FPGAs. In one or more specific implementations, such integrated circuits execute instructions stored on the circuit itself.

[0110] As is widely recognized, the use of personally identifiable information should comply with privacy policies and practices that are generally accepted to meet or exceed industry or governmental requirements for protecting user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly explained to users.

[0111] Specific embodiments within the scope of this disclosure may be implemented, in whole or in part, using a tangible computer-readable storage medium (or a plurality of tangible computer-readable storage media of one or more types) that encodes one or more computer-readable instructions. It should be understood that computer-executable instructions may be organized in any format, including applications, components, processes, software, software modules, and / or components.

[0112] Specific embodiments within the scope of this disclosure include computer-readable storage media encoding instructions organized as an application (e.g., application 842) that, when executed by one or more processing units, control device 840 for execution. Figure 8B Methods Figure 8C The methods and / or one or more other processes and / or methods described herein.

[0113] It should be recognized that the application of 842 ( Figure 8D The application 842 (shown in the diagram) can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets, or other applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, messaging applications, and / or map applications. In some embodiments, application 842 is an application pre-installed on device 840 at the time of purchase (e.g., a first-party application). In other embodiments, application 842 is an application provided to device 840 via operating system update files (e.g., a first-party or second-party application). In some embodiments, application 842 is an application provided via an app store. In some embodiments, the app store can be an app store pre-installed on device 840 at the time of purchase (e.g., a first-party app store). In some embodiments, the app store is a third-party app store (e.g., an app store provided by another app store, downloaded via a network, and / or read from a storage device).

[0114] refer to Figure 8B and Figure 8FApplication 842 obtains information (e.g., 822). In some embodiments, at 822, information is obtained from at least one hardware component of device 840. In some embodiments, at 822, information is obtained from at least one software module (e.g., instruction set) of device 840. In some embodiments, at 822, information is obtained from at least one hardware component external to device 840 (e.g., peripheral device, accessory device, and / or server). In some embodiments, the information obtained at 822 includes location information, time information, notification information, user information, environmental information, electronic device status information, weather information, media information, historical information, event information, hardware information, and / or motion information. In some embodiments, in response to obtaining information at 822 and / or thereafter, application 842 provides the information to the system (e.g., 824).

[0115] In some implementations, the system (e.g., Figure 8E The 850 shown is the operating system hosted on device 840. In some implementations, the system (e.g., Figure 8E 850 shown in the figure is an external device (e.g., a server, peripheral device, accessory and / or personal computing device) that includes an operating system.

[0116] refer to Figure 8C and Figure 8G Application 842 obtains information (e.g., 832). In some embodiments, the information obtained at 832 includes location information, time information, notification information, user information, environmental information, electronic device status information, weather information, media information, historical information, event information, hardware information, and / or motion information. In response to obtaining information at 832 and / or subsequently, application 842 performs an operation on the information (e.g., 834). In some embodiments, the operations performed at 834 include: providing notifications based on the information, sending messages based on the information, displaying information, controlling the user interface of a fitness application based on the information, controlling the user interface of a health application based on the information, controlling focus mode based on the information, setting reminders based on the information, adding calendar entries based on the information, and / or calling the API of system 850 based on the information.

[0117] In some implementations, execution is performed in response to a trigger. Figure 8B Methods and / or Figure 8C The method involves one or more steps. In some implementations, triggering includes event detection, notifications received from system 850, user input, and / or responses to calls to APIs provided by system 850.

[0118] In some implementations, when the instructions of application 842 are executed, the control device 840 executes them by calling an application programming interface (API) (e.g., API 852) provided by system 850. Figure 8B Methods and / or Figure 8C The method. In some implementations, application 842 performs without calling API 852. Figure 8B Methods and / or Figure 8C At least a part of the method.

[0119] In some implementation schemes, Figure 8B Methods and / or Figure 8C One or more steps of the method involve calling the API (e.g., API 852) using one or more parameters defined by the API. In some implementations, one or more parameters include constants, keys, data structures, objects, object classes, variables, data types, pointers, arrays, lists, or pointers to functions or methods and / or references to data or other items to be passed via the API in another way.

[0120] refer to Figure 8D Example 840 is shown. In some embodiments, device 840 is a personal computing device, smartphone, smartwatch, fitness tracker, head-mounted display (HMD) device, media device, public utility, speaker, television, and / or tablet computer. Device 840 includes application 842 and operating system (not shown) (e.g., Figure 8E The system 850 shown. Application 842 includes application implementation instructions 844 and API call instructions 846. System 850 includes API 852 and implementation instructions 854. It should be understood that, with Figure 8D and Figure 8E Compared to the illustrated components, device 840, application 842, and / or system 850 may include more, fewer, and / or different components.

[0121] In some implementations, application implementation instructions 844 are software modules comprising a set of one or more computer-readable instructions. In some implementations, the set of one or more instructions in instructions 844 corresponds to one or more operations performed by application 842. For example, when application 842 is a messaging application, application implementation instructions 844 may include operations for receiving and sending messages. In some implementations, application implementation instructions 844 communicate with API call instructions to access API 852 (… Figure 8E (As shown) communicates with system 850.

[0122] In some implementations, API call instruction 846 is a software module that includes a set of one or more computer-executable instructions.

[0123] In some implementations, implementing instruction 854 is a software module that includes one or more computer-executable instructions.

[0124] In some implementations, API 852 is a software module comprising a set of one or more computer-executable instructions. In some implementations, API 852 provides an interface that allows different sets of instructions (e.g., API call instruction 846) to access and / or use one or more functions, processes, procedures, data structures, classes, and / or other services provided by implementation instructions 854 of system 850. For example, API call instruction 846 can access features of implementation module 854 through one or more API calls or references exposed by API 852 (e.g., embodied by function or method calls), and can pass data and / or control information via API calls or references using one or more parameters. In some implementations, API 852 allows application 842 to use services provided by a Software Development Kit (SDK) library. In some implementations, application 842 combines calls to functions or methods provided by the SDK library and those provided by API 852, or uses data types or objects defined in the SDK library and provided by API 852. In some implementations, API call instruction 846 makes an API call via API 852 to access and use the features of implementation instruction 854 specified by API 852. In such implementations, implementation instruction 854 may return a value to API call instruction 846 via API 852 in response to the API call. This value may report to application 842 the capabilities or status of hardware components of device 840, including those capabilities or statuses related to aspects such as input capabilities and status, output capabilities and status, processing capabilities, power status, storage capacity and status, and / or communication capabilities. In some implementations, API 852 is implemented in part by firmware, microcode, or other low-level logic executed in part on hardware components.

[0125] In some implementations, API 852 allows the developer of API call instruction 846 (which may be a third-party developer) to utilize features provided by implementation instruction 854. In such implementations, one or more sets of API call instructions (e.g., including API call instruction 846) may exist to communicate with implementation instruction 854. In some implementations, API 852 allows multiple sets of API call instructions written in different programming languages ​​to communicate with implementation instruction 854 (e.g., API 852 may include features for translating calls and returns between implementation instruction 854 and API call instruction 846), and API 852 is implemented in a specific programming language. In some implementations, API call instruction 846 calls APIs from different providers, such as one set of APIs from an OS provider, another set of APIs from a plugin provider, and / or another set of APIs from another provider (e.g., a software library provider) or the creator of another set of APIs.

[0126] Examples of API 852 may include one or more of the following: pairing API (e.g., for establishing a secure connection, such as with an accessory), device detection API (e.g., for locating nearby devices, such as media devices and / or smartphones), payment API, UIKit API (e.g., for generating user interfaces), location detection API, locator API, map API, health sensor API, sensor API, messaging API, push notification API, streaming API, collaboration API, video conferencing API, app store API, advertising service API, web browser API (e.g., WebKit API), transportation API, networking API, WiFi API, Bluetooth API, NFC API, UWB API, fitness API, smart home API, contact transfer API, photo API, camera API, and / or image processing API. In some implementations, the sensor API is an API for accessing data associated with the sensors of device 840. For example, the sensor API may provide access to raw sensor data. Alternatively, the sensor API may provide data derived (and / or generated) from raw sensor data. In some implementations, sensor data includes temperature data, image data, video data, audio data, heart rate data, IMU (Inertial Measurement Unit) data, LiDAR data, location data, GPS data, and / or camera data. In some implementations, the sensors include one or more of an accelerometer, temperature sensor, infrared sensor, optical sensor, heart rate sensor, barometer, gyroscope, proximity sensor, and / or biometric sensor.

[0127] In some embodiments, implementation module 854 is a system (e.g., an operating system, a server system) software module (e.g., a set of computer-readable instructions) configured to perform operations in response to receiving an API call via API 852. In some embodiments, implementation instructions 854 are configured to provide an API response (via API 852) as a result of processing the API call. For example, implementation module 854 and API call instructions 846 may each be any of an operating system, library, device driver, API, application, or other module. It should be understood that implementation instructions 854 and API call instructions 846 may be the same or different types of software modules. In some embodiments, implementation instructions 854 are at least partially embodied in firmware, microcode, or other hardware logic.

[0128] In some implementations, implementation instruction 854 returns a value via API 852 in response to an API call from API call instruction 846. While API 852 defines the syntax and results of the API call (e.g., how to reference the API call and what the API call can do), API 852 may not reveal how implementation instruction 854 performs the function specified by the API call. Various API calls are transmitted via one or more application programming interfaces between API call instruction 846 and implementation instruction 854. Transmitting API calls may include issuing, initiating, referencing, calling, receiving, returning, and / or responding to function calls or messages. In other words, transmission may describe the action performed by API call instruction 846 or implementation instruction 854. In some implementations, function calls or other references to API 852 transmit and / or receive one or more parameters via parameter lists or other structures.

[0129] In some implementations, implementation instruction 854 provides more than one API, each API providing a different view or aspect of the functionality implemented by implementation instruction 854. For example, one API of implementation instruction 854 may provide a first set of functionality and be exposed to third-party developers, while another API of implementation instruction 854 may be hidden (e.g., not exposed) and provide a subset of the first set of functionality, and also provide another set of functionality, such as testing or debugging functionality not in the first set of functionality. In some implementations, implementation instruction 854 calls one or more other components via lower-level APIs, and is therefore both a set of API call module instructions and a set of implementation module instructions. It should be recognized that implementation instruction 854 may include additional functions, procedures, classes, data structures, and / or other features not specified through API 852 and not available for API call instruction 846. Furthermore, it should be recognized that API call instruction 846 may be on the same system as implementation instruction 854, or may be remotely located and accessed via a network using API 852. In some implementations, implementation instructions 854, API 852, and / or API call instructions 846 are stored in a machine-readable medium, which includes any means for storing information in a machine-readable (e.g., computer or other data processing system) form. For example, a machine-readable medium may include a magnetic disk, optical disk, random access memory, read-only memory, and / or flash memory devices.

[0130] In some implementations, process 600 and / or 700 ( Figure 6 , Figure 7 The application is executed on a first computer system (as described herein) via a system process (e.g., an operating system process, a server system process) that is different from one or more applications that are executed and / or installed on the first computer system.

[0131] In some implementations, process 600 and / or 700 ( Figure 6 , Figure 7 At the first computer system (as described herein), execution is performed by an application distinct from the system processes (e.g., a trusted application, such as one provided or verified by the system manufacturer). In some implementations, the application's instructions, when executed, control the execution of processes 600 and / or 700 of the first computer system by invoking an application programming interface (API) provided by the system processes. Figure 6 , Figure 7 In some implementations, the application executes at least a portion of procedures 600 and / or 700 without invoking the API.

[0132] In some implementations, the application can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets or other applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, messaging applications and / or map applications.

[0133] In some implementations, the application is an application pre-installed on the first computer system at the time of purchase (e.g., a first-party application). In some implementations, the application is an application provided to the first computer system via operating system update files (e.g., a first-party application). In some implementations, the application is an application provided via an app store. In some implementations, the app store is pre-installed on the first computer system at the time of purchase (e.g., a first-party app store) and allows the download of one or more applications. In some implementations, the app store is a third-party app store (e.g., an app store provided by another device, downloaded via a network, and / or read from a storage device). In some implementations, the application is a third-party application (e.g., an application provided by an app store, downloaded via a network, and / or read from a storage device). In some implementations, the application controls the execution of processes 600 and / or 700 on the first computer system by calling an application programming interface (API) provided by a system process using one or more parameters. Figure 6 , Figure 7 ).

[0134] In some implementations, at least one API is a software module that provides an interface (e.g., a set of computer-readable instructions) that allows different sets of instructions (e.g., API call instructions) to access and use one or more functions, methods, procedures, data structures, classes, and / or other services provided by an implementation set of instructions from a system process. The API may define one or more parameters passed between the API call instructions and the implementation instructions.

[0135] For example, the callable APIs could correspond to APIs provided by system processes to receive pointer style definitions and UI element effect definitions from the application. For instance, an application could call such an API to provide definition declarations to the system. Similarly, a system process could call such an API to retrieve definition declarations.

[0136] As described above, in some implementations, the application controls the execution of processes 600 and / or 700 of the first computer system by calling an application programming interface (API) provided by a system process using one or more parameters. Figure 6 , Figure 7 ).

[0137] In some implementations, exemplary APIs provided by system processes include one or more of the following: pairing API (e.g., for establishing a secure connection, such as with an accessory), device detection API (e.g., for locating nearby devices, such as media devices and / or smartphones), payment API, UIKit API (e.g., for generating user interfaces), location detection API, locator API, map API, health sensor API, sensor API, messaging API, push notification API, streaming API, collaboration API, video conferencing API, app store API, advertising service API, web browser API (e.g., WebKit API), transportation API, networking API, WiFi API, Bluetooth API, NFC API, UWB API, fitness API, smart home API, contact transfer API, photo API, camera API, and / or image processing API.

[0138] In some implementations, API 852 defines a first API call API call instruction 846, wherein the definition of the first API call specifies the relevant call parameters according to the variables and elements discussed above.

[0139] In some implementations, API 852 defines a first API call response that can be provided to an application by API call instruction 846, wherein the first API call response provides response communication and output corresponding to the process discussed above.

[0140] In some embodiments, the implementation instruction set is a system software module (e.g., a set of computer-readable instructions) configured to perform operations in response to receiving an API call via an API. In some embodiments, the implementation instruction set is configured to provide an API response (via an API) as a result of processing an API call. In some embodiments, the implementation instruction set is included in a device (e.g., 840) running the application. In some embodiments, the implementation instruction set is included in an electronic device separate from the device running the application.

[0141] Those skilled in the art will recognize that the various exemplary blocks, modules, elements, components, methods, and algorithms described herein can be implemented as electronic hardware, computer software, or a combination of both. To illustrate this interchangeability between hardware and software, the various exemplary blocks, modules, elements, components, methods, and algorithms have been generally described above in terms of functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in different ways for each specific application. Various components and blocks can be arranged differently (e.g., in different orders or divided in different ways) without departing from the scope of the subject matter.

[0142] It should be understood that any particular order or hierarchy of the boxes in the process disclosed in this invention is an example of the exemplary method. Based on design preferences, it should be understood that the particular order or hierarchy of the boxes in the process may be rearranged or all illustrated boxes may be executed. Any box within these boxes may be executed simultaneously. In one or more embodiments, multitasking and parallel processing may be advantageous. Furthermore, the division of the various system components in the embodiments described above should not be construed as requiring such division in all embodiments, and it should be understood that the program components (e.g., computer program products) and the system may generally be integrated together in a single software product or packaged into multiple software products.

[0143] As used in this specification and any claim of this patent application, the terms "base station," "receiver," "computer," "server," "processor," and "memory" refer to electronic devices or other technical devices. These terms exclude persons or groups of persons. For the purposes of this specification, the terms "display" or "being displayed" mean displaying on an electronic device.

[0144] As used herein, the phrase "at least one of" following a series of items separated by the terms "and" or "or" modifies the list as a whole, not each member of the list (i.e., each item). The phrase "at least one of" does not require selection of at least one of each of the listed items; rather, it allows for the inclusion of at least one of any one of the items and / or at least one of any combination of items and / or at least one of each item. For example, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" respectively refer to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0145] The predicates “configured to,” “operable to,” and “programmed to” do not imply any specific tangible or intangible modification to a particular subject matter but are intended to be used interchangeably. In one or more embodiments, a processor configured to monitor and control operations or components may also mean that the processor is programmed to monitor and control operations or that the processor is operable to monitor and control operations. Similarly, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0146] Phrases such as aspect, that aspect, on the other hand, some aspects, one or more aspects, implementation, that implementation, another implementation, some implementations, one or more implementations, implementation scheme, that implementation scheme, another implementation scheme, some implementation schemes, one or more implementation schemes, configuration, that configuration, other configuration, some configurations, one or more configurations, subject matter technology, disclosure, this disclosure, other variations thereof, etc., are for convenience only and do not imply that disclosures involving one or more such phrases are essential to the subject matter technology, nor do they imply that such disclosures apply to all configurations of the subject matter technology. Disclosures involving one or more such phrases may apply to all configurations or one or more configurations. Disclosures involving one or more such phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects, and vice versa, and this applies similarly to the other foregoing phrases.

[0147] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” or “example” is not necessarily to be construed as preferred or superior to other specific embodiments. Furthermore, the terms “comprising,” “having,” etc., as used in the specification or claims, are intended to be inclusive, similar to how the term “comprising” is interpreted when used as a transitional word in the claims.

[0148] All structural and functional equivalents of elements throughout the various aspects described herein that are known or later become apparent to those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly stated in the claims. No claim element should be interpreted in accordance with 35 U.S.SC §112(f) unless the element is expressly stated using the phrase “component for…” or, in the case of a method claim, using the phrase “step for…”.

[0149] The preceding description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, this claim is not intended to be limited to the aspects shown herein, but rather to be consistent with the language of the claim, wherein references to singular elements are not intended to mean “only one” but rather “one or more” unless specifically indicated. Unless otherwise specifically stated, the term “some” means one or more. Male pronouns (e.g., his) include female and neutral pronouns (e.g., her and its), and vice versa. Titles and subtitles (if any) are used for convenience only and do not limit the disclosure of this subject matter.

Claims

1. A method, the method comprising: At the system process of the electronic device, a definition of a first pointer style managed by the application for at least a first portion of a first user interface window for the user interface (UI) is received from an application running on the electronic device. When a user inputs an indicator pointer element in the first part of the first user interface window, the system process displays the pointer element according to the definition of the first pointer style; as well as When the user input indicates that the pointer element is in a portion of the first user interface window other than the first portion of the first user interface window, the system process displays the pointer element in a style other than the first pointer style.

2. The method of claim 1, wherein the user input corresponds to an initial interaction.

3. The method of claim 1, wherein the pointer element is displayed by the system process, without providing the application with an indication of the pointer element in the first portion of the first user interface window.

4. The method of claim 1, wherein the styles other than the first pointer style are provided to the system process by the application in a definition declaration.

5. The method according to claim 1, further comprising: The system process receives initial user input that moves the pointer element out of the first portion of the first user interface window; as well as Change the pointer style of the pointer element to a style other than the first pointer style.

6. The method of claim 5, wherein the initial user input includes at least one of gaze input indicating interaction with the first user interface window or finger gesture controlling the pointer element.

7. The method according to claim 1, further comprising: The system process receives active input from the user to provide the active input to the application; The activity input is provided to the application, and the activity input includes the position of the pointer element in the first user interface window; In response to the active input, the system process receives an updated definition of the first pointer style for the first portion of the first user interface window; as well as When the user input indicates that the pointer element is in the first part of the first user interface window, the system process displays the pointer element according to the updated definition of the first pointer style.

8. The method of claim 1, wherein the definition of the first pointer style includes a description of the first pointer style, the description including one or more of a size parameter, a color parameter, an offset parameter, or a shape parameter, wherein the definition of the first pointer style includes an indication of a bounded region corresponding to the first portion of the first user interface window, wherein the method further includes: Receives the definition of one or more additional pointer styles, where each pointer style indicates the portion of the UI to which the pointer style is applied.

9. The method of claim 1, wherein the definition of the first pointer style includes the location of the rendered output of the application, and wherein displaying the pointer element according to the definition of the first pointer style includes: The output of the rendering by the application is redirected to the position corresponding to the pointer element.

10. The method according to claim 1, further comprising: When the system process displays the pointer element according to the definition of the first pointer style, a resizing effect is applied to the first UI element that intersects with the pointer element, and the resizing effect changes the display of the first UI element from a first state to a second state.

11. An apparatus, the apparatus comprising: One or more processors; and A computer-readable medium, wherein the one or more processors are configured to perform operations including the following: The system process of the device receives from an application running on the device a definition of a resizing effect managed by the application for a first user interface (UI) element, the definition of which includes a configuration of a second state for the first UI element; When a user inputs an instruction to interact with the first UI element while it is in the first state, the system process changes the display of the first UI element from the first state to the second state of the first UI element according to the definition of the resizing effect. The system process determines the intent to provide user input information to the first UI element when it is in the second state; as well as The user input information is provided to the application, and the user input information indicates the location of the user input and the element ID of the first UI element.

12. The device of claim 11, wherein the interaction is a resizing interaction.

13. The device of claim 11, wherein the location of the user input is outside the area of ​​the first state of the first UI element and inside the area of ​​the second state of the first UI element.

14. The device of claim 11, wherein the display size of the second state is greater than the display size of the first state.

15. The device according to claim 11, wherein, The second state of the first UI element includes one or more additional UI elements.

16. The device of claim 15, wherein the operation further comprises: When user input in the first UI element indicates further instructions regarding a second UI element among the one or more additional UI elements, an effect is applied to the second UI element, wherein the effect is applied according to an effect definition received from the application.

17. The device of claim 15, wherein when the first UI element is in the first state, the one or more additional UI elements are unavailable.

18. The apparatus of claim 11, wherein the operation further comprises: The system process receives user input when the first UI element is in the second state, and the user input indicates that the position of the user input is outside the intersection with the first UI element; as well as The system process changes the display of the first UI element from the second state to the first state.

19. The apparatus of claim 11, wherein the operation further comprises: The system process provides the pointer style for the pointer element based on the position input by the user; as well as When the position input by the user changes from the first redefined region to the second predefined region, the pointer style for the pointer element is updated.

20. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform operations including: At the system process of the electronic device, a definition of a first pointer style managed by the application for at least a first portion of a first user interface window for the user interface (UI) is received from an application running on the electronic device. When a user inputs an indicator pointer element in the first part of the first user interface window, the system process displays the pointer element according to the definition of the first pointer style; as well as When the user input indicates that the pointer element is in a portion of the first user interface window other than the first portion of the first user interface window, the system process displays the pointer element in a style other than the first pointer style.

Images