Attention Detection Service

Abstract

The present disclosure relates to an attention detection service. The attention detection service can monitor various peripheral devices in the device for an indication that the user is paying attention to the device. Various clients can register for notifications of attention detection and attention loss (no longer detecting attention) events, or can poll the service for the events. If the user is not paying attention to the device, it may be possible to take various actions to allow for reduced energy consumption.

Description

[0001] This application is a divisional application of the invention patent application No. 201880036703.6, which entered the Chinese national phase with an international filing date of January 24, 2018 and a title of "Attention Detection Service". Technical Field

[0002] The embodiments described herein relate to attention detection on a mobile device. Background Art

[0003] Mobile devices have become ubiquitous, and for most mobile device users, access to their mobile devices is crucial to how they spend their time. Mobile devices can include various "smartphones" that include cellular phone functionality and general processing capabilities for executing various applications or apps on the phone. Smartphones typically can have Internet access (e.g., via a cellular network or via wifi). Other mobile devices can include tablet computers, laptop computers, etc.

[0004] Typically, instead of or in addition to operating from a fixed power source such as a wall outlet, mobile devices are designed to operate from a mobile power source such as a battery. A fixed power source typically has essentially infinite energy available, while a mobile power source may have a limited amount of stored energy before it needs to be replenished. Thus, energy is a scarce resource that is desirable to conserve.

[0005] While other types of devices (e.g., devices plugged into a wall outlet) may not have as pressing a need to conserve energy, such devices may desire to reduce energy usage for other reasons. For example, energy consumption is often associated with an increase in the heat energy that needs to be removed from the device. Thus, inefficient energy consumption can lead to higher heat energy in the system, which requires more cooling to extract the heat energy. Summary of the Invention

[0006] In one embodiment, a device can include an attention detection service. The attention detection service can monitor various peripheral devices in the device for indications that a user is attending to the device. Various clients can register for notifications of attention detection and attention loss (no longer detecting attention) events, or can poll the service for events. If the user is not attending to the device, it may be possible to take various actions to allow for reduced energy consumption. Brief Description of the Drawings

[0007] The following detailed description refers to the accompanying drawings, which are briefly described now.

[0008] Figure 1 is a block diagram of one embodiment of the system.

[0009] Figure 2 is available atFigure 1 Block diagram of one embodiment of a software component executed on the system shown.

[0010] Figure 3 Shows Figure 2 Flowchart of the operation of one embodiment of the attention detection service shown in.

[0011] Figure 4 Shows Figure 2 Flowchart of the operation of one embodiment of the attention detection service shown in response to a registration request from a client.

[0012] Figure 5 Shows Figure 2 Flowchart of the operation of one embodiment of the attention detection service shown in response to a poll from a client.

[0013] Figure 6 Shows Figure 2 Flowchart of the operation of one embodiment of the attention detection service shown in response to an attention event.

[0014] Figure 7 Shows Figure 2 Flowchart of the operation of one embodiment of the attention detection service shown in response to a timeout.

[0015] Figure 8 Block diagram of one embodiment of a computer-accessible storage medium.

[0016] Although the embodiments described in this disclosure may be subject to various modified forms and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that the drawings and the specific implementation thereof are not intended to limit the embodiments to the specific forms disclosed, but on the contrary, the present invention is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the appended claims. The headings used herein are for organizational purposes only and are not intended to limit the scope of the specification. As used throughout this application, the word "may" is used in a permissive sense (i.e., meaning having the possibility) rather than a mandatory sense (i.e., meaning must). Similarly, the words "comprising," "including" mean including but not limited to. As used herein, unless otherwise expressly stated, the terms "first," "second," etc. serve as labels for the nouns that follow and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.).

[0017] Within this disclosure, different entities (which may be variously referred to as “units,” “circuits,” other components, etc.) may be described or claimed as “configured to” perform one or more tasks or operations. This expression—[entity] [configured to [perform one or more tasks]]—is used herein to refer to a structure (i.e., a physical thing, such as an electronic circuit). More specifically, this expression is used to indicate that the structure is arranged to perform one or more tasks during operation. A structure may be described as “configured to” perform a certain task even if the structure is not currently being operated. A “clock circuit configured to generate an output clock signal” is intended to encompass, for example, a circuit that performs that function during operation, even if the circuit in question is not currently in use (e.g., the circuit is not connected to a power source). Thus, an entity described or stated as “configured to” perform a certain task refers to a physical thing, such as a device, a circuit, a memory storing program instructions executable to implement the task, etc. This phrase is not used herein to refer to intangible things. Generally, the circuits that form the structure corresponding to “configured to” may include hardware circuits. The hardware circuits may include any combination of the following: combinational logic circuits, clock storage devices (such as flip-flops, registers, latches, etc.), finite state machines, memories (such as static random access memories or embedded dynamic random access memories), custom-designed circuits, analog circuits, programmable logic arrays, etc. Similarly, for ease of description, various units / circuits / components may be described as performing one or more tasks. Such descriptions should be construed to include the phrase “configured to.”

[0018] The term “configured to” is not intended to mean “capable of being configured to.” For example, an unprogrammed FPGA should not be considered “configured to” perform a particular function, although it may be “capable of being configured to” perform that function. After appropriate programming, the FPGA may then be configured to perform the function.

[0019] A unit / circuit / component or other structure recited in the appended claims as configured to perform one or more tasks is expressly intended not to be interpreted under 35 U.S.C. § 112(f) for that claim. Thus, no claim in the present application being filed is intended to be interpreted as having a means-plus-function element. If the applicant wishes to rely on 35 U.S.C. § 112(f) during the application process, it will phrase the claim element using the “means for” [performing a function] structure.

[0020] In one embodiment, a hardware circuit according to the present disclosure can be implemented by encoding a description of the circuit in a hardware description language (HDL) such as Verilog or VHDL. The HDL description can be synthesized for a cell library designed for a given integrated circuit manufacturing technology and can be modified for timing, power, and other reasons to obtain a final design database that can be transmitted to a factory to generate masks and ultimately produce an integrated circuit. Some hardware circuits or portions thereof can also be custom designed in a schematic editor and captured into an integrated circuit design along with the synthesized circuit. The integrated circuit can include transistors and can further include other circuit elements (e.g., passive elements such as capacitors, resistors, inductors, etc.) and interconnects between the transistors and circuit elements. Some embodiments can implement multiple integrated circuits coupled together to implement the hardware circuit, and / or discrete elements can be used in some embodiments. Alternatively, the HDL design can be synthesized into a programmable logic array such as a field programmable gate array (FPGA) and implemented in the FPGA.

[0021] As used herein, the terms “based on” or “depending on” are used to describe one or more factors that affect a determination. This term does not exclude the possibility that additional factors may affect the determination. That is, the determination can be based solely on the specified factors or on the specified factors and other unspecified factors. Consider the phrase “determine A based on B”. This phrase specifies that B is a factor used to determine A or that it affects the determination of A. This phrase does not exclude the possibility that the determination of A may also be based on some other factor such as C. This phrase is also intended to cover embodiments in which A is determined based solely on B. As used herein, the phrase “based on” is synonymous with the phrase “at least partially based on”.

[0022] This specification includes references to various embodiments to indicate that the present disclosure is not intended to refer to a single specific implementation, but rather to a series of embodiments that fall within the scope of the present disclosure, including the appended claims. Specific features, structures, or characteristics can be combined in any suitable manner consistent with the present disclosure.

[0023] The present disclosure further contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. For example, in the case of using facial recognition to unlock and / or authorize a device, personal information from the user should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of those legitimate uses. Additionally, such collection should only occur after the user's informed consent. Additionally, such entities should take any necessary steps to safeguard and protect access to such personal information data and ensure that others with access to the personal information data comply with their privacy policies and procedures. Additionally, such an entity may subject itself to third-party assessments to demonstrate its compliance with widely accepted privacy policies and practices.

[0024] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, the techniques of the present invention may be configured to allow a user to select to "opt-in" or "opt-out" of participating in the collection of personal information data during a registration service. Detailed Description

[0025] Turning now to Figure 1 , which shows a block diagram of one embodiment of system 10. In the illustrated embodiment, system 10 may include a processing system 12 that is coupled to one or more peripheral devices ("peripherals") 14A - 14I. Although various peripherals are shown by way of example in Figure 1 , any collection of peripherals may be included in various embodiments, including subsets of peripherals 14A - 14I, subsets of peripherals 14A - 14I and additional peripherals, supersets of peripherals 14A - 14I and additional peripherals, and the like. Generally, system 10 may be any type of digital system. For example, system 10 may be a portable device, such as a personal digital assistant, a smart phone that integrates mobile phone features and application execution, a tablet computer, or a laptop computer. System 10 may also be a desktop computer, a stand-alone server computer, a rack-mounted computer (e.g., a server), a workstation, and the like.

[0026] The processing system 12 may include one or more processors configured to execute instructions defined in an instruction set architecture implemented by the system 10. The instruction set architecture may define the instructions, including their encoding, operands, and operations to be performed in response to executing each instruction. The operands may include data stored in one or more sets of registers implemented in the processor, data stored in a memory location identified by an address formed from the operand data and / or other data, etc. The instruction set architecture may further specify interrupts, exceptions, and other operational factors not directly related to instruction execution. The instruction set architecture may define the processor state, including the aforementioned operand registers and various configuration and control registers in which various processor modes may be programmed.

[0027] Generally, the processor may have any microarchitecture. For example, the microarchitecture may be scalar or superscalar, speculative or non-speculative, sequential or out-of-order, pipelined or superpipelined, etc. Various performance enhancement features such as caches, branch prediction, register renaming, centralized or distributed scheduling, reorder buffers, memory buffers, etc. may be employed in various embodiments. In some embodiments, in addition to any of the above features, the processor may also employ microcode. In some embodiments, the processor may include multithreading acceleration capabilities. The processor may be multi-core or single-core and may include an integrated processor with other hardware in a system-on-chip (SOC) configuration, or may be a stand-alone multi-core or single-core integrated circuit (IC). The processor may be implemented in a multi-chip module with other chips. The SOC / IC including the processor may be further included in a stacked package or a stacked die configuration with other chips.

[0028] In some embodiments, the processing system 12 may include additional circuitry, such as a memory controller. The memory controller may interface to the main memory system of the system 10, which may be any type of memory. For example, the memory may be various forms of random access memory (RAM), such as dynamic RAM (DRAM), double data rate DRAM (DDR DRAM), including various mobile DDR (mDDR) and / or low power DDR (LP DDR) DRAM. The memory may also include various non-volatile RAM, such as flash memory.

[0029] The processing system 12 may further include various other circuits, such as a power management circuit, additional peripherals, a peripheral interface circuit for a peripheral interface (including the peripheral interface to which one or more of the peripherals 14A - 14I are coupled), and the like. The processing system 12 may include circuitry for processing data received from one or more of the peripherals 14A - 14I (e.g., one or more image signal processors (ISPs) for processing images from the camera 14B). In one embodiment, the processing system 12 may be or may include a system - on - a - chip (SOC) that includes a processor and one or more other components (including the examples above).

[0030] The peripherals 14A - 14I may include various peripherals that may provide data indicating that a user of the system 10 is paying attention to the system 10. If a user is interacting with or making contact with the system 10, the user may be paying attention to the system 10. The interaction may be visual (e.g., the user is looking at the system 10 and / or at the display screen of the system 10), auditory (e.g., speaking to a virtual assistant or other voice - recognition component on the system and / or using a pre - determined sound as a trigger to indicate attention), and / or the interaction may be physical (e.g., using one or more input / output interfaces of the system 10 and / or moving a portable version of the system 10 in a manner indicative of use). As previously mentioned, any set of peripherals may be used in various embodiments.

[0031] The touch screen 14A may be a peripheral that detects a user's touch on the screen surface. The screen may be a display screen, or it may be a touch area (e.g., a touchpad) on a device such as a laptop computer where the touch is interpreted as a mouse movement. The screen may also be a touch - enabled function row on a keyboard. In some embodiments, touch sensing may include sensing direct contact with the touch screen 14A, or it may also include near - touch sensing (e.g., the user's contact, such as a finger, within a defined distance from the touch screen 14A). In addition to detecting the presence or absence of a touch, touch sensing may further include differentiating various contact intensities on the touch screen 14A. That is, in some embodiments, the touch screen 14A may include tactile sensing, similar to the tactile sensor 14C discussed in more detail below.

[0032] System 10 may include one or more cameras 14B. The cameras 14B may capture any kind of images of the environment in front of the cameras 14B. For example, the cameras 14B may include one or more visible light cameras that capture images in the visible spectrum, such as charge-coupled device (CCD) cameras, optical cameras, etc. In some embodiments, the cameras 14B may further include one or more IR cameras that capture images in the infrared (IR) spectrum. The IR images may be used for, for example, face detection. The IR cameras may further include structured light cameras that project dots on the user's face for depth detection.

[0033] The tactile sensor 14C may also be a type of touch sensor that senses the intensity of contact. For example, the tactile sensor 14C may be used for one or more buttons on the system 10 that can be pressed by the user, and the sensed contact intensity may be used to trigger different events within the system 10 (e.g., a first contact intensity on the button may trigger one event, while a second contact intensity may trigger another event). In one embodiment, the tactile sensor 14C may include a force sensor. Alternatively, in another embodiment, the tactile sensor 14C may include a pressure sensor.

[0034] The keyboard 14D may be a physical keyboard on which the user types. For example, the keyboard 14D may be an integrated keyboard in a laptop computer, or if the system 10 is a laptop computer, a desktop computer, a server, etc., it may be a keyboard connected to an input / output port of the system 10 such as a universal serial bus (USB) port, a serial port, etc. The keyboard 14D may also be wirelessly connected to the system 10 via Bluetooth or other wireless interconnections. Some devices may support a virtual keyboard displayed on a part (or all) of the touch screen 14A (e.g., PDAs, smartphones, and tablet computers), and thus the keyboard 14D may not be included on those devices (or may be included by connecting to a port if needed).

[0035] The external connection sensor 14E may detect that a device external to the system 10 has been connected to a port or an external interface of the system 10. For example, if an external display is connected to a video port (e.g., a video graphics adapter (VGA) or a high-definition media interface (HDMI) port), the connection may be sensed.

[0036] The mouse / pointing device 14F may be any kind of input device that provides for the movement of a cursor or other visual indication on the screen. Thus, the mouse / pointing device 14F may include a mouse, a trackball device, a touchpad, etc. In some embodiments, the touch screen 14A may also be a pointing device. The mouse / pointing device 14F may be connected to the system 10 using various wired or wireless connections, similar to the discussion of the keyboard 14D above.

[0037] The button sensor 14G can sense user presses on buttons on the system 10. For example, the button sensor 14G on a smart phone or tablet can detect a power on / off button, volume control buttons, a mute switch / button, etc. In one embodiment, the button sensor 14G can be used for buttons that do not have tactile sensing via the tactile sensor 14G.

[0038] The device motion sensor 14H can be various sensors that detect the motion of the system 10 within its environment. For example, the device motion sensor 14H can include one or more of any of the following: a gyroscope, an accelerometer, a magnetometer, a photodetector, a proximity sensor, a temperature sensor, etc. The device motion sensor 14H can be used to detect the orientation and / or movement of the system 10 within the surrounding environment of the system 10. The device motion sensor 14H can also be used to detect properties of the environment (e.g., temperature, light, etc.).

[0039] The biometric sensor 14I can detect biometric data from a user (e.g., fingerprint, iris scan, etc.). Biometric data can be sufficiently diverse in a population such that biometric data can be highly reliable in identifying a user. In one embodiment, face detection via the camera 14B can also be a form of biometric sensing.

[0040] As described above, in some embodiments, various peripheral devices 14A - 14I can be used to detect whether a user is paying attention to the system 10. If a user is touching the screen (touch screen 14A), then the user is likely physically interacting with the system 10 and thus paying attention. In some cases, the touch screen 14A may report touches that are not from the user (e.g., when the system 10 is in the user's pocket, the pocket can cause touch events to occur). Thus, in some embodiments, data from the touch screen 14A can be filtered or verified by data from other sensors. Images captured by the camera 14B and processed by the ISP in the processing system 12 can be used to detect a face, and based on the face and specifically on the position of the eyes, the processing system 12 can determine whether the user is looking at the system 10. If the user is looking at the system 10, then the user is likely paying attention to the system 10. If the tactile sensor 14C detects a sufficient contact intensity, then the user is likely interacting with the system 10 and thus may be paying attention. Similarly, if the keyboard 14D, mouse / pointing device 14F, button sensor 14H, and / or biometric sensor 14I detect user input, then the user is likely paying attention. If the external connection sensor 14E detects that an external monitor is connected and is displaying output from the system 10, then the user is likely paying attention to the monitor and thus paying attention to the system 10. If the device motion sensor 14H detects movement and / or orientation indicating that the user is manipulating the system 10, then the user is likely paying attention to the system 10.

[0041] The combination of data from the aforementioned peripheral devices 14A - 14I can be used to detect attention, and / or data from the peripheral devices 14A - 14I can be filtered with one or more other peripheral devices 14A - 14I. Additionally, in some embodiments, one or more of the peripheral devices 14A - 14I may be more "expensive" to operate. For example, one or more of the peripheral devices 14A - 14I may consume significantly more power than other peripheral devices 14A - 14I. On a mobile device operating from a mobile power source (e.g., a battery), the cost of higher power consumption can be, for example, a reduction in battery life before recharging is required. The less expensive peripheral devices 14A - 14I can be used as proxies for detecting attention, and when such peripheral devices 14A - 14I do not detect attention over a certain period of time, the more expensive peripheral devices 14A - 14I can be used to verify that the user is not paying attention.

[0042] Various sensors have been described herein as contact intensity sensors, force sensors, pressure sensors, or touch sensors (e.g., the tactile sensor 14C and / or the touchscreen or other touch device 14A). A contact intensity sensor can include, for example, one or more piezoresistive strain gauges, capacitive force sensors, electro - mechanical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch - sensitive surfaces, or other intensity sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch - sensitive surface). The contact intensity sensor receives contact intensity information (e.g., force information, pressure information, or a surrogate for force or pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed or adjacent to a touch - sensitive surface (e.g., the touchscreen / device 14A). In some embodiments, at least one contact intensity sensor is located on the rear of the device, opposite the touchscreen / device 14A located on the front of the device. Force sensors and pressure sensors can be examples of contact intensity sensors.

[0043] As used herein, the “intensity” of a contact on a touch-sensitive surface refers to the force or pressure of a contact (e.g., finger contact or stylus contact) on the touch-sensitive surface (force per unit area), or to a surrogate for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact can have a range of values that includes at least four different values and more typically includes hundreds of different values (e.g., at least 256). The intensity of a contact can be determined (or measured) using a variety of methods and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensitive surface can measure the force at various points on the touch-sensitive surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average or summation) to determine an estimated contact force. Similarly, a pressure-sensitive tip of a stylus can determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change can be used as a surrogate for the force or pressure of a contact on the touch-sensitive surface. In some embodiments, the surrogate measurement of contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measurement). In some embodiments, the surrogate measurement of contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure or a force threshold measured in units of force). Using the intensity of a contact as an attribute of user input allows a user to access additional device functions that the user might not otherwise be able to access on a smaller device that has limited real estate for displaying affordances and / or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).

[0044] Figure 2 is a block diagram of an embodiment of a software component that can be executed in the processing system 12. In the illustrated embodiment, there can be one or more clients 20A - 20N, an attention detection service 22, an event processor 24, and one or more device drivers 26.

[0045] The clients 20A - 20N can be software code sequences (e.g., processes, threads, or applications), which can modify their behavior based on whether the user is paying attention to the system 10. For example, the clients 20A - 20N responsible for driving frames to the display can determine whether the user is paying attention, and if not, can dim the display to save power and / or wear on the display. If the user continues to lack attention, the clients 20A - 20N can turn off the display. As another example, the clients 20A - 20N can drive a ringtone on a smart phone when a call comes in. If the user is paying attention to the device, the clients 20A - 20N can lower the volume of the ringtone (or perhaps even mute the ringtone) so that the user is not overwhelmed by the sound when deciding whether to answer the call. As yet another example, the clients 20A - 20N can be a voice recognition interface for the system 10. If the user utters a key phrase, the voice recognition interface can wake up and start interpreting the user's speech as a command / request to the system 10. If the user is paying attention to the system 10, the voice recognition interface can wake up and can display a notification that the voice recognition interface is ready to accept commands. The user can skip the key phrase and start with a request / command. As yet another example, the clients 20A - 20N that perform power management on the system 10 can determine whether the user is not paying attention to the system 10 and can reduce the performance level of one or more components of the system 10 to reduce power consumption. For example, the processor in the processing system 12 can lower its performance state, allowing for a lower operating voltage and frequency. Or, the power manager can cause the graphics adapter to switch the screen resolution to a lower resolution (or can switch to a lower power, lower performance adapter).

[0046] In one embodiment, the attention detection service 22 can provide an application programming interface (API) to the clients 20A - 20N to allow the clients 20A - 20N to detect user attention. The attention detection service 22 can support polling clients and timeout clients.

[0047] The polling client can transmit a request to the attention detection service 22 to determine whether the user is paying attention at the current time. The current time can be the time when the polling request is transmitted to the attention detection service 22. In one embodiment, the polling client can also provide a polling duration that indicates how long after the current time the polling client will be notified of a change in the user's attention. In one embodiment, the polling client can provide a fail tolerance that indicates whether a newly acquired attention sample should be acceptable for a poll that responds to the attention at the current time and the maximum age of the newly acquired attention sample. In the above example, the ringtone driver can be the polling client that polls when a call is received. The polling duration can be the amount of time the ringtone driver should transmit the ringtone before going to voicemail. In one embodiment, the polling client can optionally specify the attention timeout length discussed below, but can actively poll for a portion of the continuous attention timeout interval. For example, the polling client can specify a 1-minute attention timeout length, but can specify to actively poll for the last 30 seconds of that interval. Events from event-driven sensors (e.g., touch sensors, user input / output sensors such as keyboard or button presses, etc.) detected during the attention timeout interval can cause the timeout interval to be reset, as described below. Passive sensors (which are activated and polled for status, such as camera 14B) can be polled during the active polling interval, and if an event is detected from the passive sensor, the active polling interval can be reset.

[0048] The timeout client can register with the attention detection service 22 and can specify a maximum sample length and an attention timeout length. The maximum sample length can specify the maximum amount of time that is acceptable without detecting whether the user is paying attention to the system 10. The various low-power peripheral devices 14A - 14I can provide samples that can be interpreted for attention or non-attention. If a sample is received, the sample timeout can be reset to the maximum sample length. If a sample is not received and the sample timeout expires, the high-power peripheral devices 14A - 14I can be used to detect attention. In this embodiment, for example, the camera 14B and the ISP within the processing system 12 can be high-power peripheral devices. If the sample timeout expires, the camera 14B and the ISP can be activated and can attempt to detect the user's face and the direction the user is looking. If the user appears to be looking at the display of the system 10, the user can be considered to be paying attention to the system 10. If the user does not appear in the camera, or the user appears to be looking away from the system 10, the user can be considered not to be paying attention. If the user is paying attention, the sample timeout can be re-initialized to the maximum sample length. If the user is not paying attention and the attention timeout has expired for one or more clients, the clients can be notified that the user's attention has been lost. In the above example, the speech recognition interface and the display driver can be the timeout clients.

[0049] In Figure 2 , the client 20A can be a timeout client and the client 20N can be a polling client. Thus, the client 20A can register for the attention detection service 22 and can receive attention start (AttnStart) and attention lost (AttnLost) indications from the attention detection service 22. The AttnStart indication can be transmitted whenever it is detected by the attention detection service 22. The AttnLost indication can be transmitted when detected due to attention timeout. The client 20N can transmit a polling request when it is necessary to determine whether the user is paying attention. If the user is currently paying attention to the system 10, the attention detection service 22 can return AttnStart in response to the polling request. Otherwise, if attention is detected during the polling duration, the attention detection service 22 can return AttnStart. If attention is lost after AttnStart is transmitted and is still within the polling duration, the attention detection service can return AttnLost. Any combination of timeout clients and polling clients can be supported. In one embodiment, the polling client can optionally request an initial attention state when triggering a polling interval. In this case, the attention detection service 22 can scan for face presence (via the camera 14B) in response to the request of the polling client and respond with AttnStart if a face is present, or return an indication that the polling duration has been triggered if no face is currently present. If AttnStart is detected, the scan of the face can stop.

[0050] In one embodiment, the register and poll calls to the attention detection service 22 can be APIs to the attention detection service 22, and the attention detection service 22 can return AttnStart and AttnLost via callbacks to the clients 20A - 20N.

[0051] The event processor 24 can receive various events from the device driver 26, which can control the peripheral devices 14A - 14I. The events can include events that can indicate that the user is paying attention to the system 10. The event processor 24 can transmit these events to the attention detection service 22. The event processor 24 can also process other events and send various events (including one or more of the events provided to the attention detection service 22) to other services. In one embodiment, if the attention detection service 22 detects that the user is not paying attention while the attention detection service 22 is scanning the face (via a peripheral device other than the camera 14B), the attention detection service 22 can stop scanning the face. Subsequent interaction of the user with the system 10 can result in an AttnStart event or can cause the attention detection service 22 to start scanning the face using the camera 14B.

[0052] Now turning to Figure 3 FIG. 3, which shows a flow chart of the general operation of an exemplary attention detection service 22. However, for ease of understanding, the block diagram is shown in a particular order, and other orders may also be used. The attention detection service 22 may include instructions that, when executed in the system 10 (and more specifically, in one embodiment, by the processing system 12), may implement Figure 3 the operations shown. Thus, when stored in memory and executed, the attention detection service 22 may be configured to implement Figure 3 the operations shown.

[0053] The attention detection service 22 may detect a registration request and may register the client that sent the registration request (decision block 100, "yes" branch and block 102). Generally, registering a client may include capturing a client identifier or other mechanism for communicating with the client (e.g., a callback address), a set of events that the client is interested in (e.g., events that are interpreted by the client as indicating user attention), and potentially other parameters such as an attention loss timeout, a sample length timeout, a polling duration for polling the client, etc. The attention detection service 22 may monitor the peripheral devices 14A-14I for events (block 104); when an event is detected, send an indication of attention to the clients that have registered to be interested in those events (block 106); and send an indication of attention loss when a timeout occurs without any events being detected within the timeout period (block 108).

[0054] Although Figure 3 the attention detection service 22 is illustrated as executing in a loop, the execution may actually be thread-based, and the threads may be executed when triggered. For example, registration may be triggered in response to a registration request, monitoring of the peripheral devices may be triggered in response to an event or a sample timeout, and the communication of attention and attention loss may be triggered by events and timeouts.

[0055] Now turning to Figure 4 FIG. 4, which shows a flow chart of the operation of an exemplary attention detection service 22 in response to registration requests from timeout clients 20A-20N. Figure 4 In one embodiment, a portion of block 102 in Figure 3 may be shown. However, for ease of understanding, the block diagram is shown in a particular order, and other orders may also be used. The attention detection service 22 may include instructions that, when executed in the system 10 (and more specifically, in one embodiment, by the processing system 12), may implement Figure 4 the operations shown. Thus, when stored in memory and executed, the attention detection service 22 may be configured to implement Figure 4 the operations shown.

[0056] The registration request may include a sample length timeout, one or more attention loss timeouts, and a set of events that the clients 20A - 20N are interested in, and the attention detection service may record the data of the clients (block 30). The attention loss timeout can be a set such that an attention loss event can have multiple levels of timeout. For example, for screen dimming and screen off performed by the display controller, screen dimming can be the first (shortest) timeout in this set, and screen off can be a longer timeout. The attention loss timeout can be represented as a set, so each client can specify the number of timeouts. That is, independent of the other clients 20A - 20N, each client 20A - 20N can determine how many attention timeouts it will have. The set of events is drawn from the events that the attention detection service 22 supports as indicating attention to the system. The clients 20A - 20N can select all of these events or any subset as needed. For example, the events can be specified as a bitmask, where each bit corresponds to one of the events. The bit can be set to include the event in the events for the client 20A - 20N, or can be cleared to exclude the event. Other implementations can use the opposite encoding or any other encoding as needed. The events can include touches on the touchscreen 14A, face detection via the camera 14B (and ISP), touches on the tactile sensor 14C (and contact intensity), keyboard activity on the keyboard 14D, external connections sensed by the external connection sensor 14E, mouse / pointing device activity on the mouse / pointing device 14F, button presses on the button sensor 14G, data from the device dynamics sensor 14H, and activity on the biometric sensor 14I.

[0057] The attention detection service 22 can maintain the current timeout as the minimum of the sample length timeouts from the various timeout clients 20A - 20N. If the sample length timeout specified by the request is less than the current timeout (decision block 32, "yes" branch), then the attention detection service 22 can update the current timeout to be equal to the sample length timeout specified by the registration request (block 34). If the current sample corresponding to at least one registered event indicates user attention (decision block 36, "yes" branch), then the attention detection service 22 can transmit AttnStart to the registration client (block 38).

[0058] Now turning to Figure 5 , which shows a flowchart illustrating the operation of one implementation of the attention detection service 22 in response to a polling request from the polling clients 20A - 20N. Figure 5 In one implementation, it can be shown Figure 3a portion of the box 102 in. However, for ease of understanding, the block diagrams are shown in a specific order, and other orders may also be used. The attention detection service 22 may include instructions that, when executed in the system 10 (and more specifically, in one embodiment, by the processing system 12), may implement Figure 5 the operations shown. Thus, when stored in the memory and executed, the attention detection service 22 may be configured to implement Figure 5 the operations shown.

[0059] The polling request may include a polling duration and a set of events that the polling client is interested in (e.g., the event mask as described above). Additionally, in some embodiments, the polling request may include a sample length timeout and one or more attention loss timeouts. The attention detection service 22 may record the polling duration, sample length timeout, attention loss timeout, and the set of events of the polling client (block 40). Additionally, if the sample length timeout in the polling request is shorter than the current timeout, the attention detection service 22 may update the current timeout. That is, block 40 may be similar to Figure 4 blocks 30, 32, and 34 in. If the polling client allows a fail tolerance, the attention detection service 22 may determine whether one or more of the most recent events in the events that the polling client has registered (in its event mask) have occurred within the fail tolerance of the current time (decision block 42). For example, if no fail tolerance is allowed, the fail tolerance may be zero. If the fail tolerance is non-zero, the fail tolerance may be subtracted from the current time to be compared with the timestamp of the most recent event. If at least one event has occurred within the fail tolerance (decision block 42, "yes" branch), the attention detection service 22 may transmit AttnStart to the polling client (block 44). As described above, in some embodiments, the polling request may include a request for an initial state. If so, the attention detection service 22 may initiate a scan of the face via the camera 14B if AttnStart has not been detected yet, and may return an indication that the scan has started.

[0060] Although Figure 5 the flowchart shown in shows no communication with the polling client when the user is not currently paying attention to the system (decision block 42, "no" branch), other embodiments may send an AttnLost message to explicitly indicate that the user is not paying attention. Other embodiments may define a third message to indicate the lack of current attention.

[0061] Figure 6 is a flowchart showing the operations of one embodiment of the attention detection service 22 in response to an event signal issued by the event processor 24. Figure 6Examples of embodiments that may be part of blocks 104 and 106 in one embodiment. However, for ease of understanding, the block diagrams are shown in a particular order, and other orders may also be used. The attention detection service 22 may include instructions that, when executed in system 10 (and more specifically, in one embodiment, by processing system 12), may implement Figure 6 the operations shown. Thus, when stored in memory and executed, the attention detection service 22 may be configured to implement Figure 6 the operations shown.

[0062] The attention detection service 22 may select a client from the active clients (block 50). The active clients may include timeout clients that have registered with the attention detection service 22 or polling clients that have an active polling duration pending. If the selected client is interested in the event, the event may have been registered for the selected client. If so (decision block 52, "yes" branch), and the client has not received an AttnStart message since the most recent AttnLost message, or if the client has never received an AttnStart since registration (decision block 54, "no" branch), then the attention detection service 22 may send an AttnStart signal to the selected client (block 66). Additionally, if the event is registered for the selected client (decision block 52, "yes" branch and decision block 54, either path), then the attention detection service 22 may reset the timeout (sample length and one or more attention loss timeouts) of the client to its initial value because the event registered by the client has been received (block 58).

[0063] In the case where the event is not registered for the client (decision block 52, "no" branch), there is no signal sent to the client and the timeout of the client is not reset. If there are more active clients to process (decision block 60, "yes" branch), then the process represented by blocks 50, 52, 54, 56, and 58 may be repeated.

[0064] It should be noted that Figure 6 the illustration in Figure 6The updated client is shown. Additionally, the transmission of the AttnStart message can be delayed until the processing is complete for each client because the message is called back to the client and can cause exiting from the attention detection service 22 for a period of time if one of the clients is scheduled to a processor that is executing the attention detection service 22.

[0065] Figure 7 is a flowchart showing the operation of one embodiment of the attention detection service 22 in response to the expiration of the current timeout maintained by the attention detection service 22. Figure 7 Examples of embodiments that can be part of blocks 104, 106, and 108 in one embodiment are shown. However, for ease of understanding, the block diagrams are shown in a specific order, and other orders can also be used. The attention detection service 22 can include instructions that, when executed in the system 10 (and more specifically, in one embodiment, by the processing system 12), can implement Figure 7 the operations shown. Thus, when stored in memory and executed, the attention detection service 22 can be configured to implement Figure 7 the operations shown.

[0066] The attention detection service 22 can sample any high-cost sensors and / or passive sensors (if any) (block 70). High-cost sensors can be sensors that are not continuously turned on due to their cost (e.g., energy cost). For example, face detection performed by the camera 14B can be a high-cost sensor. Passive sensors can be sensors that can collect data related to attention but do not emit event signals on their own. Such sensors can be read by the attention detection service 22 when the current timeout expires. If the high-cost and / or passive sensors indicate an attention event (decision block 72, "yes" branch), the event can be processed (block 74) as an event (e.g., in the Figure 6 manner).

[0067] The attention detection service 22 may select an active client (block 76). In the case of event detection as shown in block 72, an active client that is not registered for the detected event is selected because other clients have been processed for the detected event. The attention detection service 22 may adjust the timeout and / or duration associated with the selected client (block 78). For example, the timeout may be adjusted by subtracting the current timeout and saturating at zero. Alternatively, the timeout that initially starts at zero may be added to the current timeout and saturated at the timeout specified by the client. If one of the attention loss timeouts associated with the client has expired (decision block 80, "yes" branch), the attention detection service 22 may send an AttnLost signal to the client (block 82). Additionally, if the client is a polling client and the duration has expired (decision block 84, "yes" branch), the attention detection service 22 may unregister the client because the polling is complete (block 86). If there are more clients still to be processed (decision block 88, "yes" branch), the process represented by blocks 76, 78, 80, 82, 84, and 86 may be repeated for another client.

[0068] Similar to the discussion above for Figure 6 , Figure 7 the illustration in is merely one mechanism for handling timeouts and may only illustrate the actions taken for a given client. There may be many other specific implementations. For example, the timeout / duration for each client not processed by the detected event in blocks 72 and 74 may (in one implementation) be updated in a single operation rather than iteratively on a client-by-client basis. The processing represented by blocks 76, 78, 80, 82, 84, and 86 may be performed in parallel for various clients. Additionally, the transmission of the AttnLost message may be delayed until the processing is complete for each client because the message is called back to the client and may cause the attention detection service 22 to exit for a period of time if one of the clients is scheduled on the processor executing the attention detection service 22.

[0069] In one embodiment, access to the attention detection service 22 can be controlled. For example, clients 20A - 20N may be required to have authorization to the attention detection service 22 in order to use the APIs in the attention detection service 22. The authorization can be the right to be granted to various code sequences based on the level of trust that system 10 has in the code sequences. For example, a client authorized to access the attention detection service 22 can be a client that is part of the operating system in system 10, or a client produced by the same company that produces the attention detection service 22. In one embodiment, the company can carefully audit certain clients and then grant them authorization to the attention detection service 22. In one embodiment, there can be multiple authorization levels. The most trusted clients can have free access to the attention detection service 22. Clients with lower credibility can have restricted access rights (e.g., a minimum timeout can be enforced, the frequency of allowing polling can be enforced, etc.).

[0070] It should be noted that in some embodiments, events from a given peripheral device 14A - 14I can be filtered by events from other peripheral devices 14A - 14I, or can be used to filter events from other peripheral devices 14A - 14I. For example, if a proximity sensor detects a touch on the touchscreen 14A indicating that an object is adjacent to system 10 (e.g., a pocket or the user's head), then the touch can be a false touch that does not indicate attention. Thus, the proximity sensor output can be used to filter touch detections.

[0071] Figure 8FIG. 0 is a block diagram of one implementation of a computer-accessible storage medium 200. Generally speaking, a computer-accessible storage medium can include any storage medium that can be accessed by a computer during use to provide instructions and / or data to the computer. For example, a computer-accessible storage medium can include storage media such as magnetic or optical media, e.g., disks (fixed or removable), tapes, CD-ROMs, DVD-ROMs, CD-Rs, CD-RWs, DVD-Rs, DVD-RWs, or Blu-ray. The storage medium can further include volatile or non-volatile memory media, such as RAM (e.g., synchronous dynamic RAM (SDRAM), Rambus DRAM (RDRAM), static RAM (SRAM), etc.), ROM, or flash memory. The storage medium can be physically included in the computer to which the storage medium provides the instructions / data. Alternatively, the storage medium can be connected to the computer. For example, the storage medium can be connected to the computer via a network or a wireless link such as a network-attached storage device. The storage medium can be connected via a peripheral interface such as a universal serial bus (USB). Generally, the computer-accessible storage medium 200 can store data in a non-transitory manner, where non-transitory can mean not transmitting instructions / data via a signal. For example, a non-transitory storage device can be volatile (and may lose the stored instructions / data in response to a power outage) or non-volatile.

[0072] Figure 8 The computer-accessible storage medium 200 in FIG. 4 can store code that forms clients 20A-20N, an attention detection service 22, an event processor 24, and / or a device driver 26. The clients 20A-20N, the attention detection service 22, the event processor 24, and / or the device driver 26 can include instructions that, when executed, implement the operations described above for these components. Specifically, the attention detection service 22 can include instructions that, when executed on the system 10, implement Figure 3 - 7 the operations. A carrier medium can include a computer-accessible storage medium and a transmission medium such as a wired or wireless transmission.

[0073] Once the above disclosure is fully understood, many variations and modifications will become apparent to those skilled in the art. The present invention is intended that the following claims be construed to cover all such variations and modifications.

Claims

1. A method, comprising: Monitoring a plurality of peripheral devices in a system for a plurality of events indicative of a potential connection of a user to the system, wherein the plurality of peripheral devices includes a first one or more peripheral devices and a second one or more peripheral devices, and wherein each of the first one or more peripheral devices consumes more power during use than each of the second one or more peripheral devices, and wherein, during a sample length timeout interval, the first one or more peripheral devices are not sampled; In response to a first event among the plurality of events from one of the second one or more peripheral devices, sending an indication of a potential user connection to one or more of a plurality of clients capable of executing on the system; In response to the sample length timeout interval expiring without detecting an event from the second one or more peripheral devices, sampling the first one or more peripheral devices; Detecting, during the sampling, a third event from one of the first one or more peripheral devices indicative of a potential user connection; and In response to detecting the third event indicative of a potential user connection, sending an indication of a potential user connection to the one or more clients.

2. The method according to claim 1, wherein the first one or more peripheral devices include a camera.

3. The method according to claim 2, wherein the camera is configured to detect a face of the user.

4. The method according to claim 1, wherein the first one or more peripheral devices include one or more passive sensors.

5. The method according to claim 1, further comprising: In response to a first event among the plurality of events from one of the second one or more peripheral devices, updating a current timeout to a sample length timeout.

6. The method according to claim 1, further comprising: In response to not detecting the third event, updating an attention loss timeout.

7. The method according to claim 6, further comprising: Detecting an expiration of an attention loss timeout associated with a first client among the one or more clients; And Sending an indication of a lost connection to the first client.

8. The method according to claim 1, further comprising: Receiving a polling request from a second client among the one or more clients, wherein the polling request specifies a current connection indication and a polling duration during which a user connection will be reported to the second client; In response to a third event among the plurality of events, detecting that the user is currently connecting to the system; In response to detecting that the user is currently paying attention, sending an indication of the attention to the second client.

9. The method according to claim 8, further comprising: Detecting a second event among the plurality of events during the polling duration, wherein the second event is specified in the polling request as an event among the plurality of events that is interpreted by the second client as indicating a connection; And In response to detecting the second event and not detecting that the user is currently establishing contact with the system when the polling request is received, send the indication of the attention to the second client.

10. The method according to claim 8, further comprising: Detecting a timeout during a period after receiving the polling request when the plurality of events are not detected during the polling duration; And In response to detecting the timeout, sending an indication of lost contact establishment to the second client.

11. The method according to claim 8, further comprising: Detecting that the polling duration has expired; And In response to detecting that the polling duration has expired, interrupting the sending of the indication of contact establishment and the indication of lost contact establishment to the second client.

12. The method according to claim 8, wherein the polling request further comprises an active polling interval, and the method further comprises polling the plurality of events during the active polling interval.

13. A computer-accessible storage medium comprising a plurality of instructions which, when executed in a system, implement the method according to any one of claims 1-12.

14. A system, comprising: A plurality of peripheral devices; And A processing system coupled to the plurality of peripheral devices, wherein the processing system includes one or more processors, the one or more processors being coupled to a memory system storing a plurality of instructions, and wherein the plurality of instructions, when executed by the one or more processors, implement the method according to any one of claims 1-12.

Images