Attention-based notification

Abstract

To provide attention-based notifications.SOLUTION: A system may present notifications to an occupant of a vehicle and determine whether the occupant has given attention to the notification. The system oscillates a visual notification at an output frequency. The system captures images of an eye of the occupant to determine an eye pupil oscillation frequency associated with the occupant. The system compares the eye pupil oscillation frequency to the output frequency. The system determines whether attention was given to the notification based on whether the output frequency and the eye pupil oscillation frequency match. If the system determines that attention was not given, the system may cause performance of one or more follow-up operations.SELECTED DRAWING: Figure 4

Description

Background Art

[0001] Fields of Various Embodiments Various embodiments generally relate to device notifications, and more specifically, to attention-based notifications.

[0002] Description of Related Art During operation, a vehicle can distribute different information to vehicle occupants such as drivers. The information can include information about the state of the vehicle and / or its internal subsystems, information about the environment around the vehicle, information about in-vehicle device operations or entertainment, and the like. The vehicle can distribute the information to the occupants visually, audibly, or tactilely.

[0003] Conventional approaches for visual information distribution by vehicles typically include lighting indicators and / or displaying warnings or notifications on a display. The disadvantage of these approaches is that the vehicle does not recognize whether the occupant has actually directed any attention to the indicator or notification unless the occupant interacts with the indicator or notification. For example, a notification can prompt the occupant to provide a yes / no response input and can remain displayed until the occupant provides the input. As another example, the notification can be displayed indefinitely unless the occupant closes it. The forced interaction with the indicator or notification can distract the occupant from more important tasks such as operating the vehicle or taking care of another vehicle occupant.

[0004] Attempts to solve this disadvantage involve detecting the direction of the occupant's line of sight. For example, the vehicle can include a camera system that tracks the occupant's eyes. When the line of sight aligns with a notification, the occupant is determined to have directed attention to the notification. The disadvantage of this approach is that the line of sight is an unreliable criterion for determining attention. Even if a notification is in the occupant's focus view, the occupant may not direct attention to the notification. For example, the occupant may be daydreaming and thus not directing attention to a notification in the focus view. As another example, there may be multiple notifications in the focus view. It can be difficult to determine that attention has been directed to each of the multiple notifications based on the line of sight.

[0005] As described above, what is needed is a more effective technique for determining that attention has been directed to a notification.

Summary of the Invention

Means for Solving the Problems

[0006] One or more embodiments describe a computer-implemented method that includes varying a first visual stimulus at a first frequency, determining a second frequency related to user characteristics, and performing one or more operations based on a determination that the second frequency does not match the first frequency.

[0007] Further embodiments provide, among other things, a system and one or more computer-readable storage media configured to implement the above method.

[0008] At least one advantage and technical improvement of the disclosed techniques is that a computing system (e.g., a vehicle computing device) can determine the attention directed to a notification or the lack thereof without requiring an interactive response from the user. Thus, the computing system can check whether the user (e.g., a vehicle occupant) is directing attention to a notification without distracting the user from other tasks.

[0009] To enable a more detailed understanding of the features listed above in various embodiments, a more specific description of the concepts of the present invention, briefly summarized above, can be made with reference to the various embodiments, some of which are shown in the accompanying drawings. However, since the accompanying drawings only show typical embodiments of the concepts of the present invention, they should not be considered as limiting the scope in any way, and it should be noted that there are other equally effective embodiments. For example, the present application provides the following items. (Item 1) Varying a first visual stimulus at a first frequency, Determining a second frequency related to user characteristics, Based on the determination that the second frequency does not match the first frequency, causing one or more operations to be performed, A computer-implemented method including. (Item 2) The method according to the above item, wherein varying the first visual stimulus at the first frequency includes varying the luminance amplitude of the first visual stimulus at the first frequency. (Item 3) The method according to any one of the above items, wherein the first visual stimulus includes an indicator lamp. (Item 4) The method according to any one of the above items, wherein varying the first visual stimulus at the first frequency includes flashing the indicator lamp at the first frequency. (Item 5) The method according to any one of the above items, wherein the first visual stimulus includes a notification displayed via a display device. (Item 6) The method according to any one of the above items, wherein the user characteristics include the pupil diameter of the user's eyes. (Item 7) The method according to any one of the above items, wherein the second frequency corresponds to a variation in the pupil diameter of the eyes. (Item 8) Determining the second frequency includes capturing an image of the user's eyes via an imaging device and determining the pupil diameter of the eyes, according to any one of the above items of the method. (Item 9) The user characteristics include the electroencephalogram (EEG) of the user or the magnetoencephalogram (MEG) of the user, according to any one of the above items of the method. (Item 10) Further including determining an inattention to the first visual stimulus by the user based on the determination that the second frequency does not match the first frequency, according to any one of the above items of the method. (Item 11) Executing the one or more operations includes outputting a second visual stimulus related to the first visual stimulus in response to determining the inattention to the first visual stimulus by the user, according to any one of the above items of the method. (Item 12) The second visual stimulus is an enlarged version of the first visual stimulus, according to any one of the above items of the method. (Item 13) Executing the one or more operations includes outputting an auditory component related to the first visual stimulus in response to determining the inattention to the first visual stimulus by the user, according to any one of the above items of the method. (Item 14) Executing the one or more operations includes intervening in the operation of the first subsystem in response to determining the inattention to the first visual stimulus by the user, according to any one of the above items of the method. (Item 15) One or more non-transitory computer-readable storage media including instructions, which when executed by one or more processors, varying a first visual stimulus at a first frequency; determining a second frequency related to user characteristics; Based on the determination that the second frequency does not match the first frequency, determining a lack of attention by the user to the first visual stimulus; One or more non-transitory computer-readable storage media that cause the one or more processors to perform the above. (Item 16) Varying the first visual stimulus at the first frequency includes at least one of varying the luminance amplitude of the first visual stimulus at the first frequency or blinking the first visual stimulus at the first frequency, one or more computer-readable storage media according to the above item. (Item 17) When executed by the one or more processors, In response to determining the lack of attention by the user to the first visual stimulus, Outputting a second visual stimulus related to the first visual stimulus, Outputting an auditory component related to the first visual stimulus, or Intervening in the operation of the first subsystem, One or more computer-readable storage media according to any one of the above items, further including instructions that cause the one or more processors to perform at least one of the above steps. (Item 18) A memory for storing an application, When executing the application, Varying a first visual stimulus at a first frequency, Determining a second frequency related to the pupil of the eye, Based on the determination that the second frequency does not match the first frequency, determining a lack of attention to the first visual stimulus, In response to determining the lack of attention to the first visual stimulus, causing one or more operations to be performed, A processor configured as such, Including a system. (Item 19) Causing the one or more operations to be performed includes causing a second visual stimulus related to the first visual stimulus to be output at a third frequency, the system according to the item above. (Item 20) When the processor executes the application, determine a fourth frequency related to the pupil of the eye, Based on the determination that the fourth frequency matches the third frequency, determine the presence of attention to the second visual stimulus, The system according to any one of the above items, further configured as described above. (Abstract) The system can present a notification to a vehicle occupant and determine whether the occupant directed attention to the notification. The system varies the visual notification at an output frequency. The system captures an image of the occupant's eyes and determines the pupil fluctuation frequency of the eyes related to the occupant. The system compares the pupil fluctuation frequency of the eyes with the output frequency. The system determines whether attention was directed to the notification based on whether the output frequency and the pupil fluctuation frequency of the eyes match. If the system determines that attention was not directed, the system can cause the performance of one or more follow-up operations.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Mode for Carrying Out the Invention

[0011] In the following description, a plurality of specific details are described in order to provide a more complete understanding of various embodiments. However, it will be apparent to those skilled in the art that the inventive concept can be practiced without one or more of these specific details.

[0012] Figure 1 shows a block diagram of a vehicle computing system 100 according to one or more aspects of various embodiments. The computing system 100 can control and / or interact with one or more components and / or subsystems of the vehicle. The computing system 100 includes a computing device 101. In some embodiments, the computing device 101 can be a vehicle head unit. As shown, the computing device 101 includes, without limitation, one or more processors 102, an I / O device interface 104, a network interface 106, an interconnect 112, a storage 114, and a memory 116. The processor(s) 102 and the memory 116 can be implemented in any technically feasible manner. For example, without limitation, in various embodiments, any combination of the processor 102 and the memory 116 can be implemented as a stand-alone chip or as part of a more comprehensive solution implemented as an application specific integrated circuit (ASIC), a system on chip (SoC), etc. The processor(s) 102, the I / O device interface 104, the network interface 106, the storage 114, and the memory 116 can be communicatively coupled to each other via the interconnect 112.

[0013] The one or more processors 102 can include any suitable processor, such as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a tensor processing unit (TPU), any other type of processing unit, or a combination of multiple processing units (such as a CPU configured to operate with a GPU). Generally, each of the one or more processors 102 can be any technically feasible hardware unit capable of processing data and / or executing software applications and modules.

[0014] Storage 114 can include non-volatile storage for applications, software modules, and data, and can include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-Ray, HD-DVD, or other magnetic, optical, solid state storage devices, etc.

[0015] Memory 116 can include random access memory (RAM) modules, flash memory units, or any other type of memory unit, or combinations thereof. One or more processors 102, I / O device interface 104, and network interface 106 are configured to read data from and write data to memory 116. Memory 116 includes various software programs and modules (e.g., operating system, one or more applications) that can be executed by processor(s) 102, and application data associated with these software programs (e.g., data loaded from storage 114).

[0016] In some embodiments, computing device 101 is communicatively coupled to one or more networks 160. Network 160 can be any technically realizable type of communication network, which enables the exchange of data between computing device 101 and a remote system or device (not shown) such as a server, a cloud computing system, or another networked computing device or system. For example, network 160 can include a wide area network (WAN), a local area network (LAN), a wireless network (e.g., a Wi-Fi network, a cellular data network), and / or the Internet. Computing device 101 can be connected to network(s) 160 via network interface 106. In some embodiments, network interface 106 is hardware, software, or a combination of hardware and software, and is configured to connect to and interface with network(s) 160.

[0017] The I / O device 108 can include devices capable of providing inputs such as a touch pad, a touch-sensitive screen, buttons, knobs, dials, joysticks, microphones, etc., and can also include devices capable of providing outputs such as a display device, an audio speaker, an indicator lamp, etc. For example, the I / O device 108 can include audio speakers 132, one or more display devices 134, various input devices 138 (e.g., buttons, knobs, dials, etc.), and indicator lamps 136 (e.g., indicators on the dashboard, center console, driver instrument panel, side view mirror, etc.). Examples of the display 134 can include, but are not limited to, an LCD display, an LED display, a touch-sensitive screen, a transparent display, a projection system (e.g., a head-up display projection system, a windshield projection system, a micro projection system), an optical combiner, etc. Further, the I / O device 108 can include devices capable of both receiving inputs and providing outputs, such as a touch-sensitive screen, a universal serial bus (USB) port, etc.

[0018] The I / O device 108 further includes one or more sensor devices 140 that measure and / or monitor various characteristics in the environment inside and / or around the vehicle. The sensor device 140 includes one or more imaging devices 142 (e.g., RGB cameras, infrared cameras). The imaging device 142 includes at least one imaging device directed at the driver's seat in the vehicle such that at least one eye of the driver sitting in the driver's seat is within the field of view of the imaging device. The imaging device can be adjusted automatically and / or manually to position the field of view of the imaging device across one or both eyes of the driver. In some embodiments, the imaging device 142 is configured to capture images at a rate of 30 frames per second or more. The imaging device 142 can be positioned and oriented to capture images inside and / or outside the vehicle and can include additional imaging devices. The sensor device 140 can include additional sensor devices, examples of which include, but are not limited to, accelerometers, gyroscopes, biometric sensors, temperature sensors, rotation sensors, vibration sensors, door closing actuating sensors, light sensors, geographical location sensors (e.g., GPS modules), and the like.

[0019] Computing device 101 can be coupled to one or more vehicle subsystems 122 of a vehicle via I / O device interface 104. Vehicle subsystems 122 include subsystems related to various aspects, functions, and / or components of the vehicle. For example, the vehicle can include a fuel subsystem, a tire subsystem, an engine subsystem, a transmission subsystem, a driving assistance subsystem, a maintenance tracking subsystem, a vehicle door subsystem, a navigation subsystem, an infotainment subsystem, and the like. These subsystems can monitor various aspects of the vehicle (e.g., remaining fuel capacity, engine revolutions per minute, engine problems, engine temperature, tire air pressure, remaining miles until the next proposed maintenance activity, nearby pedestrians and other vehicles, etc.), and / or perform various operations with I / O devices (e.g., sensor device 140), and detect trigger conditions for notifications related to these aspects or operations (e.g., remaining fuel below a certain threshold level, tire air pressure below a certain threshold amount, periodic status notifications, pedestrians suddenly entering the vehicle's path, end of music playback, etc.). In some embodiments, one or more of sensor devices 140 and / or one or more of I / O devices 108 are included in vehicle subsystems 122. In some embodiments, vehicle subsystems 122 include hardware components, software modules, or combinations of hardware components and software modules, and these hardware components and software modules can be implemented in any technically feasible way. When vehicle subsystem 122 detects a trigger condition for a notification, vehicle subsystem 122 sends a notification signal to computing device 101 to cause the output of a visual notification via I / O device 108 (e.g., display(s) 134, indicator lamp(s) 136, etc.).

[0020] As used herein, a notification (which may also be referred to as a "warning") is a presentation of information within a vehicle. The information presented in the notification can be related to the vehicle (e.g., the state of a vehicle component), to the vehicle's occupants, and / or to the environment around the vehicle (e.g., a vehicle in a blind spot, an obstacle in the vehicle's path, etc.). The notification can be visual, tactile, and / or auditory. A visual notification can include text, graphics, symbols (plural possible) (e.g., icons), and / or any combination thereof. An auditory notification can include voice and / or non-voice sounds (e.g., tones, beeps, etc.). A tactile (or "haptic") notification can include vibration, motion, and / or force feedback. Examples of notifications include, but are not limited to, fuel level warning (which can include a number indicating an approximate remaining range), engine overheat warning, low tire pressure warning, vehicle half-door warning, lane departure warning, blind spot warning, warning warning of a short distance to an obstacle within the driving direction, maintenance action timing warning, etc.

[0021] Memory 116 includes an attention-based notification application 150 (hereinafter, "notification application 150"). The notification application 150 can be stored in storage 114 and loaded from storage 114. During operation, the notification application 150 receives notification signals from vehicle subsystem 122 and outputs notifications corresponding to these notification signals to a user (e.g., a vehicle occupant such as a driver) via I / O device 108 (e.g., display 134, indicator lamp 136, speaker 132, etc.), obtains sensor data related to the user, and determines that the user is paying attention to the notification based on the sensor data using a technique based on pupil frequency tagging. In particular, in various embodiments, the notification application 150 outputs a visual notification at a certain variable frequency (e.g., the frequency of the variation in the brightness of the notification). Further, the notification application 150 determines, based on the sensor data, whether attention is being directed to the notification (or the lack thereof), and in particular, determines whether the sensor data indicates that the user shows a response (e.g., contraction and dilation of the user's pupils) at substantially or approximately the same variable frequency. In some embodiments, the notification application 150 can perform further operations based on the determined attention or lack thereof. The operation of the notification application 150 will be further described below with reference to FIGS. 2-5.

[0022] Figure 2 shows an example of a notification that can be presented within a vehicle according to one or more aspects of various embodiments. Figure 2 shows the passenger compartment 200, particularly the view from the passenger compartment towards the front windshield 208. The passenger compartment 200 includes a driver-side dashboard or instrument panel 202, which can include various instrumentation and gauges such as a speedometer, an odometer, a tachometer, and the like. Additionally, the dashboard 202 can include one or more indicator lamps 136, each of which corresponds to a different symbol or icon that conveys specific information defined by the vehicle's manufacturer or designer. For example, the indicator lamp 218 on the dashboard 202 can illuminate a symbol indicating a vehicle's half door, and other indicator lamps 136 on the dashboard 202 can illuminate symbols having other defined meanings and / or conveying other defined information. Also, the passenger compartment 200 can include one or more displays 134, examples of which include a dashboard display 204 at the center of the dashboard 202, a center console display 214 (e.g., an infotainment and / or navigation display), and a head-up display projection system (not shown) that projects augmented reality content (e.g., notifications, landmark information, navigation information) onto the front windshield 208, or onto an optical combiner configured to output the content such that the content appears to be outside the front windshield 208, or onto another panel within the passenger compartment 200. Additionally, or alternatively, the front windshield 208 can be a part of a transparent display. The notification application 150 can present visual notifications to a vehicle occupant (e.g., a driver) via either the display 134 or the indicator lamp 136 within the passenger compartment 200 during operation.

[0023] For example, when the vehicle subsystem 122 (e.g., the fuel subsystem) determines that the remaining fuel in the vehicle is less than a threshold (which can be one of a plurality of remaining fuel thresholds), the fuel subsystem can send a notification signal to the notification application 150 to activate the presentation of a fuel remaining notification. The notification signal can include the information content to be conveyed in the notification (e.g., the remaining range of the fuel remaining notification), and the modality of the notification (e.g., whether the notification is visual, tactile, and / or auditory, whether the notification is presented via an indicator lamp and / or text, and on which display in the passenger compartment 200 the notification is presented). In some embodiments, the notification signal can include a code corresponding to the type of the notification (and implicitly to its content, and optionally to the modality). For example, the vehicle can include a predetermined code set, and each code in this code set is mapped to a specific notification that includes the relevant content and modality. The notification signal can include the code for the intended notification from the code set, and the notification application 150 generates the notification based on the mapping of the code to the corresponding notification. In some embodiments, if the information to be conveyed is associated with one modality, the modality can be omitted. Continuing with the above example, the notification signal for the fuel remaining notification can indicate that the notification is presented as text on the dashboard display 204. The notification signal can further include a value of 40 (e.g., 40 miles) for the remaining range information within the fuel remaining warning.

[0024] In response to receiving a notification signal from the fuel subsystem, the notification application 150 generates a notification to be displayed on the dashboard display 204. For example, the notification application 150 can generate a notification 206 that reads "Low fuel! Range 40 miles". In some examples, the notification application 150 can further select a variable frequency for the notification. The variable frequency is the frequency at which the notification application 150 varies the notification when the notification is displayed. In some embodiments, the frequency is selected from a predetermined set of frequencies (e.g., within a frequency table stored in the storage 114). In some other embodiments, the frequency is randomly selected from a predetermined frequency range. In some embodiments, the predetermined set of frequencies, or the predetermined frequency range, is empirically defined by the vehicle manufacturer or designer to optimize the detectability of the eye pupil variation frequency in response to the variation of the notification.

[0025] After notification 206 is generated and a variable frequency is selected, notification application 150 outputs notification 206 to dashboard display 204 with variations at the selected variable frequency. In some embodiments, notification application 150 varies the amplitude of the brightness of notification 206 on dashboard display 204 at the selected frequency. Varying the brightness (in particular, the amplitude of the brightness) can include adjusting the brightness of the content (e.g., text) of notification 206, adjusting the brightness of the background of notification 206, or a combination thereof. In some embodiments, the adjustment of the brightness includes a transition back and forth between a first brightness level (e.g., a relatively high brightness level) and a second brightness level (e.g., a relatively low brightness level). In some embodiments, the first and second brightness levels are selected to have a separation large enough to be detectable by notification application 150 in consideration of the ambient light level around the occupant. The ambient light level around the occupant can be detected via sensor device 140 (e.g., a light sensor). The transition between the first brightness level and the second brightness level can be gradational (e.g., a continuous transition) or discrete (e.g., a step transition). In some examples, the time that notification 206 is at each brightness level can be the same or different. While notification 206 is being displayed with variations, notification application 150 acquires a sequence of images of at least one eye of the vehicle occupant via imaging device 142 in the vehicle. For example, notification application 150 can perform object recognition and / or any other technically feasible technique (e.g., face box detection) based on the images captured by imaging device 142 to recognize the occupant's eye (in particular, the pupil of the eye) and optionally orient imaging device 142 towards the occupant's eye. Next, notification application 150 can capture a sequence of images of the eye via imaging device 142. Next, notification application 150 processes the sequence of images to measure the change in the pupil diameter of the eye over a certain period and determine the frequency indicated by the eye (e.g., the frequency of contraction and dilation of the pupil of the eye) based on the change in the pupil diameter.In some embodiments, eye image processing and determination of the eye pupil fluctuation frequency may cause delays in data acquisition and signal processing by checking changes in pupil diameter over a plurality of periods (e.g., 5 to 10 periods) before determining the eye pupil fluctuation frequency.

[0026] If the frequency of notification 206 and the eye pupil fluctuation frequency do not match (e.g., if the difference between the two frequencies is outside a predetermined tolerance), the notification application 150 determines that the occupant is not paying attention to notification 206. In response, the notification application 150 can then send a negative attention signal to the fuel subsystem, in which case the fuel subsystem can send another notification signal to the notification application 150. The other notification signal can indicate additional or alternative modalities for the notification and / or additional or alternative information included in the notification to escalate notification 206. Additionally or alternatively, the notification application 150 can automatically perform one or more additional operations. For example, the notification application 150 can escalate notification 206 by outputting notification 206 via an additional or alternative modality (e.g., outputting audio with a visual notification) without waiting for another notification signal from the fuel subsystem. As another example of notification escalation, the notification application 150 can escalate subsequent notifications related to notification 206 (e.g., subsequent fuel level notifications indicating a further reduced remaining range due to a further decrease in fuel). Other examples of notification escalation include displaying the notification in a larger size (e.g., a larger font size), moving the notification (e.g., displaying the notification on another larger display), increasing the brightness and / or contrast of the notification, intervening in the operation of the vehicle (e.g., activating the automatic brake), and adding an auditory component and / or a tactile component (e.g., beep sound, vibration, etc.) to the notification.

[0027] In some embodiments, the determination of whether the occupant has paid attention includes a persistent component (e.g., time, distance traveled by the vehicle, etc.). In particular, the notification application 150 can determine whether the occupant has paid attention to the notification based on frequency matching over a duration starting from when the notification (e.g., notification 206) was first presented. If the notification application 150 determines that the notification fluctuation frequency and the frequency indicated by the pupil of the eye do not match for at least a threshold duration, the notification application 150 determines that the occupant has not paid attention to the notification. In response to that determination, the notification application 150 can send a negative attention signal and / or escalate the notification. In this way, the notification is detected and the occupant is given a duration (e.g., period, travel distance) until attention is paid to the attention before the notification application 150 determines that no attention has been paid.

[0028] When the fluctuation frequency of the notification 206 and the pupil fluctuation frequency of the eye match (e.g., when the difference between the two frequencies is within a predetermined tolerance), the notification application 150 determines that the occupant has paid attention to the notification 206. In response to the determination that the occupant has paid attention to the notification 206, the notification application 150 can send a positive attention signal to the fuel subsystem. The fuel subsystem can, in response to the positive attention signal, maintain or stop the notification signal, or send to the notification application 150 another notification signal that de-escalates this notification (e.g., removes certain information from the notification, removes the notification modality, dims the notification, etc.). In response to the de-escalation notification signal, the notification application 150 can reverse one or more of the above notification escalation modes.

[0029] In some embodiments, the attention signal includes a flag indicating attention or its absence. For example, a negative attention signal includes a flag value of 0 indicating that no attention is being directed, and a positive attention signal includes a flag value of 1 indicating that attention is being directed. The notification application 150 can maintain the flag in the memory 116, set the flag to 1 in response to a determination that attention has been directed to the notification, reset the flag in response to stopping or closing the notification, and / or include the flag in the attention signal transmitted to the vehicle subsystem 122.

[0030] The notification application 150 can output other notifications in a similar manner as described above in response to notification signals from various vehicle subsystems 122. For example, the notification application 150 can display a vehicle half-door notification based on a notification signal from the vehicle door subsystem (e.g., when the vehicle door subsystem detects that the vehicle door is half-open). The notification application 150 can display and vary the half-door notification via the blinking indicator lamp 218 by blinking the indicator lamp 218 at a selected variable frequency. The notification application 150 can determine that the occupant has not directed attention to the vehicle half-door notification for at least a threshold period (e.g., duration) after the notification is displayed. In response to that determination, the notification application 150 can transmit a negative attention signal to the door subsystem and / or escalate the notification.

[0031] The notification application 150 can display an oil change timing notification 216 based on a notification signal from the maintenance tracking subsystem (e.g., when the maintenance tracking subsystem detects that the distance the vehicle has traveled since the last oil change exceeds a predetermined oil change mileage threshold). The notification application 150 can output the oil change timing notification 216 on the center console display 214 by means of a brightness (of text, background, or both) that varies at a selected variable frequency. The notification application 150 can determine that the occupant has not paid attention to the oil change notification for at least a threshold duration (e.g., length of travel distance) since the notification was displayed. In response to that determination, the notification application 150 can send a negative attention signal to the maintenance tracking subsystem and / or escalate the notification.

[0032] Furthermore, the notification application 150 can output head-up or augmented reality notifications in a similar manner. For example, the notification application 150 can display (e.g., display or project on the windshield 208) an outline or bounding box 212 based on a notification signal from the driving assistance subsystem (e.g., when the driving assistance subsystem detects a pedestrian near the vehicle). The outline 212 surrounds an object near the vehicle (e.g., the pedestrian 210 outside the vehicle) when displayed through the windshield 208. When the pedestrian 210 moves outside the vehicle, the notification application 150 can adjust the projection position of the outline 212 so that the outline 212 follows the pedestrian 210 when displayed through the windshield 208. The notification application 150 can blink the outline 212 at a selected variable frequency to determine whether the driver has directed attention to the outline 212 (and thus to the pedestrian 210). If the pupil fluctuation frequency of the eye does not match the fluctuation frequency of the outline 212, the notification application 150 can determine that the driver has not directed attention to the pedestrian 210 and, accordingly, can trigger the output of an additional notification of the presence of the pedestrian 210 (e.g., by triggering the output of an audible obstacle warning). As another example, the notification application 150 can project navigation information (e.g., the next turn notification 220) onto the windshield 208 based on a notification signal from the navigation subsystem (e.g., when the navigation subsystem detects the next turn on the navigation route). The notification application 150 can blink the next turn notification 220 at a selected variable frequency to determine whether the driver has directed attention to the turn notification 220. If the pupil fluctuation frequency of the eye does not match the fluctuation frequency of the turn notification 220, the notification application 150 can determine that the driver has not directed attention to the turn notification 220 and, accordingly, can escalate the turn notification 220 (e.g., by triggering the output of audible navigation guidance).

[0033] The above-mentioned notification variations include blinking (e.g., variations between on and off, and between visible and invisible) and brightness variations. It should be understood that other types of variations of visual notifications are possible. For example, the notification application 150 can vary the notification between two colors, between two different shades of the same color (e.g., a light shade and a dark shade), and / or between a first contrast level and a second contrast level. The variations in color, shade of color, and / or contrast level can be gradational (e.g., smooth and continuous) or discrete (e.g., stepwise).

[0034] In some embodiments, the simultaneously displayed notifications can have different variation frequencies. When selecting the variation frequency for a notification, the notification application 150 selects a frequency different from the variation frequencies of the other currently presented notifications. By having different variation frequencies for different simultaneously presented notifications, the notification application 150 can determine the attention (or lack thereof) directed to each notification by matching the eye pupil variation frequency to different notification variation frequencies. Further, in some embodiments, to account for the amount of ambient light, the notification application 150 can dynamically adjust the variation frequency of the notification and / or adjust the first and / or second variation levels (e.g., high brightness level and / or low brightness level) of the notification based on the amount of ambient light.

[0035] By using the techniques disclosed above, it is possible to determine with higher accuracy that attention is directed to a notification than by relying on checking the line of sight direction and without determining attention based on the line of sight direction. Further, since the adoption of the frequency of visual stimulation by the eye pupil based on pupil frequency tagging does not require a focused view, it is possible to use the techniques disclosed above to determine that attention is directed to a notification within the peripheral vision of an occupant.

[0036] Figures 3A - 3B illustrate a flowchart of a process 300 for presenting attention - based notifications according to one or more aspects of various embodiments. Process 300 begins at step 302, where vehicle subsystem 122 detects a notification event. Vehicle subsystem 122 can detect an event or condition that meets the trigger criteria for a notification. For example, a fuel subsystem can detect that the remaining fuel has fallen below a threshold. As another example, a maintenance tracking subsystem can detect that it is time for an oil change based on the duration or mileage driven since the last oil change. As a further example, a driving assistance subsystem can detect that the distance from the vehicle to another vehicle within the direction of travel has fallen below a threshold.

[0037] At step 304, vehicle subsystem 122 sends a notification signal to notification application 150. The notification signal can indicate the content and modality of the visual notification to be generated. At step 306, notification application 150 receives the notification signal and generates a visual notification based on this notification signal. At step 308, notification application 150 selects a notification output frequency. Notification application 150 selects a variation frequency for the notification that, if any, is different from the variation frequencies of other currently presented notifications. At step 310, notification application 150 sends a notification output signal to I / O device interface 104 for output via I / O device 108 (e.g., display 134, indicator lamp 136, etc.).

[0038] In step 312, the I / O device interface 104 receives a notification output signal. In step 314, the I / O device interface 104 outputs a notification at a selected output frequency via the I / O device 108. In step 316, the I / O device interface 104 captures at least one eye image (e.g., an image sequence) of a vehicle occupant (e.g., a driver) via the imaging device 142. In step 318, the I / O device interface 104 transmits the image to the notification application 150.

[0039] Following FIG. 3B, in step 320, the notification application 150 processes the image to detect the eye pupil diameter and the corresponding frequencies of pupil constriction and dilation. The notification application 150 recognizes the eye pupils within each image and determines the corresponding pupil diameters using any technically feasible technique (e.g., machine learning-based object recognition), resulting in a sequence of measured pupil diameter values that are ordered based on the timestamps of these images. The notification application 150 determines the eye pupil fluctuation frequency (e.g., the frequency of pupil constriction and dilation) based on the sequence of pupil diameter values. If the notification application 150 determines that the fluctuations in the eye pupil diameter do not exhibit a determinable frequency (e.g., the eye pupil diameter changes irregularly or randomly without a determinable variation or variation pattern), the notification application 150 can determine that the eye pupil fluctuation frequency is null.

[0040] In step 322, the notification application 150 determines whether the determined pupil fluctuation frequency of the eyes matches the fluctuation frequency of the notification. If the frequency is determined to match by the notification application 150, in step 328, the notification application 150 sends a positive attention signal to the vehicle subsystem 122. In step 330, the vehicle subsystem 122 receives the positive attention signal and determines, based on the positive attention signal, that the user has directed attention to the notification. In response to that determination, the vehicle subsystem 122 can maintain or stop the notification signal, or can send a notification signal for the de-escalated notification to the notification application 150.

[0041] If the notification application 150 determines that the frequencies do not match or that the pupil fluctuation frequency of the eyes is null, in step 324, the notification application 150 causes one or more follow-up operations to be performed. The follow-up operations can include sending a negative attention signal to the vehicle subsystem 122, escalating the notification, and / or intervening in the operation of the vehicle. Next, process 300 returns to step 316, where an updated pupil fluctuation frequency of the eyes can be determined by capturing a new image of the user's eyes and compared with the notification fluctuation frequency.

[0042] FIG. 4 shows a flowchart of method steps for presenting attention-based notifications according to one or more aspects of various embodiments. Although the method steps are described in relation to the systems of FIGS. 1-3B, those skilled in the art will understand that any system configured to perform the method steps is within the scope of the various embodiments in any order.

[0043] Method 400 begins at step 402, where notification application 150 receives a notification signal from a vehicle subsystem. Notification application 150 can receive from vehicle subsystem 122 the content of a visual notification to be presented to a vehicle occupant (e.g., a driver), and optionally a signal indicating a modality. Notification application 150 can generate a visual notification based on the notification signal.

[0044] At step 404, notification application 150 selects an output frequency for the presented notification. At step 406, notification application 150 causes the output of the notification by an I / O device at the output frequency. Notification application 150 outputs the visual notification to I / O device 108 (e.g., display 134, indicator lamp 136, etc.) such that the notification (e.g., the brightness of the notification) varies at the output frequency.

[0045] At step 408, notification application 150 detects the pupil diameter of the occupant's eyes over a period of time. Based on these eye pupil diameters, notification application 150 determines the fluctuation frequency of the eye pupils, i.e., the fluctuation frequency of the eye pupil diameter between a high diameter value and a low diameter value (e.g., the frequency of pupil constriction and dilation). Moreover, if notification application 150 determines that the fluctuation of the eye pupil diameter does not exhibit a determinable frequency (e.g., the eye pupil diameter changes irregularly or randomly without a determinable variation or variation pattern), notification application 150 can determine that the eye pupil fluctuation frequency has not been detected.

[0046] At step 410, notification application 150 determines whether the eye pupil fluctuation frequency matches the output frequency of the notification. If the frequencies match (e.g., within a predetermined tolerance), method 400 proceeds to step 412, where notification application 150 sends a positive attention signal to vehicle subsystem 122 to send a signal to the subsystem that attention has been directed to the notification.

[0047] If the frequencies do not match or if the pupil fluctuation frequency of the eye is not detected, method 400 proceeds to step 414, where notification application 150 causes one or more follow-up operations to be performed. Follow-up operations can include, for example, outputting follow-up notifications, displaying these notifications within additional modalities, outputting the notifications separately (e.g., making them larger, displaying them at different locations on the screen), and / or intervening in the operation of the vehicle (e.g., activating the automatic brakes). The follow-up operations can further include sending a negative attention signal to vehicle subsystem 122 to send a signal to the subsystem that attention has not been directed to the notification.

[0048] Figure 5 shows another flow diagram of method steps for presenting attention-based notifications according to one or more aspects of various embodiments. Although the method steps are described in relation to the systems of FIGS. 1-3B, those skilled in the art will understand that any system configured to perform the method steps is within the scope of the various embodiments in any order.

[0049] Method 500 begins at step 502, where notification application 150 receives a notification signal from the vehicle subsystem. Notification application 150 can receive the content of a visual notification to be presented to a vehicle occupant (e.g., the driver) and optionally a signal indicating the modality from vehicle subsystem 122. Notification application 150 can generate a visual notification based on the notification signal.

[0050] In step 504, the notification application 150 selects an output frequency for the presented notification. In step 506, the notification application 150 causes the I / O device to output the notification at the output frequency. The notification application 150 outputs a visual notification to the I / O device 108 (e.g., the display 134, the indicator lamp 136) such that the notification (e.g., the brightness of the notification) varies at the output frequency.

[0051] In step 508, the notification application 150 detects the pupil diameter of the occupant's eyes over a period of time. Based on these eye pupil diameters, the notification application 150 determines the fluctuation frequency of the eye pupils, i.e., the fluctuation frequency of the eye pupil diameter between a high diameter value and a low diameter value (e.g., the frequency of pupil constriction and dilation). Moreover, if the notification application 150 determines that the fluctuation of the eye pupil diameter does not exhibit a determinable frequency (e.g., the eye pupil diameter changes irregularly or randomly without a determinable fluctuation or fluctuation pattern), the notification application 150 can determine that the eye pupil fluctuation frequency has not been detected.

[0052] In step 510, the notification application 150 determines whether the eye pupil fluctuation frequency matches the output frequency of the notification. If the frequencies do not match, or if the eye pupil fluctuation frequency has not been detected, method 500 proceeds to step 512, where the notification application 150 escalates the notification (e.g., presents the notification in a different modality). From step 512, method 500 proceeds to step 506, where the notification (or a subsequent related notification) can continue to be output at the same or a different output frequency. Next, method 500 proceeds to step 508, where the notification application 150 can determine the eye pupil fluctuation frequency again and compare it with the notification output frequency.

[0053] When the frequencies match, method 500 proceeds to step 514, where notification application 150 sends a positive attention signal to vehicle subsystem 122 to send a signal to the subsystem that attention has been directed to the notification. In step 516, notification application 150 de-escalates the notification. If the notification is not escalated and de-escalation is not required, step 516 may be omitted.

[0054] In some embodiments, in addition to, or instead of, the pupil diameter of the eye and the associated frequency, other user characteristics may be measured to determine the frequency indicated by the user. For example, a biometric sensor may measure the user's electroencephalogram (EEG) or magnetoencephalogram (MEG). The frequency can be determined from the EEG or MEG and compared to the varying frequency of the notification.

[0055] In some embodiments, the notification application 150 can output notifications in a more complex pattern. For example, the notification application 150 can vary a notification at a first frequency for a first (e.g., short) period, and then after a short pause, vary it at a second frequency (many of which are the same as or different from the first frequency) for a second (e.g., longer) period (e.g., the brightness). In some examples, the notification application 150 can vary the notification at additional frequencies (many of which are the same as or different from the first or second frequency) for additional periods. This multiple-period pattern can be repeated. Thus, the notification can be varied according to a repeatable temporal pattern that includes variations at different frequencies and / or brightness levels for short and / or longer periods. The notification application 150 processes an image of the vehicle occupant's eye to determine whether the pupil diameter of the eye and its changes exhibit the same temporal pattern at the same frequency as the notification. If the changes in the pupil diameter of the eye exhibit the same pattern with respect to the same frequency, the notification application 150 determines that the occupant has directed attention to the notification. If the changes in the pupil diameter of the eye exhibit a different frequency than the notification and / or do not exhibit the same pattern, the notification application 150 determines that the occupant has not directed attention to the notification.

[0056] Although the techniques disclosed above have been described with respect to notifications within a vehicle, similar techniques can be implemented in other situations. For example, an electronic kiosk within a building or space (e.g., an electronic directory and map kiosk within a shopping mall, hotel resort, amusement park, etc.) can include a display device and an imaging device. When a user requests directions to a particular location at the electronic kiosk, similar to the variations in the notifications described above, the electronic kiosk can display the requested location and / or the route to the requested location on a map at a varying frequency (e.g., blinking a location marker at a varying frequency). The imaging device can capture an image of the user's eye(s), and the electronic kiosk can determine the eye pupil variation frequency based on these images. The electronic kiosk can determine whether the eye pupil variation frequency matches the variation frequency of the directions. If the electronic kiosk determines that the frequencies do not match, the electronic kiosk can escalate the presentation of the directions (e.g., zoom in on the location, move the map as if crossing the route, emit a sound). As another example, a wearable device can display augmented reality content at a varying frequency and determine the eye pupil variation frequency and compare that frequency to the variation frequency of the augmented reality content to determine whether the user is paying attention to the augmented reality content.

[0057] Furthermore, the techniques disclosed above can be suitable for use in empirical research and analysis (e.g., to assist in the design of a user interface for a vehicle). For example, in the design of a notification presentation modality (e.g., the design of a dashboard), multiple options can be considered. Designers can test and optimize various design options for the notification presentation modality by using the techniques disclosed above and determine a more effective modality option for attracting the attention of vehicle occupants.

[0058] In summary, a computing system determines whether attention has been directed to an indicator or notification. The computing system displays the notification by varying the amplitude of the notification (e.g., luminance, brightness) at a first frequency. The computer system includes one or more sensors that collect sensor data related to a user's characteristics (e.g., pupil diameter of the eye). The computer system determines whether the sensor data indicates a second frequency related to the characteristic and determines that the second frequency substantially matches the first frequency. If the first and second frequencies do not match or the sensor data does not indicate a frequency related to the characteristic, the computer system can perform one or more additional operations. Examples of additional operations include delivering the notification via a different modality, escalating the notification, automatically intervening in the operation of the device or system, etc. In some embodiments, the computer system can be implemented in a vehicle and the sensor includes a camera device.

[0059] At least one advantage and technical improvement of the disclosed techniques is that a computing system (e.g., a vehicle computing device) can determine the attention directed to a notification or the lack thereof without requiring an interactive response from a user. Thus, the computing system can determine whether a user (e.g., a vehicle occupant) is paying attention to a notification without distracting the user from other tasks. Further, the computing system can perform one or more follow-up operations based on whether attention has been directed to the notification. The follow-up operations can include presenting the notification via a different modality, escalating the notification, automatically intervening in the operation of a device or system, and the like. Another advantage and technical improvement of the disclosed techniques is that the computing system can determine which modality of delivering a notification is more likely to draw the user's attention. Thus, the computing system may prioritize a particular modality in delivering a notification. Further advantages and technical improvements are that the user's attention being directed to a notification can be detected even when the notification is not within the user's focused view. Another advantage and technical improvement is that the user's attention being directed to a notification can be determined with higher accuracy compared to conventional approaches (e.g., determination based on line of sight direction).

[0060] 1. In some embodiments, a computer-implemented method includes varying a first visual stimulus at a first frequency, determining a second frequency related to user characteristics, and causing one or more operations to be performed based on a determination that the second frequency does not match the first frequency.

[0061] 2. The method of clause 1, wherein varying the first visual stimulus at the first frequency includes varying a luminance amplitude of the first visual stimulus at the first frequency.

[0062] 3. The method of clause 1 or 2, wherein the first visual stimulus includes an indicator lamp.

[0063] 4. Varying the first visual stimulus at the first frequency includes flashing the indicator lamp at the first frequency, the method according to any one of clauses 1 to 3.

[0064] 5. The first visual stimulus includes a notification displayed via a display device, the method according to any one of clauses 1 to 4.

[0065] 6. The user characteristic has the pupil diameter of the user's eye, the method according to any one of clauses 1 to 5.

[0066] 7. The second frequency corresponds to the variation of the pupil diameter of the eye, the method according to any one of clauses 1 to 6.

[0067] 8. Determining the second frequency includes capturing an image of the user's eye via an imaging device and determining the pupil diameter of the eye, the method according to any one of clauses 1 to 7.

[0068] 9. The user characteristic includes the electroencephalogram (EEG) of the user or the magnetoencephalogram (MEG) of the user, the method according to any one of clauses 1 to 8.

[0069] 10. Further including determining the lack of attention of the user to the first visual stimulus based on the determination that the second frequency does not match the first frequency, the method according to any one of clauses 1 to 9.

[0070] 11. Executing the one or more operations includes outputting a second visual stimulus related to the first visual stimulus in response to determining the lack of attention of the user to the first visual stimulus, the method according to any one of clauses 1 to 10.

[0071] 12. The second visual stimulus is an enlarged version of the first visual stimulus, the method according to any one of clauses 1 to 11.

[0072] 13. Performing the one or more operations includes outputting an auditory component related to the first visual stimulus in response to determining the lack of attention of the user to the first visual stimulus, according to any one of clauses 1 to 12.

[0073] 14. Performing the one or more operations includes intervening in the operation of the first subsystem in response to determining the lack of attention of the user to the first visual stimulus, according to any one of clauses 1 to 13.

[0074] 15. In some embodiments, one or more non-transitory computer-readable storage media including instructions, which, when executed by one or more processors, cause the one or more processors to perform steps of varying a first visual stimulus at a first frequency, determining a second frequency related to user characteristics, and determining a lack of attention of the user to the first visual stimulus based on a determination that the second frequency does not match the first frequency.

[0075] 16. Varying the first visual stimulus at the first frequency includes at least one of varying the luminance amplitude of the first visual stimulus at the first frequency or blinking the first visual stimulus at the first frequency, according to the one or more computer-readable storage media of clause 15.

[0076] 17. The instructions further include causing the one or more processors to perform at least one of steps of outputting a second visual stimulus related to the first visual stimulus, outputting an auditory component related to the first visual stimulus, or intervening in the operation of the first subsystem in response to determining the lack of attention of the user to the first visual stimulus when executed by the one or more processors, according to the one or more computer-readable storage media of clause 15 or 16.

[0077] 18. In some embodiments, a system including a memory storing an application and a processor, wherein when executing the application, the processor varies a first visual stimulus at a first frequency, determines a second frequency related to the pupil of the eye, determines an absence of attention to the first visual stimulus based on a determination that the second frequency does not match the first frequency, and is configured to cause one or more operations to be performed in response to determining the absence of attention to the first visual stimulus.

[0078] 19. The system according to clause 18, wherein causing the one or more operations to be performed includes causing a second visual stimulus related to the first visual stimulus to be output at a third frequency.

[0079] 20. The system according to clause 18 or 19, wherein the processor is further configured to determine a fourth frequency related to the pupil of the eye when executing the application, and determine the presence of attention to the second visual stimulus based on a determination that the fourth frequency matches the third frequency.

[0080] Any combination and all combinations, in any manner, of any claim elements recited in any claim, and / or any elements described in this application, fall within the scope contemplated by the protection of the present invention.

[0081] The description of various embodiments is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0082] Aspects of the present embodiment can be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining both software and hardware aspects generally referred to herein as a "module," "system," or "computer." Further, any hardware and / or software technology, process, function, component, engine, module, or system described in the present disclosure can be implemented as a circuit or set of circuits. Further, aspects of the present disclosure can take the form of a computer program product embodied in one or more computer-readable media (s) having computer-readable program code embodied thereon.

[0083] Any combination of one or more computer-readable media (s) can be utilized. The computer-readable media can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium would include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM or flash memory), an optical fiber, a compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0084] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus to manufacture a machine. When the instructions are executed via the processor of the computer or other programmable data processing apparatus, they enable the implementation of the functions / acts specified in one or more blocks of the flowchart and / or block diagram. Such a processor can be, by way of non-limiting example, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable gate array.

[0085] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block of the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, depending upon the functionality involved, or the blocks may sometimes be executed in the reverse order. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified function or act, or by combinations of dedicated hardware and computer instructions.

[0086] Although the above is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which is determined by the appended claims.

Claims

**Claim 1** A computer-implemented method, the computer-implemented method comprising: varying a first visual stimulus in a first frequency pattern, the first frequency pattern including a variation at a first frequency over a first period and a variation at a second frequency over a second period subsequent to the first period; determining a second frequency pattern related to user characteristics; performing one or more operations based on a determination that the second frequency pattern does not match the first frequency pattern; A computer-implemented method comprising the steps of: **Claim 2** The method according to claim 1, wherein varying the first visual stimulus in the first frequency pattern includes varying a luminance amplitude of the first visual stimulus in the first frequency pattern. **Claim 3** A computer-implemented method, the computer-implemented method comprising: varying a first visual stimulus at a first frequency; determining a second frequency related to user characteristics; performing one or more operations based on a determination that the second frequency does not match the first frequency; wherein the first visual stimulus includes an indicator lamp. **Claim 4** The method according to claim 3, wherein varying the first visual stimulus at the first frequency includes flashing the indicator lamp at the first frequency. **Claim 5** The method according to claim 1, wherein the first visual stimulus includes a notification displayed via a display device. **Claim 6** The method according to claim 1, wherein the user characteristics include a pupil diameter of the user's eye. **Claim 7** The method according to claim 6, wherein the second frequency pattern corresponds to a variation in the pupil diameter of the eye. **Claim 8** The method according to claim 6, wherein determining the second frequency pattern includes capturing an image of the user's eye via an imaging device and determining the pupil diameter of the eye. **Claim 9** The method according to claim 1, wherein the user characteristics include the user's electroencephalogram (EEG) or magnetoencephalogram (MEG) of the user. **Claim 10** The method according to claim 1, further comprising determining an inattentiveness of the user to the first visual stimulus based on the determination that the second frequency pattern does not match the first frequency pattern. **Claim 11** Causing the one or more operations to be performed includes outputting a second visual stimulus related to the first visual stimulus in response to determining the lack of attention of the user to the first visual stimulus, the method according to claim 10. **Claim 12** A computer-implemented method, the computer-implemented method comprising: Varying a first visual stimulus at a first frequency; Determining a second frequency related to user characteristics; Performing one or more operations based on a determination that the second frequency does not match the first frequency; Determining a lack of attention of the user to the first visual stimulus based on the determination that the second frequency does not match the first frequency; comprising: Performing the one or more operations includes outputting a second visual stimulus related to the first visual stimulus in response to determining the lack of attention of the user to the first visual stimulus, The second visual stimulus is an enlarged version of the first visual stimulus, a computer-implemented method. **Claim 13** Performing the one or more operations includes outputting an auditory component related to the first visual stimulus in response to determining the lack of attention of the user to the first visual stimulus, the method according to claim 10. **Claim 14** A computer-implemented method, the computer-implemented method comprising: Varying a first visual stimulus at a first frequency; Determining a second frequency related to user characteristics; Performing one or more operations based on a determination that the second frequency does not match the first frequency; Determining a lack of attention of the user to the first visual stimulus based on the determination that the second frequency does not match the first frequency; comprising: Performing the one or more operations includes intervening in the operation of a first subsystem in response to determining the lack of attention of the user to the first visual stimulus, a computer-implemented method. **Claim 15** One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, Varying a first visual stimulus in a first frequency pattern, the first frequency pattern including variation at a first frequency over a first period and variation at a second frequency over a second period following the first period, step; Determining a second frequency pattern related to user characteristics; Determining a lack of attention by the user to the first visual stimulus based on a determination that the second frequency pattern does not match the first frequency pattern; One or more non-transitory computer-readable storage media for causing the one or more processors to execute.

16. One or more non-transitory computer-readable storage media including instructions which, when executed by one or more processors, Varying a first visual stimulus at a first frequency; Determining a second frequency related to user characteristics; Determining a lack of attention by the user to the first visual stimulus based on a determination that the second frequency does not match the first frequency; Causing the one or more processors to execute, One or more non-transitory computer-readable storage media, wherein varying the first visual stimulus at the first frequency includes at least one of varying the luminance amplitude of the first visual stimulus at the first frequency or flashing the first visual stimulus at the first frequency.

17. When executed by the one or more processors, In response to determining the lack of attention by the user to the first visual stimulus, Outputting a second visual stimulus related to the first visual stimulus, Outputting an auditory component related to the first visual stimulus, or Intervening in the operation of a first subsystem, The one or more computer-readable storage media according to claim 15, further including instructions for causing the one or more processors to execute at least one of the steps.

18. A memory storing an application, When executing the application, Varying a first visual stimulus at a first frequency; Determining a second frequency related to the pupil of the eye; Determining a lack of attention to the first visual stimulus based on a determination that the second frequency does not match the first frequency; In response to determining the inattention to the first visual stimulus, causing one or more operations to be performed; A processor configured to perform; A system including.

19. The system according to claim 18, wherein causing the one or more operations to be performed includes outputting a second visual stimulus related to the first visual stimulus at a third frequency.

20. When the processor executes the application, Determining a fourth frequency related to the pupil of the eye; Based on the determination that the fourth frequency matches the third frequency, determining the presence of attention to the second visual stimulus; The system according to claim 19, further configured to perform.

Images