Biofeedback method for conditioning digital content to stimulate a larger pupil radius response

Abstract

One embodiment displays a visual feature associated with an object on a display of a device, and uses a sensor of the device to obtain physiological data associated with a user's pupil response to the visual feature. The device adjusts the visual feature based on the obtained physiological data to enhance the user's pupil response to the object, and displays the adjusted visual feature to the user. For example, the adjusted visual feature can be selected based on a previously identified pupil response of the user to a particular visual feature.

Description

Technical Field

[0001] The present disclosure relates generally to displaying objects using electronic devices, and more particularly to systems, methods, and devices for adjusting visual features associated with objects to enhance a user's pupil response. Background Art

[0002] Electronic devices have different capabilities for viewing and interacting with electronic content. A variety of input mechanisms have been incorporated into a variety of user devices to provide functionality and user interaction (e.g., keyboard, mouse, touch screen, buttons, microphone for voice commands, optical sensors, etc.). For example, touch screens have been incorporated into mobile phones (e.g., smart phones), tablet computers, wearable devices (e.g., watches, glasses, head-mounted devices, etc.), and other computing devices, allowing software developers to create engaging software applications ("apps") for entertainment, productivity, health, etc. In some cases, touch screens work in conjunction with a variety of other input mechanisms to interact with devices (e.g., optical sensors, buttons, microphone for voice commands, etc.).

[0003] However, due to device size limitations, display size constraints, operational constraints, etc., many devices may have limited device interactions and control capabilities, such as small or thin user devices may have a limited number of physical buttons for receiving user input. Similarly, small user devices may have a space-limited touch screen for providing visual buttons or other visual user interface elements. In addition, some devices may have buttons or other interactive elements, which are unnatural, troublesome or uncomfortable to use under certain conditions or certain operating conditions. For example, using both hands to interact with the device may be troublesome (for example, one hand holding the device, while the other hand uses the interface element). For another example, when the user's hands are otherwise occupied or cannot be used (for example, when wearing gloves, holding things, holding a child's hand, driving a car, etc.), it may be difficult to press a small button or use a touch screen function. In other examples, device interaction may be limited in many other ways. Summary of the invention

[0004] Various specific implementations disclosed herein include devices, systems, and methods for adjusting visual features (e.g., hue, saturation, size, shape, spatial frequency, motion, highlighting, etc.) associated with an object (e.g., a button, UI icon, etc.) to enhance a user's pupil response to the display of the object. The device (e.g., a handheld device, a laptop computer, a desktop computer, or a head-mounted device (HMD)) displays the visual features associated with the object to the user on a display, and uses a sensor to obtain physiological data (e.g., pupil dilation) associated with the user's response to the visual features. Based on the acquired physiological data, the device adjusts the visual features to enhance the user's pupil response to the object.

[0005] In some implementations, the device evaluates the physiological response of a user who expects interface feedback after interacting with an object in a user interface. The physiological response can be associated with a user who expects interface feedback using multiple alternative interface feedback features. In one example, it is desirable to predict the user's intention to select a button before the user selects the button based on the user's pupil dilation when the user intends to select the button. In order to facilitate such predictions, it may be desirable to configure the button, including feedback associated with the selection of the button, to provide a significant or optimal pre-selection pupil dilation change, for example, when the user intends to make a selection and expects certain feedback after the selection, it is determined that the user's pre-selection pupil dilation is good or optimal. To this end, the device can evaluate various alternative feedback features (e.g., which of multiple colors the button should change to after the user clicks the button) to identify the characteristics of the feedback as appropriate or optimal relative to the pre-selection pupil dilation. The device can estimate how much the user's eyes dilate before clicking the object when the user expects each interface feedback feature (e.g., the object turns pink after being clicked, the object turns orange after being clicked, etc.). In some implementations, the device selects one of the alternative interface feedback features based on evaluating physiological data. For example, the device may select pink as the desired interface feedback feature based on determining that the user's eyes dilate more when the desired object turns pink than when the desired object turns orange.

[0006] Once an object feature is selected, the feature can be applied to facilitate intent detection (e.g., based on the user's eye dilation, detecting that the user intends to select a button before the user selects the button). For example, in some specific implementations, the device displays an object so that selecting the displayed object can provide interface feedback using the selected interface feedback feature. For example, after training, the object can be displayed in the application using pink interface feedback. When the user uses the user interface and selects the object, the user of the interface will observe pink interface feedback. In one or more occurrences of the feedback, the user will expect pink interface feedback and will typically present a pre-selection pupil dilation associated with the expected pink interface feedback. Therefore, in some specific implementations, the device obtains a physiological response to the displayed object such as pupil dilation before the object is selected, and determines the user's intention to interact with the object based on the obtained physiological response. Based on the determined user's intention to interact with the object, the device can initiate an interaction with the object (e.g., selection). In this example, the device is able to detect the user's intention to select the object based on the user's physiological response (e.g., pupil dilation) before the user selects the object. Furthermore, because the object is intelligently configured to provide an appropriate or optimal physiological response, the device may be able to recognize such intent more accurately than it otherwise could (e.g., a user's intent to select a button may be more accurately predicted because the user interface feedback is pink rather than orange).

[0007] In some implementations, the device adjusts the visual feature by obtaining pupil dilation data corresponding to multiple variations of the visual feature (e.g., red highlighting, blue highlighting, and green highlighting), and then selecting the variation of the visual feature based on the pupil dilation data. As used herein, the phrase "optimized" generally refers to selecting a visual feature, for example, by selecting a particular variation of the visual feature, which is selected in any manner that is advantageous relative to at least one other variation of the visual feature. For example, the device may select green highlighting for the optimized visual feature based on a determination derived from the pupil dilation data that the user's eyes respond better to green than to red. In addition, based on the determination that the user responds better to green, the device may then adjust the visual feature and display the object with green highlighting. In some implementations, the device interprets an exogenous signal corresponding to a change in pupil diameter in the pupil diameter data, the change in pupil diameter being caused by a change in ambient light, a change in chromaticity, eye adaptation, a change in content lighting, periodic pupil dilation, a change in ambient noise, a change in motion of the device, brightness of the scene, cognitive load, or caffeine ingested by the user.

[0008] In some implementations, the device acquires physiological data (e.g., pupil data) associated with a user's response to the display of the adjusted visual feature, and determines the user's intent to interact with the object based on the acquired physiological data (e.g., a detected pupil dilation pattern). In some implementations, the device may identify a correlation between a change in the user's physiological data and the user's expectation that a particular visual feature will be displayed. For example, determining the user's intent to select an object may include identifying the user's expectation that feedback on the selection will have the adjusted visual feature. In addition, in some implementations, determining the user's intent includes applying a trained machine learning technique to identify a pattern in the physiological data that corresponds to the user's intent or the user's expectation. In some implementations, detecting the pupil dilation pattern may include interpreting an exogenous signal corresponding to a change in pupil diameter in the pupil diameter data, the change in pupil diameter being caused by a change in ambient light, a change in chromaticity, eye adaptation, a change in content lighting, periodic pupil dilation, a change in ambient noise, a change in motion of the device, brightness of the scene, cognitive load, or caffeine ingested by the user.

[0009] According to some specific implementations, a non-transitory computer-readable storage medium stores computer-executable instructions for performing or causing the performance of any of the methods described herein. According to some specific implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0011] Figure 1 Devices for displaying objects and acquiring physiological data from a user according to some implementations are shown.

[0012] Figure 2 According to some specific implementations Figure 1 The pupil of a user, where the diameter of the pupil changes over time.

[0013] Figure 3 is a flow chart representation of a method for adjusting visual features to enhance a user's pupil response.

[0014] Figure 4 is a flow chart representation of a method for selecting and displaying interface feedback features to enhance a physiological response associated with a user who desires the selected interface feedback features.

[0015] Figure 5 is a diagram illustrating a training process in which a plurality of alternative interface feedbacks are evaluated to select one interface feedback.

[0016] Figure 6 It is shown Figure 5 A chart showing the application of the selected interface feedback.

[0017] Figure 7 is a graph illustrating acquired physiological data over a period of time, including moments of intent, according to some implementations.

[0018] Fig. 8A is a graph illustrating detection of physiological data patterns according to some implementations.

[0019] Figure 8B are graphs illustrating physiological data and exogenous signals according to some specific implementations.

[0020] Fig. 9 is a block diagram illustrating device components of an exemplary device according to some implementations.

[0021] Fig.10 is a block diagram of an exemplary head mounted device (HMD) according to some implementations.

[0022] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0023] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0024] Figure 1A real-world environment 5 is shown that includes a device 10 having a display 15. In some implementations, the device 10 displays an object 20 to a user 25, and a visual feature 30 is associated with the object 20. For example, the object 20 can be a button, a user interface icon, a text box, a graphic, etc. In some implementations, the visual features 30 associated with the object 20 include visual features such as hue, saturation, size, shape, spatial frequency, motion, highlighting, etc. For example, the object 20 can be displayed with a green highlighted visual feature 30 covering or surrounding the object 20.

[0025] Device 10 acquires physiological data (e.g., pupil data) from user 25 via sensor 35. Although this example and other examples discussed herein show a single device 10 in a real-world environment 5, the techniques disclosed herein are applicable to multiple devices and other real-world environments. For example, the functionality of device 10 may be performed by multiple devices.

[0026] In some specific implementations, such as Figure 1 As shown, device 10 is a handheld electronic device (e.g., a smartphone or tablet computer). In some implementations, device 10 is a laptop or desktop computer. In some implementations, device 10 has a trackpad, and in some implementations, device 10 has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some implementations, device 10 is a wearable head-mounted display ("HMD").

[0027] In some implementations, device 10 includes an eye tracking system for detecting eye position and eye movement. For example, the eye tracking system may include one or more infrared (IR) light emitting diodes (LEDs), an eye tracking camera (e.g., a near infrared (NIR) camera), and an illumination source (e.g., a NIR light source) that emits light (e.g., NIR light) toward the eyes of user 25. In addition, the illumination source of device 10 may emit NIR light to illuminate the eyes of user 25, and the NIR camera may capture images of the eyes of user 25. In some implementations, the images captured by the eye tracking system may be analyzed to detect the position and movement of the eyes of user 25, or to detect other information about the eyes such as pupil dilation or pupil diameter. In addition, the gaze point estimated from the eye tracking image may enable gaze-based interaction with content shown on the near-eye display of device 10.

[0028] In some implementations, the device 10 has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, programs, or instruction sets for performing multiple functions stored in the memory. In some implementations, the user 25 interacts with the GUI through finger contacts and gestures on the touch-sensitive surface. In some implementations, these functions include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet production, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, and / or digital video playback. Executable instructions for performing these functions may be included in a computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0029] In some implementations, the device 10 employs various physiological sensors, detection or measurement systems. The detected physiological data may include, but is not limited to: electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG), functional near infrared spectroscopy signal (fNIRS), blood pressure, skin conductance, or pupil response. In addition, the device 10 may detect multiple forms of physiological data simultaneously in order to benefit from the synchronous collection of physiological data. In addition, in some implementations, the physiological data represents involuntary data, that is, responses that are not under conscious control. For example, a pupil response may represent an involuntary movement.

[0030] In some implementations, one or both eyes 45 of user 25 (including one or both pupils 50 of user 25) present physiological data in the form of pupil responses. The pupil responses of user 25 result in changes in the size or diameter of pupil 50 via the optic nerve and the oculomotor cranial nerve. For example, the pupil responses may include a constriction response (miosis), i.e., a narrowing of the pupil, or a dilation response (mydriasis), i.e., a widening of the pupil. In some implementations, device 10 may detect a pattern of physiological data representing time-varying pupil diameter.

[0031] Figure 2 Shows Figure 1 The pupil 50 of the user 25, wherein the diameter of the pupil 50 changes over time. Figure 2 As shown, the current physiological state (eg, current pupil diameter 55) may vary compared to a past physiological state (eg, past pupil diameter 60). For example, the current physiological state may include a current pupil diameter, and the past physiological state may include a past pupil diameter.

[0032] The physiological data may vary over time, and the device 10 may use the physiological data to measure one or both of a user's physiological response to the visual feature 30 or the user's intent to interact with the object 20. For example, when an object 20 such as a list of files is presented by the device 10, the user 25 may select the object 20 without requiring the user 25 to complete a physical button press. In some implementations, the physiological data includes a response of the radius of the pupil 50 of the user 25 after the user 25 glances at the object 20, measured via eye tracking technology.

[0033] The visual features 30 associated with the object 20 may be adjusted to drive or enhance the physiological response of the user 25. Such adjustments may be implemented for the object during a specific user interaction with the object or during future interactions with the object. For example, the object 20 may be highlighted in red when the user is looking at it, then highlighted in green, and then highlighted in yellow, and based on detecting that the user responds best to green, the object may remain green for the rest of the interaction. As another example, based on determining that the user reacts to green, the visual features may change over time accordingly. For example, the green highlight around the object 20 may be gradually adjusted during the user interaction to continuously induce a greater response. In some specific implementations, the animated object darkens, brightens or changes the brightness contrast, changes the hue, changes the spatial pattern, or moves in a certain way to drive the adjustment of the pupil 50. In addition, in some specific implementations, the visual features 30 associated with the object 20 are changed, for example, before the "moment" of performing the movement or intention, so that the user establishes an expectation that the interaction will cause the visual features 30 associated with the object 20 to change (e.g., animation).

[0034] Figure 3 1 is a flowchart representation of a method 300 for adjusting visual features associated with an object (e.g., object 20) to enhance a pupil response of a user (e.g., user 25) according to some implementations. In some implementations, the method 300 is performed by one or more devices (e.g., device 10). The method 300 may be performed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. The method 300 may be performed on an HMD having a screen for displaying 3D images or a screen for viewing stereoscopic images. In some implementations, the method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 300 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0035] At block 310, method 300 displays visual features associated with the object to a user on a display of the device. In some implementations, the object is a button, a user interface icon, a text box, or a graphic. Additionally, in some implementations, the visual features associated with the object are hue, saturation, size, shape, spatial frequency, motion, or highlighting.

[0036] At block 320, method 300 acquires physiological data associated with the user's pupil response to the visual feature displayed at block 310. In some implementations, the diameter of the pupil is measured over a period of time. In some implementations, method 300 performs a training function by presenting various visual features to the user and recording physiological data associated with the pupil response to the various visual features, respectively. Thus, method 300 is able to "learn" the specific variation of the visual feature to which the user's pupil is most responsive. In some implementations, method 300 accounts for any exogenous signals that may affect the pupil response. For example, increased ambient light may result in an exogenous signal that affects the pupil response.

[0037] At block 330, method 300 adjusts the visual feature based on the physiological data acquired at block 320 to enhance the user's pupillary response to the object. In some implementations, the visual feature is changed to provide an increased pupillary response. For example, the visual feature may include a highlight of a particular color; the adjustment of the visual feature may include changing the color of the highlight to a determined color such that the color induces an enhanced pupillary response relative to the pupillary response to the original color of the visual feature.

[0038] At block 340, method 300 displays the object with the enhanced visual feature. In some implementations, method 300 identifies the user's interest in the object based on detecting a pattern in the acquired physiological data. In some implementations, the current interest in the intent is confirmed based on the user's pupil response to the enhanced visual feature.

[0039] Figure 4 4 is a flowchart representation of a method 400 for selecting and displaying interface feedback features to enhance a physiological response associated with a user who desires the selected interface feedback features according to some specific implementations. In some specific implementations, the method 400 is performed by one or more devices. The method 400 can be performed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. The method 400 can be performed on an HMD having a screen for displaying 3D images or a screen for viewing stereoscopic images. In some specific implementations, the method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 400 is performed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0040] At box 410, method 400 uses multiple alternative interface feedback features to obtain physiological responses associated with a user who expects interface feedback after interacting with an object in a user interface. In some specific implementations, method 400 presents specific interface feedback to the user multiple times and, in some embodiments, the specific interface feedback can be repeatedly presented to the user until the user's physiological response to the presentation of the object shows that the user expects the specific interface feedback to occur after interacting with the object. For example, the user can see a button and can exhibit a baseline or "original" pupil diameter of 2mm. After interacting with the object, the user may be surprised by the interface feedback of a color change in the color of the button (e.g., the color changes from blue to pink). After interacting with the object multiple times (e.g., statistically significant sampling of physiological data, or until the physiological data stabilizes), the user may expect interface feedback and when the user sees the button, for example, when the user intends to interact with the button but before interacting with the button, may exhibit an excited pupil diameter of 4mm. Furthermore, in some implementations, additional interface feedback features may be presented to the user so that the acquired user physiological response reflects the user's response to multiple different interface feedback features (e.g., color change to a different color, highlighting, audio feedback, etc.).

[0041] At box 420, method 400 evaluates the physiological response of the user of the desired interface feedback obtained at box 410. In some specific implementations, method 400 estimates the physiological response based on each different interface feedback feature or based on the number of physiological responses obtained for each different feedback feature. For example, when the user expects the object to turn pink after being clicked, method 400 can estimate the degree of dilation of the user's eyes before clicking the object, and when the user expects the object to turn orange after being clicked, method 400 can estimate how much the user's eyes dilated before clicking the object. In addition, method 400 can estimate how quickly the user is accustomed to expecting each feedback feature based on one or more factors (e.g., the number of physiological responses collected required for a stable response).

[0042] At box 430, method 400 selects one of the alternative interface feedback features based on the physiological data evaluation performed at box 420. In some implementations, method 400 seeks to maximize the difference between the user's physiological response to the presentation of the object when the user expects interface feedback and the user's original physiological response to the presentation of the object. For example, when the user expects the object to turn pink after being clicked, method 400 may estimate how much the user's eyes dilated before the user clicked the object relative to the user's baseline or original pupil diameter, and the pink change estimate may be compared with an estimate of how much the user's eyes dilated before clicking the object when the user expects the object to turn orange after being clicked, etc. In addition, method 400 may determine that the pink change estimate may cause the largest pupil diameter change between the user's original / baseline pupil diameter and the user's pupil diameter when the user expects the object to turn pink. Therefore, method 400 may select pink as the ideal interface feedback feature.

[0043] At block 440, method 400 displays the object so that selection of the displayed object can provide interface feedback using the interface feedback feature selected at block 430. For example, after training of the user or device, the object can be displayed in the application using pink interface feedback. In some implementations, before providing interface feedback using the selected interface feedback feature, method 400 obtains a user's physiological response to the displayed object before it is selected, determines the user's intent to interact with the object based on the obtained physiological response, and initiates interaction with the object (e.g., selection) based on determining the user's intent to interact with the object.

[0044] Figure 5 A training process is shown in which multiple alternative interface feedbacks are evaluated to select interface feedback. In some implementations, when a user expects to obtain interface feedback or stimulation (e.g., clicking a button) when interacting with an object, the interface feedback is selected based on an evaluation of the user's physiological response to the presentation of the object or stimulation (e.g., a button). For example, the training process may involve performing a series of trials (e.g., Trial 1 to Trial N) for each alternative interface feedback, wherein the device effectively learns the degree to which the user's expected pupil diameter responds to the post-click interface feedback.

[0045] In some implementations, a training process including multiple trials is performed for each interface feedback condition. For example, the training process may include identifying an ideal color to change a button after a user clicks the button. Thus, from Trial 1 to Trial N, the interface feedback condition of the button may be a color change from an initial color (e.g., pure blue) to another color (e.g., pure pink), and in another set of trials (e.g., 10 trials), the interface feedback condition may include the button changing from the initial color to a different color (e.g., pure orange). When the user interacts (e.g., clicks) with an object (e.g., a button) in each trial (e.g., Trial 1 to Trial N), the user repeatedly experiences the interface feedback condition and develops an expectation of obtaining the interface feedback condition.

[0046] In some implementations, during the trial, when the user expects to obtain interface feedback after interacting with the object, the device determines the user's physiological response before the user interacts with the object. For example, the training process can form a correlation between the acquired physiological data (e.g., pupil diameter) and the expectation of a visual feature (e.g., color feedback, such as highlighting or color change) associated with the object (e.g., a stimulus, such as a button). In some implementations, the user is accustomed to expecting a change in the visual feature associated with the object, and the device detects a change in the user's physiological state based on the accustomed expectation.

[0047] like Figure 5 As shown in the example of , a first series of trials may present a blue button to the user that turns pink after the user clicks it. Prior to clicking the button in Trial 1, the user may present a baseline physiological characteristic (e.g., an original pupil diameter of 2 mm). As the user performs additional trials, the user may develop an expectation of the interface feedback condition, and when the object (e.g., the button) is presented, the user may present a physiological response (e.g., an excited or expected pupil diameter) that reflects the expectation of the interface feedback condition. For example, in Trial N, because the user expects the button to turn pink after being clicked, the user may present an excited pupil diameter of 4 mm prior to interacting with the button.

[0048] Thus, in some implementations, the device conditions the user to exhibit a prominent physiological response when expecting a visual feature. Figure 5 As shown in Trial N, after N repetitions, the user has become accustomed to expecting the visual feature, and when the stimulus is obtained, the user exhibits a prominent physiological response (e.g., a pupil diameter of 4 mm) in anticipation of interacting with the stimulus.

[0049] In some embodiments, an additional series of trials is performed to determine which interface feedback feature elicits a significant or optimal physiological characteristic response from a user who expects to interact with an object (e.g., prior to interacting with the object). For example, a second series of trials may be performed in which a button changes from blue to orange after the user clicks it, and the system may determine through trial N that the user presents a pupil diameter of 3 mm before interacting with the button. In some embodiments, over the course of multiple trials, the device learns via a training process to select an ideal or optimal interface feedback (e.g., an interface feedback feature that elicits the strongest physiological response to presentation of an object prior to interacting with the object). For example, by comparing expected pupil diameters (e.g., a pupil diameter of 4 mm for a change from blue to pink, and a pupil diameter of 3 mm for a change from blue to orange), the device may determine to use pink highlighting for interface feedback to elicit a larger expected pupil diameter for the user.

[0050] Figure 6 Shows Figure 5 In some implementations, by continuing to present the interface feedback to the user, the device may determine the user's intent to interact with the object based on the user's physiological response to the presentation of the object. For example, the system may identify that the user intends to interact with the object based on whether the user presents a physiological characteristic associated with the user's expectation that the interface feedback condition will occur. Thus, in some implementations, the device may perform an interaction on behalf of the user based on the identified user's expectation of the interface feedback condition, rather than physically interacting with the object.

[0051] In some implementations, while the user is using the device, the device continues to use the selected interface feedback when the user interacts with the object, and also predicts the user's expectation or intention to interact with the object (e.g., click on the object) based on the physiological response to the object presented to the user (e.g., excited pupil diameter). In addition, by utilizing interface feedback (e.g., interface feedback that causes the largest pupil diameter when the interface feedback is expected), the device's ability to identify predicted physiological responses can be enhanced. For example, once an interface feedback of a button changing from blue to pink is selected, the device can adopt the interface feedback so that after the user clicks the button, the button changes from blue to pink; then, the device can measure the user's pupil diameter to predict the button click based on the user's expected pupil diameter.

[0052] Thus, by identifying physiological data associated with expectations for visual features, the device can identify the user's salient physiological state when presenting a stimulus and predict that the user intends to interact with the stimulus. For example, a user may be accustomed to expecting a change in a visual feature associated with an object 20 (e.g., a button) (e.g., a change in color), and may present a change in physiological state based on that expectation. In addition, the device may automatically perform an interaction with an object in response to the identified user's intent or expectation. For another example, if a user has been accustomed to expecting the dial button on his phone to change from gray to green after being clicked, and the phone has identified that the user exhibits an excited pupil diameter of 3 mm when the dial button is expected to turn green (e.g., before selecting the dial button), the phone can use the detection of an excited pupil diameter of 3 mm to make a call and display the feature of the dial button turning green (e.g., without the user physically clicking the dial button).

[0053] In some implementations, the object also includes auditory attributes that are tuned to evoke responses such as greater physiological intent that can be used to predict upcoming stimulus changes. For example, even if the object is visible, it can also have auditory features (or other sensory properties such as touch, smell, etc.) that can also change after selection (e.g., emitting sound characteristics, vibrations, special smells, etc.). In addition, because the object is defined spatially, in some implementations, spatial audio rendering is used.

[0054] In some implementations, the device obtains physiological data from the user based on identifying typical interactions of the user with the object. For example, the device may determine that the user's pupil response is associated with the interaction with the object; then, the device may adjust the visual features of the object to enhance the pupil response associated with future interactions with the object. In addition, in some implementations, changing the visual features after the user interacts with the object informs the user's physiological response in subsequent interactions with the object. For example, before the visual features are changed, the user may present an expected physiological response associated with the change in the visual features. Therefore, in some implementations, the device identifies the user's intention to interact with the object based on the expected physiological response. For example, the device can adapt or train itself by capturing or storing the user's pupil response based on the user's interaction with the object (including the user's response to the enhanced visual features of the object), and can detect the user's future intention to interact with the object by identifying the user's physiological response when the enhanced visual features are expected to be displayed (e.g., before the interaction with the object).

[0055] In some implementations, an estimator or statistical learning method is used to better understand or predict physiological data (eg, pupil diameter). For example, statistics of pupil response data can be estimated by sampling a data set with replacement (eg, bootstrapping).

[0056] Figure 7 7 is a graph 700 showing pupil response over time, including a baseline time-varying pattern 710 of physiological data, such as self-image pupil diameter (y-axis) over time (x-axis). In some implementations, device 10 may adjust the degree to which visual features 30 associated with object 20 are altered or adjusted to increase the signal-to-noise ratio found in the adjustment of pupil 50 at the beginning of an interaction. For example, the enhanced pupil signal may be thresholded or compared to a baseline pupil response (e.g., baseline time-varying pattern 710 of physiological data) by a binary classifier to detect intent for a particular item or object.

[0057] In some implementations, the feedback varies continuously over time, such that the intensity of the visual feature 30 is adjusted according to the presented pupil diameter 55, and in some implementations, the continuous feedback signal promotes error-driven learning to improve the intentional control using the pupil 50. For example, a set of predefined feedback parameters may be provided at initialization, or the device 10 may be trained within a specific user experience, wherein the predefined feedback parameters are determined to provide a more significant pupil response. In addition, limits may be defined that keep the visual appearance and feel of the visual feature 30 within an acceptable range defined by the designer so that the object 20 or the visual feature 30 does not enter an undesirable feedback loop. For example, by associating an undesirable visual feature 30 with an object, such as a fast flash or some other undesirable animation, the defined limits may prevent the device 10 from maximizing pupil accommodation. In some implementations, the device 10 may be trained with multiple sets of user physiological data and then adapted to each user individually.

[0058] In some implementations, device 10 will also account for user 25's real-world environment 5 (e.g., visual qualities such as brightness, contrast, semantic context) when evaluating how much to adjust or tweak visual feature 30 to enhance user 25's physiological response (e.g., pupil response) to visual feature 30. Figure 7 As shown, the exemplary time-varying pattern 730 of physiological data exhibits a peak 750 that coincides with the moment 720 of user intent (e.g., selection). For example, the moment 720 of user intent may coincide with a "click" performed by the user 25. Similarly, Figure 7 Also included is an exemplary time-varying pattern 740 of physiological data including a peak 760 at the time 720 indicating user intent.

[0059] In addition, the time-varying pattern 730 represents a user response to a visual feature 30 of a particular type (e.g., red highlight), and the time-varying pattern 740 represents a user response to a visual feature 30 of a different type (e.g., green highlight), so that the physiological data detected associated with the time-varying pattern 740 is greater than the physiological data associated with the time-varying pattern 730, for example, the user exhibits more physiological responses in the time-varying pattern 740 than in the time-varying pattern 730. For example, the time-varying pattern 730 may be recorded in response to a visual feature of a particular color (e.g., green), and the time-varying pattern 740 may be recorded in response to a visual feature of a different color (e.g., red). Therefore, different visual features may cause physiological responses that are more differentiated from baseline physiological responses. The time-varying patterns 730 and 740 also show a time period in which the physiological response increases after the intended moment 720. In some implementations, this increase in physiological response is based on a change, adjustment, or enhancement made to visual feature 30, so that visual feature 30 can be used to drive an increase in a time-varying pattern (e.g., time-varying pattern 730 or time-varying pattern 740) relative to a baseline response (e.g., baseline time-varying pattern 710). If it has been previously determined that the user responds best to the color green, the visual feature can drive an increase in the diameter of the user's pupil, for example, by gradually changing from yellow to green.

[0060] like Fig. 8A and Figure 8B As shown, in some implementations, the physiological data may change over time, and the device 10 may use the physiological data to detect a pattern. In some implementations, the pattern is a change in the physiological data from one time to another, and in some other implementations, the pattern is a series of changes in the physiological data over a period of time. Based on the detection of the pattern, the device 10 may identify the interest or intention of the user 25 to interact with the object 20, and may adjust the visual features 30 to further identify the interest or intention of the user 25. For example, the intention of the user 25 to select the object 20 may be identified by detecting the pattern of the user's pupil response, the visual features 30 associated with the object 20 may be adjusted (e.g., changing the background of the object from blue to green), and the user's pupil response to the adjusted visual features 30 may be used to confirm the user's intention to select the object 20. In addition, the device 10 may identify a pattern associated with the user 25's interaction with the object 20, and may adjust the visual features 30 to better facilitate future recognition of the pattern associated with the user 25's interaction with the object 20.

[0061] Fig. 8A8 is a graph 800 illustrating detection of a pattern of physiological data. Graph 800 illustrates a time-varying pattern 805 of physiological data, such as the amount of pupil dilation (y-axis) over time (x-axis). Based on displayed visual features 30 associated with object 20, pattern 805 may include a peak pattern 810 that may be interpreted by device 10 as an indication of user 25's interest in or intent with respect to object 20. In some implementations, device 10 utilizes a trained model to determine that peak pattern 810 indicates that user 25 is involuntarily signaling user 25's interest or intent during the time of peak pattern 810, for example, based on what the user is viewing or otherwise experiencing during that time period.

[0062] In some implementations, the device 10 utilizes a training or calibration sequence to adapt to the specific physiological characteristics of a particular user 25. In some implementations, the device 10 presents a training scenario to the user 25 in which the user 25 is instructed to interact with a screen item (e.g., an object 20). By providing the user 25 with a known intent or region of interest (e.g., via instructions), the device 10 may record the user's physiological data (e.g., pupil response) and identify a pattern associated with the user's intent or interest 40. In some implementations, the device 10 changes the visual features 30 associated with the object 20 to further adapt to the unique physiological characteristics of the user 25. For example, the device 10 may instruct the user to subjectively select a button in the center of the screen while counting to three, and record the user's physiological data 45 to identify a pattern associated with the user's intent or interest 40. In addition, the device 10 may change or alter the visual features associated with the button in order to identify a pattern associated with the user's physiological response to the altered visual features. In some implementations, the patterns associated with the physiological responses of user 25 are stored in a user profile associated with the user, and the user profile can be updated or recalibrated at any time in the future. For example, the user profile can be automatically modified over time during the user experience to provide a more personalized user experience.

[0063] In some implementations, a machine learning model (eg, a trained neural network) is applied to recognize patterns in physiological data, including identifying physiological responses to visual features 30 associated with subject 20 .

[0064] In addition, the machine learning model can be used to match these patterns with indications corresponding to the user 25's interest or intent in interacting with the object 20. In some implementations, the device 10 can learn patterns that are specific to a particular user 25. For example, the device 10 can begin learning by determining that a peak pattern 810 represents an indication of the user 25's interest or intent in responding to a particular visual feature 30, and use this information to subsequently identify a similar peak pattern 820 as another indication of the user's 25 interest or intent. Such learning can take into account the user's relative interactions with multiple visual features 30 in order to further adjust the visual features 30 and enhance the user's physiological response.

[0065] Figure 8B 8 is a graph 850 showing physiological data (e.g., based on measured pupil dilation) and exogenous signal 860 (e.g., a measure of ambient light at device 10). Exogenous signal 860 corresponds to a measure of any factor that may affect the physiological data. For example, the amount of ambient light at device 10 may result in a change in the amount of dilation of the user's eyes, e.g., a decrease in ambient light may result in more dilation of the eyes, etc.

[0066] In some implementations, physiological data is adjusted or otherwise interpreted based on the exogenous signal 860. For example, a peak pattern 810 corresponding to a dilation of the user's eyes can be preliminarily interpreted as an indication of the user relative to an object given the visual characteristics. Because the exogenous data 860 is horizontal during the time period of the peak 810 (e.g., indicating a constant ambient light level), a determination is accepted that the dilation should be interpreted as an indication of intent or interest.

[0067] In contrast, peak pattern 890 corresponding to dilation of the user's eyes may similarly be initially interpreted as an indication of intent or interest, but this determination may be rejected. In this example, exogenous signal 860 indicates an increase 870 and subsequent decrease 880 within the same time period as peak pattern 890. Thus, the exogenous signal may correspond to an exogenous factor (rather than the interest or intent of user 25) that caused peak pattern 890. Therefore, device 10 may reject the initial interpretation of peak pattern 890 as an indication of intent or interest. In some implementations, a model is used to interpret the exogenous signal, for example, the model is trained to interpret patterns in the physiological data that appear during the same time period as the patterns in the exogenous signal.

[0068] In some embodiments, the exogenous signal corresponding to the change in pupil diameter is caused by a change in ambient light, a change in chromaticity, eye adaptation, a change in content lighting, periodic pupil dilation, a change in ambient noise, or a change in the movement of the device. For example, an increase in ambient light may correspond to a decrease in pupil diameter. Similarly, a color change or change in content lighting may correspond to an increase or decrease in pupil diameter. In addition, the exogenous signal may also be related to the use of drugs or medications. For example, anesthetic preparations or opioids may be associated with a reduction in pupil size. In some embodiments, based on the pupil response, the device 10 detects patterns corresponding to one or more exogenous factors, for example, based on learning such patterns using a machine learning model. The device 10 can distinguish patterns of physiological data corresponding to interest or intent from patterns of physiological data corresponding to exogenous factors, for example, based on learning such distinction using a machine learning model.

[0069] In some implementations, device 10 detects the position of eyes 45 of user 25 and the position of pupil 50 of user 25, for example, by processing and analyzing images of light reflected from one or both eyes (typically infrared and / or colors produced by a red-green-blue-additive color model) in order to locate and measure the diameter of the pupil. The reflected light may originate from a light projection source of device 10, or any other natural (e.g., sunlight) or artificial (e.g., lamp) light source. Using techniques such as detecting pupil center corneal reflection (PCCR), device 10 may process and analyze images of light reflected from elements of eye 45 (including pupil 50) in order to determine the diameter of pupil 50. In addition, device 10 may process light reflected from eyes 45 of user 25 (e.g., from an illumination source on the device or elsewhere) as flash.

[0070] In some implementations, the position and features of the user's 25 head 27 (e.g., the edges of the eyes, nose, or nostrils) are extracted by the device 10 and used to find the rough position coordinates of the user's 25 eyes 45, thereby simplifying the determination of precise eye 45 features (e.g., position, gaze direction, etc.) and making pupil diameter 55 measurements more reliable and robust. In addition, the device 10 can easily combine the position of the 3D components of the head 27 with the gaze angle information obtained through the eye component image analysis to identify a given screen object viewed by the user 25 at any given time. In some implementations, the use of 3D mapping in conjunction with gaze tracking allows the user 25 to freely move his head 27 and eyes 45 while reducing or eliminating the need to actively track the head 27 using sensors or transmitters on the head 27.

[0071] By tracking the eyes 45, some implementations reduce the need to recalibrate the user 25 after the user 25 moves his head 27. In some implementations, the device 10 uses depth information to track the movement of the pupil 50, thereby enabling a reliable rendered pupil diameter 55 to be calculated based on a single calibration of the user 25. Using techniques such as pupil center corneal reflection (PCCR), pupil tracking, and pupil shape, the device 10 can calculate the pupil diameter 55 and the gaze angle of the eye 45 from a fixed point of the head 27, and use the position information of the head 27 to recalculate the gaze angle. In addition to reduced recalibration, further benefits of tracking the head 27 can include reducing the number of light projection sources and reducing the number of cameras used to track the eyes 45.

[0072] Because in some implementations, user intent is automatically determined based on user 25's eyes 45, selections and other interactions can be facilitated without necessarily requiring user 25 to take action on a physical input device. For example, conventional methods of navigating in a user interface can be improved by providing device 10 with faster, more efficient methods and interfaces for navigating in a user interface (e.g., object 20 can be a selectable button of a user interface). In addition, such methods and interfaces can reduce the cognitive burden imposed on user 25 and produce a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces can save power and increase the time interval between battery charges. In addition, some implementations enhance navigation of a user interface based on detecting patterns associated with physiological data.

[0073] According to some implementations, a user interface with one or more selectable objects is displayed on a screen of device 10, and the interest or intention of user 25 is associated with one of these selectable objects. Furthermore, in some implementations, the interest or intention of user 25 is associated with selecting one of the selectable objects. In some implementations, device 10 collects physiological data corresponding to the user's interest or intention to interact with object 20. Furthermore, device 10 may then enhance visual features 30 associated with object 20.

[0074] In some implementations, the device 10 receives the gesture as voluntary data (e.g., an action that the user 25 can control). For example, the voluntary data may be received based on the user's voice input, gesture, touch input, key press, etc. In some implementations, the user's interest or intent to interact with the object 20 is associated with multiple types of inputs (i.e., multimodal) communicated by the user 25 and the device 10. For example, more than one low-commitment voluntary interaction may be associated with the user's 25 interest or intent in combination.

[0075] A faster, more efficient method and interface is provided for computing devices for navigating user interfaces, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace conventional methods for navigating user interfaces.

[0076] In some implementations, device 10 uses the detected physiological data patterns to control functions of device 10. In some implementations, device 10 identifies a particular object 20 presented on the device's display 15 (e.g., at a location in the direction of a user's gaze), and responsively changes the state of object 20 based on the identified interest or intent of user 25.

[0077] In some implementations, changing the state of a given interactive item includes performing an operation associated with the given interactive item. For example, an interactive item may include a menu option that a user 25 may select to present specific content (e.g., a movie or television program) on display 15. In some implementations, device 10 changes the state of a given interactive item by directing input received from user 25 to the given interactive item. In some implementations, device 10 identifies a given interactive item based on other data and interacts with the interactive item based on the identified user's interest or intent. For example, if user 25 is looking at an optional button, device 10 may identify the optional button based on the user's gaze, and then select the optional button based on the identified user's interest or intent.

[0078] In some implementations, the device 10 identifies an object 20 presented on the display 15 of the device 10 at a location in the direction of the user's gaze. In addition, the device 10 may change the state of the visual feature 30 associated with the object 20 in response to the verbal speech command received from the user 25 and the identified interest or intent of the user 25. For example, the object 20 may be an icon associated with a software application, and the user 25 may gaze at the icon, speak the word "select" to select the application, and a highlight effect may be applied to the icon. The device 10 may then use other physiological data in response to the visual feature 30 to further identify the interest or intent of the user 25 as a confirmation of the user's verbal command. In some implementations, the device 10 is configured to identify a given interactive item in response to the direction of the user's gaze, and manipulate the given interactive item in response to physiological data (e.g., pupil data). The device 10 may then confirm the direction of the user's gaze based on further identifying the user's interest or intent with the physiological data. In some implementations, the device 10 removes the interactive item or object based on the identified interest or intent. In other implementations, device 10 automatically captures images of content upon determining the interest or intent of user 25 .

[0079] In some implementations, the device 10 is configured to provide a progressive interface. The progressive interface can facilitate the availability of voluntary input from the user 25 combined with physiological data (e.g., one or more involuntary features of the user 25) or supported by physiological data. For example, by adjusting the visual features 30, the device 10 can provide the user 25 with progressive feedback on the identified interest or intention of the user 25, while causing an enhanced physiological response to the adjustment of the visual features 30. In some implementations, the device 10 begins a low-commitment interaction with the user 25 in response to detecting a pattern of physiological data or in response to receiving voluntary user input. For example, in response to one or more low-confidence detections of a user's interest or intention to interact with the object 20, the device 10 can modify or adjust the visual features associated with the object 20 (e.g., highlighting the object with an optimal color). In some implementations, the modification of the visual features 30 can instruct the user 25 to progressively perform higher-commitment actions to confirm the user's interest or intention to interact with the object 20. For example, the device 10 can execute or delete an item in response to further input from the user 25.

[0080] As a power saving feature, the device 10 may detect when the user 25 is not looking at the display and the device 10 may activate power saving techniques, such as disabling physiological sensors when the user 25 looks away for more than a threshold period of time. In addition, in some implementations, when the user 25 is not looking at the display, the device 10 dims or completely blacks out the display (e.g., reduces the brightness). When the user 25 looks at the display again, the device 10 may deactivate the power saving techniques. In some implementations, the device 10 tracks the physiological attribute using a first sensor and then activates a second sensor based on the tracking to obtain physiological data. For example, the device 10 may use a camera to identify that the user 25 is looking in the direction of the device 10, and then activate an eye sensor when it is determined that the user 25 is looking in the direction of the device 10.

[0081] In some implementations, a combination of determining user intent and input device 10 is used to create an interactive user interface that utilizes input device 10 to identify screen interactive items and determine the user's interest or intent in interacting with the screen interactive items. For example, user 25 can use a mouse to select an interactive item on the screen based on the user's determined interest or intent, such as a "mouse click" type event triggered by determining the user's interest or intent rather than a mouse click. In some implementations, a combination of determining user intent and gaze tracking is used to create an interactive user interface that can detect which screen interactive item the user 25 is looking at (e.g., gaze tracking) and determine the user's interest or intent in interacting with the screen interactive item, thereby avoiding the need for a mouse and / or keyboard.

[0082] In addition, the combination of determining user interests or intentions with other modalities (such as gaze tracking, facial gesture detection, 3D mapping / gesture detection, and / or voice detection) enables user 25 to fully control screen objects without using a mouse or touch screen. In this way, user 25 can perform a full range of pointing and selection functions, including searching a large number of information items and selections. The combined interface mode can also be used to search and perform control functions in the context of an interactive item, such as performing find, cut, copy, and paste functions within an open file. In some embodiments, device 10 identifies a first interest or a first interest group, and then progressively identifies a second interest or a second interest group based on previously identified interests.

[0083] In some implementations, the device 10 utilizes a training or calibration sequence to adapt to the specific physiological characteristics of a particular user 25. In some implementations, the device 10 presents a training scenario to the user 25 in which the user 25 is instructed to interact with a screen item. By providing the user 25 with one or more known visual features 30 associated with a known object 20, the device 10 can record the user's physiological data and identify patterns associated with the user's intent or interest. For example, the device 10 can instruct the user 25 to focus on a red button in the center of the display 15 while counting to three, and record the user's physiological data to identify patterns associated with the user's intent or interest when the green visual feature 30 is involved. The device 10 can then repeat the same process using a plurality of other buttons of different colors in order to identify the color that elicits the strongest physiological response from the user 25. In some implementations, patterns associated with the user's intent or interest, physiological data associated with a particular visual feature 30, or rankings or analyses of a particular visual feature 30 relative to the user 25 are stored in a user profile associated with the user, and the user profile can be updated or recalibrated at any time in the future. For example, a user profile may be automatically modified over time during the user experience to provide a more personalized user experience.

[0084] Fig. 9is a block diagram of an example of a device 10 according to some implementations. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 10 includes one or more processing units 902 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 906, one or more communication interfaces 908 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 910, one or more displays 912, one or more internally-facing and / or externally-facing image sensors 914, memory 920, and one or more communication buses 904 for interconnecting these and various other components.

[0085] In some implementations, one or more communication buses 904 include circuits that interconnect and control communications between system components. In some implementations, one or more I / O devices and sensors 906 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0086] In some implementations, one or more displays 912 are configured to present a user experience to the user 25. In some implementations, one or more displays 912 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), retinal projection system and / or similar display types. In some implementations, one or more displays 912 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, the device 10 includes a single display. For another example, the device 10 includes a display for each eye of the user 25, such as an HMD. In some implementations, one or more displays 912 can present MR content, including VR or AR content.

[0087] In some implementations, the one or more image sensor systems 914 are configured to acquire image data corresponding to at least a portion of the face of the user 25, including the eyes of the user 25. For example, the one or more image sensor systems 914 include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), a monochrome camera, an IR camera, an event-based camera, etc. In various implementations, the one or more image sensor systems 914 also include an illumination source that emits light on a portion of the face of the user 25, such as a flash or a strobe light source.

[0088] The memory 920 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 920 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 920 optionally includes one or more storage devices located away from the one or more processing units 902. The memory 920 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 920 or the non-transitory computer-readable storage medium of the memory 920 stores the following programs, modules and data structures or their subsets, including an optional operating system 920 and a user experience module 940.

[0089] The operating system 930 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the user experience module 940 is configured to present a user experience that utilizes physiological data via one or more input / output (I / O) devices and sensors 906 to identify the interests or intentions of the user 25 by performing context-aware dynamic distortion correction on the user 25. To this end, in various implementations, the user experience module 940 includes a physiological feature tracking unit 942, a presentation unit 944, and a visual feature adjustment unit 946.

[0090] In some implementations, the physiological feature tracking unit 942 is configured to acquire physiological data (e.g., pupil dilation, electroencephalogram, etc.) and use the acquired physiological data to identify patterns in the physiological data. To this end, in various implementations, the physiological feature tracking unit 942 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0091] In some implementations, the presentation unit 944 is configured to present objects associated with the visual features via the one or more displays 912. To this end, in various implementations, the presentation unit 944 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0092] In some implementations, the visual feature adjustment unit 946 is configured to adjust the visual feature to enhance a physiological response (e.g., a pupil response) to the adjusted visual feature. To this end, in various implementations, the visual feature adjustment unit 946 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0093] Although physiological feature tracking unit 942, rendering unit 944, and visual feature adjustment unit 946 are shown as residing on a single device (eg, device 10), it should be understood that in other implementations, any combination of these units may be located in separate computing devices.

[0094] also, Fig. 9 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 7 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0095] Fig.10 A block diagram of an exemplary head mounted device 1000 is shown according to some implementations. The head mounted device 1000 includes a housing 1001 (or enclosure) that houses various components of the head mounted device 1000. The housing 1001 includes (or is coupled to) eye pads (not shown) disposed at the proximal (of the user 25) end of the housing 1001. In various implementations, the eye pads are plastic or rubber pieces that comfortably and snugly hold the head mounted device 1000 in place on the face of the user 25 (e.g., around the eyes of the user 25).

[0096] Housing 1001 houses display 810, which displays images, emits light toward or onto the eyes of user 25. In various implementations, display 1010 emits light through an eyepiece having one or more lenses 1005, which refracts the light emitted by display 1010, so that the display appears to user 25 as a virtual distance farther than the actual distance from the eye to display 1010. In order to enable user 25 to focus on display 1010, in various implementations, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 7 cm). In addition, in order to provide a better user experience, in various implementations, the virtual distance is greater than 1 meter.

[0097] The housing 1001 also houses a tracking system that includes one or more light sources 1022, a camera 1024, and a controller 1080. The one or more light sources 1022 emit light onto the eyes of the user 25, which is reflected as a light pattern (e.g., a flashing circle) that can be detected by the camera 1024. Based on the light pattern, the controller 880 can determine the eye tracking characteristics of the user 25. For example, the controller 1080 can determine the gaze direction and / or blink state (eyes open or closed) of the user 25. As another example, the controller 1080 can determine the pupil center, pupil size, or focus point. Therefore, in various specific implementations, light is emitted by the one or more light sources 1022, reflected from the eyes of the user 25, and detected by the camera 1024. In various specific implementations, the light from the eyes of the user 25 reflects from the hot mirror or passes through the eyepiece before reaching the camera 1024.

[0098] The display 1010 emits light within a first wavelength range, and the one or more light sources 1022 emit light within a second wavelength range. Similarly, the camera 1024 detects light within the second wavelength range. In various implementations, the first wavelength range is a visible wavelength range (e.g., a wavelength range of approximately 400-700 nm within the visible spectrum), and the second wavelength range is a near infrared wavelength range (e.g., a wavelength range of approximately 700-1400 nm within the near infrared spectrum).

[0099] In various implementations, eye tracking (or specifically, determined gaze direction) is used to enable user interaction (e.g., user 25 selecting an option on display 1010 by looking at it), provide perforated rendering (e.g., presenting a higher resolution in the area of ​​display 1010 that user 25 is looking at and a lower resolution elsewhere on display 1010), or correct distortion (e.g., for an image to be provided on display 1010).

[0100] In various implementations, one or more light sources 1022 emit light toward the eyes of user 25, which is reflected in the form of multiple flashes.

[0101] In various implementations, camera 1024 is a frame / shutter based camera that generates images of the user's 25 eyes at a particular point in time or multiple points in time at a frame rate. Each image includes a matrix of pixel values ​​corresponding to pixels of the image, the pixels corresponding to locations of the camera's light sensor matrix. In a specific implementation, each image is used to measure or track pupil dilation by measuring changes in pixel intensity associated with one or both of the user's pupils.

[0102] In various implementations, camera 1024 is an event camera that includes multiple light sensors (e.g., a light sensor matrix) at multiple corresponding locations that generate event messages indicating specific locations of specific light sensors in response to the specific light sensors detecting changes in light intensity.

[0103] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0104] As described above, one aspect of the present technology is to collect and use physiological data to improve the user's electronic device experience in interacting with electronic content. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a specific person or can be used to identify the interests, characteristics, or tendencies of a specific person. Such personal information data may include physiological data, demographic data, location-based data, phone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0105] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the interaction and control capabilities of electronic devices. Therefore, the use of such personal information data enables planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0106] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for the privacy and security of maintaining personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of entities, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any required steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0107] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively prevent the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow anonymous information to be transmitted for use in improving the functionality of the device.

[0108] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences or settings based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0109] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data may be stored anonymously (e.g., without identifying and / or personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user may access their stored data from a user device that is different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access their stored data.

[0110] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other cases, methods, devices, or systems known to those of ordinary skill in the art are not described in detail in order to avoid obscuring the claimed subject matter.

[0111] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0112] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0113] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined or divided into sub-boxes. Some boxes or processes can be executed in parallel.

[0114] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0115] It will also be understood that, although the terms "first", "second", etc. may be used to describe various objects in this article, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0116] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms "a", "an" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "or" used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the term "comprises" or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, objects or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, objects, components or groups thereof.

[0117] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0118] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A biofeedback method, comprising: At a device comprising a processor, a computer-readable storage medium, a display, and a sensor: displaying visual features associated with the object to a user on the display; acquiring, using the sensor, first physiological data associated with a first pupillary response of the user to the visual feature; determining the user's interest in interacting with the object, wherein determining the user's interest in interacting with the object comprises detecting a pupil dilation pattern by considering an exogenous signal corresponding to a pupil diameter change based on the obtained first physiological data; determining that adjusting a visual feature associated with the object will produce a pupil response greater than the first pupil response; adjusting the visual feature based on the acquired first physiological data to enhance a pupil response of the user relative to the adjusted visual feature associated with the object in response to determining that adjusting the visual feature will produce a pupil response greater than the first pupil response; as well as The object is displayed with the adjusted visual features.

2. The method according to claim 1, further comprising: acquiring second physiological data associated with a second pupillary response of the user to the adjusted visual feature; as well as The user's intention to interact with the object is determined based on the acquired second physiological data. 3 . The method according to claim 1 , wherein the first physiological data comprises a pupil diameter of an eye of the user or a change in the pupil diameter of the eye of the user.

4. The method of claim 1 , wherein adjusting the visual feature comprises: obtaining pupil dilation data corresponding to each of a plurality of variations of the visual feature; as well as One of the variations is selected based on the pupil dilation data to adjust the visual feature.

5. The method of claim 3, wherein detecting the pupil dilation pattern comprises: Interpreting exogenous signals corresponding to pupil diameter changes in the pupil diameter data, the pupil diameter changes being caused by ambient light changes, chromaticity changes, adaptation of the eye, content lighting changes, periodic pupil dilation, changes in ambient noise, changes in motion of the device, brightness of the scene, cognitive load, or caffeine intake by the user.

6. The method of claim 1, wherein determining the user's intent to interact with the object comprises applying a machine learning technique trained to recognize patterns in physiological data corresponding to user intent.

7. The method of claim 1, wherein determining the user's intent to interact with the object comprises identifying an intent to perform a movement, make a decision, or select an object at a particular moment in time or in the future.

8. The method according to claim 1, wherein the device is a handheld device, a laptop computer, or a desktop computer, or a head mounted device (HMD).

9. The method of claim 1, wherein the visual feature is a color attribute of the object, a size of the object, a shape of the object, or an attribute of content proximate to the object.

10. The method of claim 1, wherein the first physiological data is acquired based on identifying an interaction of the user with the displayed object.

11. A biofeedback system comprising: Devices with displays and sensors; processor; as well as A computer-readable storage medium comprising instructions that, when executed by the processor, cause the system to perform operations comprising: displaying visual features associated with the object to a user on the display; acquiring, using the sensor, first physiological data associated with a first pupillary response of the user to the visual feature; determining the user's interest in interacting with the object, wherein determining the user's interest in interacting with the object comprises detecting a pupil dilation pattern by considering an exogenous signal corresponding to a pupil diameter change based on the obtained first physiological data; determining that adjusting a visual feature associated with the object will produce a pupil response greater than the first pupil response; adjusting the visual feature based on the acquired first physiological data to enhance a pupil response of the user relative to the adjusted visual feature associated with the object in response to determining that adjusting the visual feature will produce a pupil response greater than the first pupil response; and The object is displayed with the adjusted visual features.

12. The system of claim 11, wherein the operations further comprise: acquiring second physiological data associated with a second pupillary response of the user to the adjusted visual feature; as well as The user's intention to interact with the object is determined based on the acquired second physiological data. 13 . The system according to claim 11 , wherein the first physiological data comprises a pupil diameter of an eye of the user or a change in the pupil diameter of the eye of the user.

14. The system of claim 11, wherein adjusting the visual feature comprises: obtaining pupil dilation data corresponding to each of a plurality of variations of the visual feature; as well as One of the variations is selected based on the pupil dilation data to adjust the visual feature.

15. The system of claim 13, wherein detecting the pupil dilation pattern comprises: Interpreting exogenous signals corresponding to pupil diameter changes in the pupil diameter data, the pupil diameter changes being caused by ambient light changes, chromaticity changes, adaptation of the eye, content lighting changes, periodic pupil dilation, changes in ambient noise, changes in motion of the device, brightness of the scene, cognitive load, or caffeine intake by the user.

16. The system of claim 11, wherein determining the user's intent to interact with the object comprises applying a machine learning technique trained to recognize patterns in physiological data corresponding to user intent.

17. The system of claim 11, wherein determining the user's intent to interact with the object comprises identifying an intent to perform a movement, make a decision, or select an object at a particular moment in time or in the future.

18. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor of a device comprising a display and a sensor, cause the processor to perform the method according to any one of claims 1-10.

19. A computer program product comprising instructions which, when executed by a processor of a device comprising a display and a sensor, cause the processor to perform the method according to any one of claims 1 to 10.

Images