Seamless motivation based on cognitive state

Abstract

A method, device and product for seamless stimulation based on cognitive state are provided. The method comprises: obtaining the cognitive state of a human subject; determining a target cognitive state of the human subject; determining a significance level of stimulation based on the cognitive state and the target cognitive state, wherein the significance level is configured to cause the human subject to direct spatial attention to the stimulation, wherein the significance level is configured to cause the stimulation to be seamless to the cognitive state of a given human subject; and outputting the stimulation at the significance level so that the human subject can perceive it.

Description

Technical Field

[0001] The present disclosure relates generally to motivation and alerting, and more particularly to seamless motivation that is dependent on the cognitive state of its subject. Background Art

[0002] Partially autonomous cars have become common over the past few years, and it is foreseeable that their number will gradually increase in the coming years.

[0003] Attempts to apply hybrid systems, where vehicle control is divided between the driver and the self-driving machine, have so far met with very limited success. It seems difficult to strike a beneficial balance between these two entities. On the one hand, some human drivers seem overconfident in the machine and simply don't want to accept the call to take over, while others are anxious about machine control and do not benefit from self-driving.

[0004] Previous attempts to mitigate these challenges have focused on building fully autonomous cars or better warning systems. However, the need to minimize false negatives in such systems often results in higher false alarm rates and intervention requests. Summary of the invention

[0005] An exemplary embodiment of the disclosed subject matter is a method comprising: obtaining a current cognitive state of a human subject; determining a target cognitive state of the human subject; determining a saliency level of a stimulus based on the cognitive state and the target cognitive state, wherein the saliency level is configured to cause the human subject to direct spatial attention directly to the stimulus, wherein the saliency level is configured to cause the stimulus to be seamless given the current cognitive state of the human subject; and outputting the stimulus at the saliency level so that the human subject can perceive it.

[0006] Optionally, the human subject is a driver of a partially automated vehicle.

[0007] Optionally, the target cognitive state is related to danger faced by the partially automated vehicle.

[0008] Optionally, said determining the significance level of the stimulus is performed based on a model, wherein the model is updated based on the driver's identified responses to a plurality of stimuli.

[0009] Optionally, said determining a significance level of the incentive is performed based on a cognitive state of the driver, an assessed risk level and an assessed urgency.

[0010] Optionally, the stimulus is above the level of conscious perception of a human subject.

[0011] Optionally, the stimulus is a visual stimulus presented to be perceived by the peripheral vision of a human subject.

[0012] Optionally, the target cognitive state comprises spatial attention in the direction of the target, wherein the stimulus is associated with the location of the target.

[0013] Optionally, the stimulus is a visual stimulus that is presented in a relative position in the field of view of a human subject proximate to the target relative to the field of view.

[0014] Optionally, the stimulus is an audio stimulus configured to be perceived by a human subject having an associated relative position, wherein the relative position corresponds to a position of the target relative to the human subject.

[0015] Optionally, the stimulus is a visual stimulus, wherein the significance level is at least one of: a size of the visual stimulus, an opacity level of the visual stimulus, a brightness level of the visual stimulus, and a duration of presentation of the visual stimulus.

[0016] Optionally, the excitation is an audio excitation, wherein the significance level is at least one of: a volume level of the audio excitation, a frequency spectrum of the audio excitation, a similarity measure of the audio excitation to background noise, and a duration for which the audio excitation is provided.

[0017] Optionally, the target cognitive state is minimized by the following function: , where c is the cognitive state, where Risk(c) is a function of risk with respect to the cognitive state, where WellBeing(c) is a function of the well-being of a human subject given the cognitive state, and where α and β are coefficients.

[0018] Optionally, Risk(c) is a function of expected damage, defined as Risk(c) = Damage∙Probability(c), where Damage is the expected damage from the hazard and Probability(c) is a function of the probability that the damage will occur given the current cognitive state of the human subject.

[0019] Optionally, determining the significance level of the incentive is based on the well-being of the human subject, wherein the well-being of the human subject is assessed using at least one of: facial expressions of the human subject; eye glances of the human subject; one or more body postures of the human subject; and one or more changes in the body temperature of the human subject.

[0020] Optionally, determining the significance level of the stimulus is performed using a predictive model, wherein the predictive model is configured to evaluate the significance level of the stimulus so as to provide seamless stimulus while directing spatial attention to the stimulus, wherein the predictive model provides the evaluated significance level based on cognitive state, target cognitive state, and characteristics of the human subject.

[0021] Optionally, the predictive model is trained based on crowdsourced information about different people driving different vehicles.

[0022] Optionally, the determining of the significance level of the stimulus is performed based on a predictive model; wherein the method further comprises: identifying a response of a human subject to the stimulus; and updating the predictive model to improve the determination of the significance level in future use of the predictive model.

[0023] Optionally, the identifying a response comprises obtaining a new cognitive state of the human subject, wherein the updating the prediction model is based on a difference between the new cognitive state and a target cognitive state.

[0024] Optionally, the response indicates that the stimulus is not seamless to the human subject, and wherein updating the prediction model comprises updating the model to provide a lower significance level than a significance level determined in said determining significance under the same conditions.

[0025] Optionally, the response indicates that the stimulus did not cause the human subject to direct spatial attention to the stimulus, wherein updating the prediction model includes updating the model to provide a significance level that is higher than the significance level determined in determining the significance under the same conditions.

[0026] Another exemplary embodiment of the disclosed subject matter is a device comprising a processor and coupled memory, wherein the processor is adapted to perform: obtaining a current cognitive state of a human subject; determining a target cognitive state of the human subject; based on the cognitive state and the target cognitive state, determining a saliency level of a stimulus, wherein the saliency level is configured to cause the human subject to direct spatial attention to the stimulus, wherein the saliency level is configured to cause the stimulus to be seamless given the current cognitive state of the human subject; and outputting the stimulus at the saliency level so that the human subject can perceive it.

[0027] Optionally, the device is integrated into a partially autonomous vehicle, wherein the human subject is the driver of the partially autonomous vehicle.

[0028] Optionally, the target cognitive state is related to a hazard faced by the partially automated vehicle.

[0029] Optionally, said determining the significance level of the stimulus is performed based on a model, wherein the model is updated based on the driver's identified responses to a plurality of stimuli.

[0030] Optionally, determining the significance level of the incentive is performed based on the driver's cognitive state, the assessed risk level and the assessed urgency.

[0031] Optionally, the stimulus is a visual stimulus presented to be perceived by the peripheral vision of a human subject.

[0032] Optionally, the target cognitive state comprises spatial attention in the direction of the target, wherein the stimulus is associated with the location of the target.

[0033] Optionally, the stimulus is a visual stimulus that is presented in a relative position in the field of view of a human subject proximate to the target relative to the field of view.

[0034] Optionally, the stimulus is an audio stimulus configured to be perceived by a human subject having an associated relative position, wherein the relative position corresponds to a position of the target relative to the human subject.

[0035] Optionally, the stimulus is a visual stimulus, wherein the significance level is at least one of: a size of the visual stimulus, an opacity level of the visual stimulus, a brightness level of the visual stimulus, and a duration of presentation of the visual stimulus.

[0036] Optionally, the excitation is an audio excitation, wherein the significance level is at least one of: a volume level of the audio excitation, a frequency spectrum of the audio excitation, a similarity measure of the audio excitation to background noise, and a duration for which the audio excitation is provided.

[0037] Optionally, the target cognitive state is minimized by the following function: , where c is the cognitive state, where Risk(c) is a function of risk with respect to the cognitive state, where WellBeing(c) is a function of the well-being of a human subject given the cognitive state, and where α and β are coefficients.

[0038] Optionally, determining the significance level of the incentive is based on the well-being of the human subject, wherein the well-being of the human subject is assessed using at least one of: facial expressions of the human subject; eye glances of the human subject; one or more body postures of the human subject; and one or more changes in the body temperature of the human subject.

[0039] Optionally, determining the significance level of the stimulus is performed using a predictive model, wherein the predictive model is configured to evaluate the significance level of the stimulus so as to provide seamless stimulus while directing spatial attention to the stimulus, wherein the predictive model provides the evaluated significance level based on cognitive state, target cognitive state, and characteristics of the human subject.

[0040] Optionally, the predictive model is trained based on crowdsourced information about different people driving different vehicles.

[0041] Optionally, the determining of the significance level of the stimulus is performed based on a predictive model; wherein the processor is configured to perform: identifying a response of a human subject to the stimulus; and updating the predictive model to improve the determination of the significance level in future use of the predictive model.

[0042] Optionally, the identifying a response comprises obtaining a new cognitive state of the human subject, wherein the updating the prediction model is based on a difference between the new cognitive state and a target cognitive state.

[0043] Optionally, the response indicates that the stimulus is not seamless to the human subject, and wherein updating the prediction model comprises updating the model to provide a lower significance level than a significance level determined in said determining significance under the same conditions.

[0044] Optionally, the response indicates that the stimulus did not cause the human subject to direct spatial attention to the stimulus, wherein updating the prediction model includes updating the model to provide a significance level that is higher than the significance level determined in determining the significance under the same conditions.

[0045] Yet another exemplary embodiment of the disclosed subject matter is a non-transitory computer-readable medium that retains program instructions that, when read by a processor, cause the processor to perform the following operations: obtain a current cognitive state of a human subject; determine a target cognitive state of the human subject; based on the cognitive state and the target cognitive state, determine a saliency level for a stimulus, wherein the saliency level is configured to cause the human subject to direct spatial attention to the stimulus, wherein the saliency level is configured to cause the stimulus to be seamless to the current cognitive state of a given human subject; and output the stimulus at a saliency level so that the human subject can perceive it.

[0046] Optionally, the non-transitory computer-readable medium is at least partially loaded into a storage unit of a partially autonomous vehicle, wherein the human subject is a driver of the partially autonomous vehicle.

[0047] Optionally, the target cognitive state is related to a hazard faced by the partially automated vehicle.

[0048] Optionally, said determining the significance level of the stimulus is performed based on a model, wherein the model is updated based on the driver's identified responses to a plurality of stimuli.

[0049] Optionally, said determining a significance level of the incentive is performed based on the driver's cognitive state, the assessed risk level and the assessed urgency.

[0050] Optionally, the stimulus is a visual stimulus presented to be perceived by the peripheral vision of a human subject.

[0051] Optionally, the target cognitive state comprises spatial attention in the direction of the target, wherein the stimulus is associated with the location of the target.

[0052] Optionally, the stimulus is a visual stimulus that is presented in a relative position in the field of view of a human subject proximate to the target relative to the field of view.

[0053] Optionally, the stimulus is an audio stimulus configured to be perceived by a human subject having an associated relative position, wherein the relative position corresponds to a position of the target relative to the human subject.

[0054] Optionally, the stimulus is a visual stimulus, wherein the significance level is at least one of: a size of the visual stimulus, an opacity level of the visual stimulus, a brightness level of the visual stimulus, and a duration of presentation of the visual stimulus.

[0055] Optionally, the excitation is an audio excitation, wherein the significance level is at least one of: a volume level of the audio excitation, a frequency spectrum of the audio excitation, a similarity measure of the audio excitation to background noise, and a duration for which the audio excitation is provided.

[0056] Optionally, the target cognitive state is minimized by the following function: f(c)=∝∙Risk(c)-β∙WellBeing(c), where c is the cognitive state, where Risk(c) is a function of risk with respect to the cognitive state, where WellBeing(c) is a function of the well-being of the human subject given the cognitive state, and where α and β are coefficients.

[0057] Optionally, determining the significance level of the incentive is based on the well-being of the human subject, wherein the well-being of the human subject is assessed using at least one of: facial expressions of the human subject; eye glances of the human subject; one or more body postures of the human subject; and one or more changes in the body temperature of the human subject.

[0058] Optionally, determining the significance level of the stimulus is performed using a predictive model, wherein the predictive model is configured to evaluate the significance level of the stimulus so as to provide seamless stimulus while directing spatial attention to the stimulus, wherein the predictive model provides the evaluated significance level based on cognitive state, target cognitive state, and characteristics of the human subject.

[0059] Optionally, the predictive model is trained based on crowdsourced information about different people driving different vehicles.

[0060] Optionally, the determining of the significance level of the stimulus is performed based on a predictive model; wherein the processor is configured to perform: identifying a response of a human subject to the stimulus; and updating the predictive model to improve the determination of the significance level in future use of the predictive model.

[0061] Optionally, the identifying a response comprises obtaining a new cognitive state of the human subject, wherein the updating the prediction model is based on a difference between the new cognitive state and a target cognitive state.

[0062] Optionally, the response indicates that the stimulus is not seamless to the human subject, and wherein updating the prediction model comprises updating the model to provide a lower significance level than a significance level determined in said determining significance under the same conditions.

[0063] Optionally, the response indicates that the stimulus did not cause the human subject to direct spatial attention to the stimulus, wherein updating the prediction model includes updating the model to provide a significance level that is higher than the significance level determined in determining the significance under the same conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The subject matter of the present disclosure will be more fully understood through the following detailed description in conjunction with the accompanying drawings, in which corresponding or similar numbers or characters indicate corresponding or similar components. Unless otherwise indicated, the accompanying drawings provide exemplary embodiments or aspects of the present disclosure and do not limit the scope of the present disclosure. In the accompanying drawings:

[0065] Figure 1 A schematic diagram showing an environment according to some exemplary embodiments of the disclosed subject matter;

[0066] Figure 2 A flow chart showing a method according to some exemplary embodiments of the disclosed subject matter;

[0067] Figure 3 Schematic diagrams showing systems and methods according to some exemplary embodiments of the disclosed subject matter;

[0068] Figure 4 A block diagram showing an apparatus according to some exemplary embodiments of the disclosed subject matter; and

[0069] Figure 5A-5B Shown is a schematic diagram of visual stimulation in accordance with the disclosed subject matter. Specific embodiments

[0071] One technical problem solved by the disclosed subject matter is to provide a system and method for managing the attention of a human subject. In particular, the technical problem may be applicable to the field of driving a partially automated vehicle. However, such embodiments are only examples, and the disclosed subject matter is not limited to such fields and may be used for other human subjects and other scenarios.

[0072] It may be desirable to be able to manage the driver's acceptance and engagement. In some cases, it may be desirable to provide incentives to change the driver's cognitive state to a target cognitive state.

[0073] Another technical issue may be providing seamless incentives to the driver that are still useful for modifying the driver's cognitive state. Seamless incentives can be above-conscious incentives, which are above the driver's conscious perception level, rather than subconscious incentives. In some cases, a human driver may not be able to detect the incentives given the driver's available cognitive resources. As an example, the driver's cognitive state can include cognitive engagement, utilizing the driver's cognitive resources, and causing inadvertent blindness. However, the disclosed subject matter is not limited to seamless incentives caused by inadvertent blindness, and other cognitive mechanisms can cause incentives above consciousness to be perceived as seamless incentives.

[0074] In some exemplary embodiments, the same stimulus over awareness may be seamless for one driver and not seamless for another driver. Additionally or alternatively, the same stimulus over awareness may be seamless but not seamless for the same driver in different cognitive states. It may be desirable to determine which stimulus to apply so that the driver perceives the stimulus as a seamless stimulus in the current situation.

[0075] In some cases, seamless stimulation can be configured to direct spatial attention to a target. The target can be, for example, a pedestrian crossing the road and visible to the driver. In this case, the spatial attention can be directed to a portion of the windshield. As another example, the target can be a speeding car that is located behind the driver's vehicle and can be seen through the vehicle's rearview mirror. In this case, the spatial attention can be directed to the mirror.

[0076] Spatial attention can be part of a human subject's current cognitive state. In some exemplary embodiments, when a human subject commits cognitive resources to process an area, the human subject can be said to direct spatial attention to that area. In some cases, spatial attention allows humans to selectively process visual information by prioritizing a region within the visual field. A spatial region within the visual field can be selected to attract attention, and the information within the region is then subject to further processing. In some cases, when spatial attention is directed, an observer can often detect targets that appear in expected locations more quickly and accurately than in unexpected locations.

[0077] In some cases, cognitive engagement can be the coupling of spatial attention to a specific object over a period of time. In some cases, when the driver of a partially automated vehicle performs other non-driving related tasks, the driver may be performing cognitive engagement, thereby reducing his available cognitive resources. The driver's cognitive acceptance can indicate the driver's total available cognitive resources. Acceptance can be affected by internal reasons, such as mind wandering, tiredness, etc. Acceptance can be affected by cognitive engagement of the human object, because such engagement reduces cognitive resources available for other purposes.

[0078] One technical solution provided by the disclosed subject matter is to determine a significance level of an incentive. The significance level may be determined based on a current cognitive state of a human subject, such as a driver. The significance level may be determined based on a desired target cognitive state of the driver. It may be desirable to change the cognitive state of the driver from his given cognitive state to a target cognitive state. In some cases, the target cognitive state may be associated with a hazard, such as a static hazard facing a partially automated vehicle, the hazard always facing the vehicle at that location, a dynamic hazard facing the vehicle, etc.

[0079] In some cases, a model may be used to determine the significance level of the incentive. The model may be a prediction model defined using deep learning techniques, using a convolutional neural network (CNN), a recurrent neural network (RNN), using supervised learning, using a support vector machine (SVM), using linear regression, using a decision tree, and the like. In some cases, reinforcement learning may be used to determine the model. Additionally or alternatively, the model may be determined based on the driver's response. Additionally or alternatively, a model may be determined based on the driver's crowdsourced response, which may be used to reflect the significance level required for the expected incentive, which is estimated to be seamless for a driver with given driver-specific characteristics.

[0080] In some exemplary embodiments, one or more factors may be used to manipulate the significance of the stimulus. For example, in a visual stimulus, the significance of the stimulus may be controlled by the size of the stimulus, the opacity of the stimulus, the brightness of the stimulus, the duration of visual presentation of the stimulus, and the like. In some cases, the significance level of the stimulus may be controlled by the position of the stimulus relative to the subject's field of view. For example, a visual stimulus may be presented in the subject's peripheral field of view to be perceived by the visual mechanism surrounding the subject. In some exemplary embodiments, when the significance level is high, the stimulus may be presented closer to the center of gaze. For example, as the significance level decreases, the angle between the point of fixation of the subject's gaze and the stimulus may decrease. In some exemplary embodiments, for a first range of significance levels, the stimulus may be presented in a distant peripheral field of view (e.g., above an angle of approximately 60 degrees). For a second range of significance levels higher than the first range, the stimulus may be presented in the middle of the peripheral field of view (e.g., between an angle of approximately 30 degrees and 60 degrees from the fixed point). For a third range of significance levels, which is higher than the second range, the stimulus may be presented in a close peripheral field of view (e.g., between approximately 18 degrees and 30 degrees). In some cases, a stimulus with a relatively high significance level may be presented in the central view of the object. Additionally or alternatively, the relative position of the function may be determined according to the significance level, such as approaching the center of fixation in a linear scale, a nonlinear scale, as the significance level increases.

[0081] In some exemplary embodiments, in the audio excitation, the significance of the excitation can be controlled by the volume of the audio excitation, the spectrum of the audio excitation, the duration of providing the audio excitation, etc. In some exemplary embodiments, the audio excitation can be provided in a manner similar to background noise. Additionally or alternatively, a pattern matching the background noise can be used for the audio excitation. In some exemplary embodiments, as the significance level decreases, the audio excitation may become more and more similar to the background noise, thereby becoming relatively undetectable.

[0082] In some exemplary embodiments, the target cognitive state may be determined by considering at least two factors: on the one hand, the well-being of the subject, and on the other hand, the risk level. For example, the target cognitive state is minimized by the following function: f(c)=α∙Risk(c)-β∙WellBeing(c), where c is the cognitive state, Risk(c) is a function of risk relative to the cognitive state, WellBeing(c) is a function of the well-being of the human subject given the cognitive state, and where α and β are coefficients. In some cases, the disclosed subject matter may avoid creating distractions that would reduce the well-being of the driver if the distraction does not provide sufficient risk reduction for the vehicle. For example, if the driver is reading an article, taking their attention away from the screen would be considered a distraction and would only be performed if the driver's attention is needed to significantly reduce the risk. Alternatively, a minor distraction, such as provided using seamless incentives, may free up cognitive resources (e.g., resources used to cognitively engage in non-driving activities) from non-driving activities in a relatively seamless manner without the driver noticing the reason for such cognitive resource release. The driver's altered cognitive state can be one that puts the driver in a better position to respond to an apparent risk, such as responding to another vehicle running a red light in front of the partially automated vehicle.

[0083] In some exemplary embodiments, the disclosed subject matter can provide a system including sensors, processors, stimulators, and displays that can passively monitor and actively detect the state of a driver, thereby continuously monitoring the driver's acceptance and engagement. Such monitored attributes of the driver can be compared to corresponding target levels. In some exemplary embodiments, the system can include a dynamic control module that utilizes a spectrum of incentives and various types of rewards that are operable to change the driver's engagement and acceptance and train the driver to obtain adequate responses, thereby promoting rapid and highly adaptive reactions.

[0084] According to some embodiments of the disclosed subject matter, a system for real-time monitoring and detection of driver attention can be provided. The system can allow for progressive participation, providing drivers with relevant information for handling road hazards, thereby reducing the possible range of relevant and adaptive responses. In some exemplary embodiments, the system can be tailored for each user, taking into account the physical condition of each user (e.g., age, vision, etc.), the physical condition at a specific time (e.g., fatigue, alertness, mood, damage, etc.), and the dynamics of continuous attention allocation. The system can be personalized based on the data collected about the driver. Additionally or alternatively, the system can be personalized based on the collected data about the driver and other drivers, such as crowdsourced data. The system can continuously monitor the state and behavior of the driver and detect the driver through a set of incentives that may be customized. In some exemplary embodiments, the system can use control engineering and reinforcement learning to provide the driver with the best set of incentives, allowing the degree of participation and acceptance to be increased to the desired level. In some embodiments, the system can provide enhanced guidance for the driver to produce an improved response.

[0085] One technical effect of the disclosed subject matter can be to cause a seamless change in a current cognitive state of a human subject. In contrast to a compelling incentive, a seamless incentive may not (at least not substantially) affect the well-being of the human subject. In some exemplary embodiments, the seamless incentive does not interrupt the driver's non-driving activities. Additionally or alternatively, the seamless incentive does not irritate the driver.

[0086] In some exemplary embodiments, false alarms may generally be considered a problem. Systems with high false alarm rates are generally problematic because users learn to ignore such alarms. On the other hand, a lower false alarm rate may be achieved by reducing the sensitivity of the system, but such a configuration may also have a correspondingly higher false alarm rate, i.e., no alarm is sounded when an alarm should be sounded. In critical systems, safety-related systems, missed alarms are generally not accepted. Considering the safety system of a vehicle, it is considered unacceptable that a potential collision will be ignored and not warned, allowing for a reduction in false alarm warnings (i.e., warnings of collisions where such collisions are unlikely to occur). The disclosed subject matter may provide an effective solution to this gap. If the alerts are provided in a seamless manner, situations with a high false alarm rate are essentially meaningless and the user does not actively perceive them. The user is not stimulated by such false alarms, the user experience is not adversely affected, and he does not (and will not effectively) learn to ignore such alarms.

[0087] The disclosed subject matter may provide one or more technical improvements over any prior art and any technology that has previously become conventional or customary in the art.

[0088] In view of this disclosure, other technical problems, solutions and effects may be apparent to those of ordinary skill in the art.

[0089] Reference now Figure 1 , which shows an environment according to some exemplary embodiments of the disclosed subject matter.

[0090] The in-vehicle client 120 may be configured to monitor the driver's cognitive state, the assessed risk level, the assessed urgency level, etc. The assessed risk level may reflect the potential risk of the vehicle, such as reflected by static hazards (e.g., vehicles entering a school zone, vehicles arriving at a T-intersection, etc.), dynamic hazards (e.g., a ball rolling on the ground, crossing a crosswalk, etc.), etc. Static hazards may be obtained from a navigation system, a third-party server, etc. based on the vehicle's location data (e.g., obtained from a GPS receiver). Forward sensors, sensors configured to provide measurements outside and near the vehicle, and data received from other in-vehicle clients 120 (e.g., information about other vehicles near the vehicle) may be used to detect dynamic hazards. The assessed urgency level may be the urgency level for human driver intervention. In some cases, if human intervention is required, there is enough time to respond, so the risk level may be high, but the urgency level may be low. Additionally or alternatively, in the case where the likelihood of the risk being realized is actually low, a low risk level may be presented with a high urgency, but if the risk is realized, the driver must react immediately.

[0091] Additionally or alternatively, the vehicle client 120 may be configured to monitor the driver's cognitive state. The driver's cognitive state may be determined using sensors facing the driver, such as microphones, cameras, biometric sensors, and the like.

[0092] In some exemplary embodiments, the in-vehicle client 120 may be configured to determine a target cognitive state based on the driver's current cognitive state, assessed risk level, assessed urgency level, etc. In some exemplary embodiments, the target cognitive state may be determined using a model that is updated based on the determined driver's response to multiple stimuli. In some cases, the model may be improved over time to take into account estimated risk level, estimated required urgency, driver well-being, etc. to optimize its functionality.

[0093] In some exemplary embodiments, each in-vehicle client 120 may collect data related to the driver of the vehicle. The data may be used to determine a customized model to determine a target cognitive state of the driver to determine a model for selecting a significance level relative to the driver. In some exemplary embodiments, the data may be transmitted over the network 105 and may be stored in a data repository such as crowdsourced data 130. In some exemplary embodiments, the model generator and distributor 110 may collect crowdsourced data 130 to generate a model based on the crowdsourced data. In some exemplary embodiments, the model may be a predictive model for determining an estimated significance level of an incentive so as to provide a seamless incentive while directing spatial attention to the incentive. The model may receive as input the current cognitive state, the target cognitive state, and the characteristics of the driver. In some exemplary embodiments, the model may be trained based on crowdsourced information related to different people driving different vehicles. In some cases, using crowdsourced data 130, big data analysis may be enabled to utilize drivers with similar characteristics, such as cognitive patterns, demographic attributes, behavioral attributes, etc., to provide an estimated significance level for the driver, even without relevant past information. In some cases, the model may be updated based on the driver's response (or lack of response) by the vehicle client 120 or by the model generator and distributor 110. In some exemplary embodiments, the model generator and distributor 110 may be configured to distribute the generated or updated model to the vehicle client 120, for example, periodically, on demand, or based on the client's initiation, etc.

[0094] Reference now Figure 2 , which shows a flow chart of a method according to the disclosed subject matter.

[0095] In step 210, the cognitive state of the driver is obtained. The current cognitive state of the driver can be obtained using various sensors, such as but not limited to a camera facing the driver, a microphone, an eye tracker, a sensor for biometrics, etc. The cognitive state may include the cognitive acceptance of the driver. Additionally or alternatively, the cognitive state may include cognitive attention to an object or task. Additionally or alternatively, the cognitive state may include cognitive engagement and its duration. Additionally or alternatively, the cognitive state may include cognitive engagement in non-driving activities. Additionally or alternatively, the cognitive state may include cognitive engagement in driving activities.

[0096] In step 220, road hazards may be determined. Road hazards may be hazards faced by a vehicle that is (at least partially) driven by a driver. Road hazards may be determined using a GPS or positioning module in combination with data from a navigation system. Additionally or alternatively, road hazards may be determined based on vehicle sensors such as external cameras, microphones, proximity sensors, etc. Additionally or alternatively, dynamic hazards may be reported by other connected devices, such as Internet of Things (IoT) devices, connected cars, etc. For example, another vehicle may report hazards that exist at a location that the vehicle is about to arrive at (e.g., within about one minute). Thus, road hazards for the vehicle may be determined.

[0097] In step 230, a target cognitive state may be determined. The target cognitive state may be different from the current cognitive state. The target cognitive state may be determined based on an assessed urgency, an assessed risk, a hazard facing the vehicle, etc. In some exemplary embodiments, the target cognitive state may include spatial attention in the direction of a target. The target may be, for example, a target associated with a hazard facing the vehicle, such as a ball on the road, a pedestrian crossing the road, a car running a red light, a curve in the road, or other areas not adequately monitored by the vehicle's sensors, etc.

[0098] In some exemplary embodiments, the target cognitive state may be a state that maximizes the driver's well-being and minimizes potential risk. For example, the target cognitive state may be determined as a c-state that minimizes the following function: f(c)=α∙Risk(c)-β∙WellBeing(c). In some exemplary embodiments, WellBeing(c) may be a function of the well-being of a human subject for a given cognitive state. In some cases, a driver's well-being may be reduced if he or she is unable to engage in required non-driving tasks. In some exemplary embodiments, facial expressions and gestures, pupil dilation, eye saccades, body temperature, and other physiological parameters are used to assess the driver's well-being. In some exemplary embodiments, driver profiles and histories are used to assess the driver's well-being. Additionally or alternatively, the driver's well-being may also be affected by non-seamless incentives that interfere with his or her activities. In some exemplary embodiments, α and β may be coefficients, such as giving relative weights to risk and well-being factors. In some exemplary embodiments, the coefficients may be between 0 and 1 and sum to 1. In some exemplary embodiments, Risk(c) can be a function of risk relative to cognitive state. In some exemplary embodiments, given cognitive state, Risk(c) can be a function of the expected damage. In some exemplary embodiments, Risk(c) can be defined as . Damage is the expected damage due to the hazard. Given the driver’s cognitive state, Probability(c)It may be a function of the probability that damage will occur.

[0099] In step 240, the significance level of the incentive can be determined. In some exemplary embodiments, the significance and content of the incentive can be determined. In some exemplary embodiments, the significance and content are compatible with the environmental conditions and the urgency of the situation. In some exemplary embodiments, the significance of the incentive can be determined by multiple parameters including modality, duration, intensity, and location. In some exemplary embodiments, the significance of the incentive can be gradually increased while monitoring the driver's response until the target cognitive state is reached. In some exemplary embodiments, using this paradigm, the system can operate in a minimally invasive manner, thereby potentially minimizing the reduction in the driver's well-being.

[0100] In some exemplary embodiments, the salience level of the stimulus may be determined based on cognitive state, goal state, etc. In some exemplary embodiments, the salience level is configured to cause a human subject to direct spatial attention to the stimulus. In some exemplary embodiments, the salience level is configured to cause a stimulus for a given cognitive state to be seamless to a human subject.

[0101] In some exemplary embodiments, the modality of the stimulus may be a visual modality. The significance level may be affected by the position in the driver's field of view, such as in a focused area, in a close-range peripheral field of view, in a long-range peripheral field of view, etc. Additionally or alternatively, the significance level may be the size of the visual stimulus, the opacity level of the visual stimulus, the brightness level of the visual stimulus, the duration of presenting the visual stimulus, the angle with the driver's gaze, or similar.

[0102] In some exemplary embodiments, the modality of the excitation may be an audio modality. The significance level may be the volume level of the audio excitation, the frequency spectrum of the audio excitation, a measure of similarity of the audio excitation relative to background noise, the duration of providing the audio excitation, etc. In some exemplary embodiments, there may be background noise and a pattern that may be matched to the background noise, which pattern is used for the audio excitation. The difference from the background noise may be significant - the greater the difference, the more significant the excitation.

[0103] In some exemplary embodiments, a predictive model may be used to determine a significance level for an incentive. In some exemplary embodiments, the predictive model is configured to evaluate an estimated significance level of an incentive so as to provide a seamless incentive while directing spatial attention to the incentive. In some exemplary embodiments, the predictive model provides an estimated significance level based on cognitive state, target cognitive state, and the like. In some exemplary embodiments, the predictive model may be trained based on the driver's response to past incentives in different cognitive states. Additionally or alternatively, the predictive model may be based on crowdsourced data, such as data of a population having similar characteristics to the driver, such as similar demographic information, similar sensory perceptions (e.g., same hearing, similar visual acuity, etc.), and similar behavioral patterns, etc.

[0104] In step 250, the stimulus may be output. The stimulus may be output using an output module appropriate to the stimulus modality, for example, a speaker for audio stimulus, a display for visual stimulus, etc. The output may be provided at a determined significance level.

[0105] In step 260, the driver's response can be determined. In some exemplary embodiments, sensors can be used to track and monitor the driver's activities. If the driver responds to the stimulus in a conscious or unconscious manner, the response can be identified.

[0106] In some exemplary embodiments, if the driver perceives the stimulus in a non-seamless manner (262), this information may be used to update the prediction model for future use (270). In some exemplary embodiments, the model may be updated to reduce the significance level to avoid future non-seamless stimulus. Additionally or alternatively, if there is an effect on the driver's cognitive state (264), the effect may be analyzed and if it does not match the desired effect, the model may be updated (270). In some exemplary embodiments, if the stimulus does not cause the driver to reach the target cognitive state, the model may be updated accordingly to increase the significance in future attempts. In some cases, the next stimulus of a higher significance level may then be determined and output (e.g., steps 240-270) to reach the target cognitive state of step 230. Additionally or alternatively, the effect of the stimulus on the cognitive state may be determined, and the difference between the driver's new cognitive state and the target cognitive state may be identified and used to update the prediction model.

[0107] In some exemplary embodiments, information collected based on driver responses can be provided to a server for collection. The crowdsourced data can then be used in an aggregated manner to improve models for multiple drivers using multiple vehicles.

[0108] In some exemplary embodiments, the predictive model may provide features of the stimulus, including multiple different significance levels in different significance attributes. Additionally or alternatively, the predictive model may provide a modality of the stimulus. In such embodiments, modification of the model may cause future predictions to provide different incentives with different features, including some portions that are more significant than before and some portions that are less significant than before. For example, the model may determine that a brighter stimulus is displayed for a shorter duration. As another example, the model may determine a visual stimulus rather than an audio stimulus.

[0109] Reference now Figure 3 , which shows a schematic diagram of systems and methods according to some exemplary embodiments of the disclosed subject matter. A sensor 320 is used to monitor the status of the driver 310 .

[0110] In some exemplary embodiments, artificial intelligence (AI) preprocessing 330 may process data from sensor 320 to generate second-level features. As an example, AI preprocessing 330 may generate features related to gaze direction within a visible scene, features related to pupil dilation, features indicating low receptive behavior patterns, and the like. In some exemplary embodiments, facial expressions and body postures, pupil dilation, eye saccades, and other physiological parameters may be used by AI preprocessing 330 to assess driver well-being. In some exemplary embodiments, driver profiles and histories are used to assess driver well-being. In some embodiments, AI preprocessing 330 may utilize predictive models, such as autoregressive (AR) models, Kalman filters, particle filters, and deep neural networks with long short-term memory, and the like. Predictive models may be used to predict any aspect of a driver's cognitive state, but are not limited to, such as, driver's attention, driver's cognitive engagement, and the like.

[0111] In some exemplary embodiments, the second level features may be converted into a vector of acceptability level indicators along with corresponding confidence levels, which may be determined as part of AI pre-processing 330. In some exemplary embodiments, if the confidence level is low (e.g., below a threshold), the system may probe the driver 310 by using conditional detection 350. Conditional detection 350 may be used to generate various stimuli and monitor the driver 310's response to those stimuli. Given the driver's response, as measured by sensors 320 and processed by AI pre-processing 330, a new set of acceptability indicators may be generated, perhaps with a higher confidence level. In some exemplary embodiments, the acceptability indicators may include features related to the response to the probe, the response to a particular form of probe, and the like.

[0112] In some exemplary embodiments, the receptivity indicator may be fed into the cognitive engagement control module 360 ​​along with the target cognitive state determined by the target determiner 365. The cognitive engagement control module 360 ​​may use these inputs at time T and inputs from previous times T-1, T-2, ... TN to generate a prediction of the cognitive state at T + 1. In some exemplary embodiments, N input vectors may be used to generate control incentives to be provided to the driver 310 using the incentive provider 370. In some exemplary embodiments, the following method may be used to determine and provide a positive or negative reward incentive provider 370 to the driver 310. In some exemplary embodiments, the reward may be configured to change the focus of the driver's receptivity to enable the driver to comply with the situation and promote adaptive reactions. In some exemplary embodiments, the reward may be determined to improve the driver's response quality in the long term. In some embodiments, the enhancement and reward to the driver 310 include positive and negative enhancements, which can be used to adjust the driver 310 and improve attention, acceptance, and adaptability to the desired object. In some embodiments, low-significance noise and disruption can be used to induce negative rewards, which will make the driver 310 feel slightly uncomfortable when the response to the incentive is significantly deviated from the optimal incentive. In some embodiments, long-term positive rewards include virtual tokens that can be converted into various digital goods or shopping and dining discounts, and long-term negative rewards can be obtained by reducing the number of such virtual tokens in the driver 310 account. In some embodiments, the gamification paradigm is used to provide positive and negative reinforcement as part of the game. In some embodiments, reinforcement can be coupled with incentives to create a seamless and non-intrusive form of driver 310 behavior. This can be used to maintain the acceptance of the driver 310 at a certain level while minimizing unnecessary perceptual load.

[0113] In some exemplary embodiments, the stimulus provider 370 can be used to display visual incentives with various levels of significance. For example, incentives can be displayed for a short time (e.g., 100 milliseconds, 1.5 seconds, etc.) in the peripheral area of ​​the driver's field of view. As another example, an almost invisible audio signal can be provided. The almost unnoticed audio signal may be customizable. In some cases, the almost unnoticed audio signal may be compatible with the driver's preferences and the driver's environment. In some exemplary embodiments, the almost imperceptible audio signal is based on manipulating the ambient sound detected by the system acoustic sensor, for example, by adding noise with similar characteristics, modulating the amplitude in a short time while monitoring the driver's reaction. As another example, a tactile module embedded in the driver's seat or steering wheel can be used to introduce tactile incentives of different amplitudes. In some embodiments, the system utilizes a subconscious incentive set that does not increase the cognitive burden of the driver, thereby improving the driver's experience.

[0114] In some exemplary embodiments, given the acceptance level evaluated by the acceptance level, the system can reflect the driver's ability to handle driving related situations. In some exemplary embodiments, module 360

[0115] It can be configured to use sensor data to assess the level of engagement and corresponding engagement in other non-driving activities, such as text messaging, reading, navigation, utilization of vehicle entertainment and information systems, etc. This information can be used as an indication of driving engagement, which can be determined by cognitive engagement in the driving module 342. If the confidence level relative to driving engagement is low, condition detection 350 can be used. Condition detection 350 can be used to generate various incentives and monitor the driver's response to these incentives. The cognitive engagement control module 360 ​​can use the engagement indicator and the target cognitive state determined by the target determiner 365 to determine its incentive and significance level.

[0116] In some exemplary embodiments, the goal determiner 365 may be configured to determine a target cognitive state, such as a desired level of engagement that the driver 310 needs to achieve. The goal determiner 365 may, for example, automatically determine the target cognitive state based on the current context, current activity, etc.

[0117] Reference now Figure 4 , which shows a block diagram of an apparatus according to some exemplary embodiments of the disclosed subject matter.

[0118] In some exemplary embodiments, device 400 may include one or more processors 402. Processor 402 may be a central processing unit (CPU), a microprocessor, an electronic circuit, an integrated circuit (IC), etc. Processor 402 may be used to perform calculations required by device 400 or any of its subcomponents.

[0119] In some exemplary embodiments of the disclosed subject matter, device 400 may include an input / output (I / O) module 405. I / O module 405 may be used to provide output and receive input. For example, I / O module 405 may be configured to receive sensory information from a sensor observing a driver, a sensor observing a vehicle environment, and the like. Additionally or alternatively, I / O module 405 may be configured to communicate with external devices such as connected cars, IoT devices, servers, and the like, thereby enabling information to be transferred to and from device 400.

[0120] In some exemplary embodiments, the device 400 may include a storage unit 407. The storage unit 407 may be a hard drive, a flash drive, a random access memory (RAM), a memory chip, etc. In some exemplary embodiments, the memory unit 407 may retain program code operable to cause the processor 402 to perform actions associated with any subcomponent of the device 400.

[0121] In some exemplary embodiments, cognitive state determiner 410 may be configured to determine the cognitive state of the driver. The cognitive state may be determined based on sensor data, based on reference drivers with similar characteristics, etc. In some exemplary embodiments, cognitive state determiner 410 may be configured to determine cognitive engagement in different tasks, e.g., driving-related tasks, non-driving-related tasks, etc. The duration of cognitive engagement in each such task may be determined. In some exemplary embodiments, cognitive state determiner 410 may determine the cognitive resources of the driver that are free and available for cognitive attention associated with driving-related tasks.

[0122] In some exemplary embodiments, the cognitive target determiner 420 may be configured to determine a target cognitive state for the driver. The target cognitive state may be determined based on urgency, imminent risk level, the driver's desired well-being, and the dangers facing the vehicle, etc. Additionally or alternatively, the target cognitive state may be determined based on the driver's current cognitive state determined by the cognitive state determiner 410.

[0123] In some exemplary embodiments, the significance level determiner 430 may be configured to determine the significance level for the incentive and the modality of the incentive, etc. The characteristics of the incentive may be determined to provide the driver with a seamless incentive given the driver's cognitive state, which is determined by the cognitive state determiner 410 and is configured to direct the driver's attention to the goal or shift cognitive attention to driving-related activities. In some exemplary embodiments, the incentive characteristics may be determined so as to cause a change to affect the current cognitive state, changing it to be closer to the target cognitive state, if not exactly the same as the target cognitive state.

[0124] In some exemplary embodiments, the prediction model 440 may be a model for predicting significance levels and other incentive characteristics to provide seamless incentives that achieve the desired change in the driver's cognitive state. The model 440 may be trained locally in the device 400 and may be obtained from, for example, Figure 1 A server, such as the model generator and distributor 110 of , receives the model 440. In some exemplary embodiments, the prediction model 440 may be determined using training data collected from various drivers and responses to incentives provided thereto.

[0125] In some exemplary embodiments, the excitation output 450 can be a module configured to implement the supply of excitation in a determined mode, and the mode has a significance level as determined by the prediction model 440. In some exemplary embodiments, the excitation output 450 can be configured to determine the position of the excitation. The position can be determined based on the driver's line of sight, and the driver's attention relative to the line of sight is guided and focused on the target. As an example, visual excitation can be provided at a certain angle, and if the driver moves its focus to this angle, the driver will watch the target object, such as an object related to danger. Additionally or alternatively, directional sound can be provided using unbalanced audio excitation. Directional sound can be configured to a relative position, and the relative position has the same angle as the object of interest from the driver. For example, if the object of interest such as a pedestrian is located at a position 110 degrees away from the driver's line of sight, audio excitation can be provided in a manner that simulates the sound emitted from the position of the object of interest.

[0126] Figure 5A-5B Shown is a schematic diagram of visual stimulation in accordance with the disclosed subject matter.

[0127] exist Figure 5A In the embodiment of the present invention, the driver 510 of the partially automated vehicle directs his attention in the display 520. The display 520 may be a display on the windshield. The display 520 may display content unrelated to driving activities, such as movies, content of websites, video conferences, etc. The driver 510 is not currently engaged in driving activities, and the automated driver is driving the vehicle.

[0128] Pedestrian 505 may be crossing the road, and there is a risk that the vehicle will hit pedestrian 505. Therefore, if driver 510 needs to take over driving activities to avoid a collision, it may be desirable to ensure that more reminders are given to driver 510 and enable driver 510 to react.

[0129] like Figure 5A As shown, pedestrian 505 is visible outside the windshield. However, the driver's 510 vision focus is at a different location.

[0130] In some exemplary embodiments, the incentive 530a may be visually presented in the display 520 at a location relative to both the driver 510 and the pedestrian 505. In some exemplary embodiments, the incentive 530a is presented at a location where the driver 510 is looking, an area of ​​interest where the pedestrian 505 is closer. In some exemplary embodiments, the incentive 530a may be presented at a location in the peripheral field of vision of the driver 510. In some exemplary embodiments, the incentive 530a may be presented using different levels of prominence, such as for a limited duration, partially transparent, limited size, etc.

[0131] Additionally or alternatively, stimulus 530b is another example of a potential stimulus. In contrast to stimulus 530a, stimulus 530b is displayed on the exterior of active display 520 being used by driver 510. Stimulus 530b may be displayed at a closer location relative to pedestrian 505.

[0132] In some exemplary embodiments, if the driver needs to take over the driving activity, the incentive (530a, 530b) can be provided as a seamless incentive that is unknown to the driver 510 but affects the cognitive state of the driver 510, thereby improving the driver's responsiveness to the pedestrian 505. The incentive can be configured to free up some cognitive resources from the cognitive engagement that the driver 510 has with respect to the display 520 so that a quick and effective take-over maneuver can be made when such a maneuver is needed.

[0133] exist Figure 5B 5, the driver 510 is using the mobile device 540 for an activity. It may be desirable to direct the driver's 510 attention upward and diagonally in order to raise the focus of gaze to a position where he may be able to gaze to see the pedestrian 505. The stimulus 530c may be a pattern of dots, appearing in sequence, directing his attention in a particular direction. For example, the first dot may be the smallest, and each dot may have an increasing size. The last dot may be the dot in the upper right corner of the mobile device 540. Thus, the size and appearance pattern may instinctively draw attention toward the top and right side of the driver 510, such as closer to the position of the pedestrian 505.

[0134] In some exemplary embodiments, the disclosed subject matter may be utilized with respect to an aircraft pilot, particularly in a configuration where the pilot utilizes an autopilot system to control the aircraft. In certain situations, the pilot's cognitive state may affect his or her ability to respond to emergency situations, unexpected events, etc. Therefore, it may be desirable to monitor the pilot's cognitive state and utilize seamless stimulation in accordance with the disclosed subject matter to guide the pilot's cognitive state to a desired target cognitive state.

[0135] It should be noted that although the subject matter of the present disclosure is described using an implementation of a partially automated vehicle and its driver, the subject matter of the present disclosure is not limited to this implementation and may be used in other scenarios involving the cognitive state of a human subject.

[0136] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0137] A computer-readable storage medium may be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium may be, but is not limited to, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A more specific non-exhaustive list of computer-readable storage media includes the following: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a raised structure in a groove having the above recorded instructions, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be understood as being a transient signal itself, for example, a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0138] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0139] The computer-readable program instructions for performing the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code. The source code or object code is written in one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc. and conventional procedural programming languages, such as "C" programming languages ​​or similar programming languages. The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, partially on a user's computer as a stand-alone software package, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or a connection may be established with an external computer (for example, via the Internet using an Internet service provider). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions to implement aspects of the present invention.

[0140] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0141] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions are executed via a computer or other processor to execute a programmable data processing device to create a device for implementing the functions / actions specified in the flowchart and / or block diagram box. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, and / or other devices to function in a specific manner, so that the computer-readable storage medium in which the instructions are stored contains an article containing the instructions, which includes instructions for implementing various aspects of the functions / actions specified in the flowchart and / or block diagram box.

[0142] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device, so that a series of operating steps are performed on the computer, other programmable device, or other device to produce an implemented computer processing process, so that the instructions executed on the computer, other programmable device, or other device implement the functions / actions specified in the flowchart and / or block diagram blocks.

[0143] The flow chart and block diagram in the accompanying drawings show the possible implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each square block in the flow chart or block diagram can represent a module, a fragment or a part of an instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative embodiments, the function pointed out in the square block may not occur in the order pointed out in the figure.For example, depending on the function involved, two square blocks of continuous display can actually be executed substantially at the same time, or these square blocks can be executed in reverse order sometimes.It should also be noted that each square block of the block diagram and / or flow chart and the combination of the square blocks of the block diagram and / or flow chart can be realized by a hardware-based special-purpose system for a specific purpose, and the special-purpose system performs a specified function or action or performs a combination of hardware and computer instructions for a special purpose.

[0144] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" shall also include the plural forms unless the context clearly indicates otherwise. It will also be understood that the terms "comprise" and / or "include" when used in this specification specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups.

[0145] All methods or steps in the following claims plus corresponding structures, materials, actions and equivalents in functional elements are intended to include any structure, material or action for performing functions together with other elements as specifically claimed. The description of the present invention has been given for the purpose of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present invention. The embodiments are selected and described in order to best explain the principles and practical applications of the present invention and to enable other persons of ordinary skill in the art to understand the various embodiments of the present invention with various modifications suitable for the intended specific use.

Claims

1. A method for seamless motivation based on cognitive state, comprising: obtaining the current cognitive state of a human subject; determining a target cognitive state for a human subject, wherein the target cognitive state is different from the current cognitive state of the human subject; Based on the current cognitive state and the target cognitive state, determining a saliency level for the stimulus, wherein the saliency level is configured to cause the human subject to direct spatial attention directly to the stimulus, wherein the saliency level is configured to cause the stimulus to be seamless given the current cognitive state of the human subject; outputting the stimulus at the saliency level to enable the human subject to perceive it; wherein the stimulus is presented as a visual stimulus perceived by the peripheral vision of a human subject, the visual stimulus comprising a pattern of one or more visual markers appearing in a specific direction, wherein the appearance pattern of the one or more visual markers is configured to cause the human subject to direct the spatial attention to the target.

2. The method according to claim 1, wherein: The human subject is a driver of a partially automated vehicle.

3. The method according to claim 2, wherein: The target cognitive state is associated with a hazard facing a partially automated vehicle.

4. The method of claim 2, wherein determining the significance level of the incentive is performed based on a model, wherein: The model is updated based on the driver's identified responses to the plurality of stimuli.

5. The method according to claim 2, wherein: Determining the significance level of the incentive is performed based on the driver's cognitive state, the assessed risk level, and the assessed urgency.

6. The method according to claim 1, wherein: The stimulus is above the conscious perception level of a human subject.

7. The method according to claim 1, wherein: The target cognitive state includes directing the spatial attention to a target, the stimulus being associated with the location of the target.

8. The method according to claim 7, wherein: The stimulus is a visual stimulus that is presented in a relative position in the field of view of a human subject proximate to a target relative to the field of view.

9. The method according to claim 7, wherein: The stimulus is an audio stimulus configured to be perceived by a human subject having an associated relative position, the relative position corresponding to a position of a target relative to the human subject.

10. The method according to claim 1, wherein: The stimulus is a visual stimulus, and the significance level is at least one of the following: Visually stimulating dimensions, Visually motivated opacity levels, visually stimulating brightness levels, and The duration for which the visual stimulus is presented.

11. The method according to claim 1, wherein: The stimulus is an audio stimulus, and the significance level is at least one of the following: The volume level of the audio stimulus, The frequency spectrum of the audio stimulus, a measure of the similarity of the audio stimulus relative to the background noise, and Provides the duration of the audio stimulus.

12. The method according to claim 1, wherein: The target cognitive state is minimized by the following function: f(c)=α∙Risk(c)-β∙WellBeing(c), where (c) is the cognitive state, Risk(c) is a function of risk relative to the cognitive state, WellBeing(c) is a function of the well-being of the human subject given the cognitive state, and α and β are coefficients.

13. The method according to claim 12, wherein: The Risk(c) is a function of the expected damage, defined as Risk(c) = Damage∙Probability(c), where Damage is the expected damage from the hazard and Probability(c) is a function of the probability that the damage will occur given the current cognitive state of the human subject.

14. The method according to claim 1, wherein: Determining the significance level of the incentive is based on the well-being of the human subject, wherein the well-being of the human subject is assessed using at least one of: facial expressions of human subjects; Eye saccades of human subjects; One or more body postures of a human subject; and One or more changes in the body temperature of a human subject.

15. The method according to claim 1, wherein: Determining the saliency level of a stimulus is performed using a predictive model, wherein the predictive model is configured to assess the saliency level of a stimulus so as to provide a seamless stimulus while directing the spatial attention to the stimulus, wherein the predictive model provides the assessed saliency level based on cognitive state, target cognitive state, and characteristics of the human subject.

16. The method according to claim 15, wherein: The predictive model is trained based on crowdsourced information about different people driving different vehicles.

17. The method of claim 1, wherein determining the significance level of the incentive is performed based on a prediction model; wherein: The method further comprises: Identify responses of human subjects to stimuli; Updates the predictive model to improve determination of significance levels for future uses of the predictive model.

18. The method according to claim 17, wherein: The identifying a response of the human subject to a stimulus includes obtaining a new cognitive state of the human subject, wherein the updating the prediction model is based on a difference between the new cognitive state and a target cognitive state.

19. The method according to claim 17, wherein: The response indicates that the stimulus is not seamless to the human subject, wherein updating the prediction model includes updating the model to provide a lower significance level than a significance level determined in the determining a significance level of the stimulus under the same conditions.

20. The method according to claim 17, wherein: The response indicates that the stimulus did not cause the human subject to direct the spatial attention to the stimulus, wherein updating the prediction model includes updating the model to provide a significance level that is higher than the significance level determined in the significance level of the determined stimulus under the same conditions.

21. A device for seamless stimulation based on cognitive state, comprising a processor and a coupled memory, wherein: The processor is adapted to perform: obtaining the current cognitive state of a human subject; determining a target cognitive state of a human subject, wherein the target cognitive state is different from the current cognitive state of the human subject; Determining a saliency level of a stimulus based on a current cognitive state and a target cognitive state, wherein the saliency level is configured to cause a human subject to direct spatial attention to the stimulus, wherein the saliency level is configured to cause the stimulus to be seamless to a given human subject's current cognitive state; outputting the stimulus at the saliency level to enable the human subject to perceive it; wherein the stimulus is presented as a visual stimulus perceived by the peripheral vision of a human subject, the visual stimulus comprising a pattern of one or more visual markers appearing in a specific direction, wherein the appearance pattern of the one or more visual markers is configured to cause the human subject to direct the spatial attention to the target.

22. The apparatus according to claim 21, wherein The device is integrated into a partially automated vehicle, wherein the human subject is a driver of the partially automated vehicle.

23. The apparatus of claim 22, wherein: The target cognitive state is associated with a hazard faced by the partially automated vehicle.

24. The apparatus of claim 22, wherein: Determining the significance level of the stimulus is performed based on a model, wherein the model is updated based on the driver's identified responses to the plurality of stimuli.

25. The apparatus of claim 22, wherein: Determining the significance level of the incentive is performed based on the driver's cognitive state, the assessed risk level, and the assessed urgency.

26. The apparatus of claim 21, wherein: The target cognitive state includes directing the spatial attention to a target, wherein the stimulus is associated with a location of the target.

27. The apparatus of claim 26, wherein: The stimulus is a visual stimulus that is presented in a relative position in the field of view of a human subject proximate to a target relative to the field of view.

28. The apparatus of claim 26, the stimulus being an audio stimulus configured to be perceived by a human subject having an associated relative position, wherein: The relative position is relative to the human subject, corresponding to the position of the target.

29. The apparatus of claim 21, wherein: The stimulus is a visual stimulus, wherein the significance level is at least one of: Visually stimulating dimensions, Visually motivated opacity levels, visually stimulating brightness levels, and The duration for which the visual stimulus is presented.

30. The apparatus of claim 21, wherein the stimulus is an audio stimulus, wherein The significance level is at least one of the following: The volume level of the audio stimulus, The frequency spectrum of the audio stimulus, a measure of the similarity of the audio stimulus relative to the background noise, and Provides the duration of the audio stimulus.

31. The apparatus of claim 21, wherein: The target cognitive state is minimized by the following function: f(c)=α∙Risk(c)-β∙WellBeing(c), where (c) is the cognitive state, where Risk(c) is a function of risk relative to the cognitive state, where WellBeing(c) is a function of the well-being of a human subject given the cognitive state, and where α and β are coefficients.

32. The apparatus of claim 21, wherein: Determining the significance level of the incentive is based on the well-being of the human subject, wherein the well-being of the human subject is assessed using at least one of: facial expressions of human subjects; Eye saccades of human subjects; One or more body postures of the human subject; and One or more changes in the body temperature of a human subject.

33. The apparatus of claim 21, wherein: Determining the saliency level of a stimulus is performed using a predictive model, wherein the predictive model is configured to assess the saliency level of a stimulus so as to provide a seamless stimulus while directing the spatial attention to the stimulus, wherein the predictive model provides the assessed saliency level based on cognitive state, target cognitive state, and characteristics of the human subject.

34. The apparatus of claim 33, wherein: The predictive model is trained based on crowdsourced information about different people driving different vehicles.

35. The apparatus of claim 21, wherein: The determining of the significance level of the incentive is performed based on a prediction model; wherein the processor is configured to perform: Identify responses of human subjects to stimuli; Updates the predictive model to improve determination of significance levels for future uses of the predictive model.

36. The apparatus of claim 35, wherein: The identifying a response of the human subject to a stimulus includes obtaining a new cognitive state of the human subject, wherein the updating the prediction model is based on a difference between the new cognitive state and a target cognitive state.

37. The apparatus of claim 35, wherein: The response indicates that the stimulus is not seamless to the human subject, wherein updating the prediction model includes updating the model to provide a lower significance level than a significance level determined in the determining a significance level of the stimulus under the same conditions.

38. The apparatus of claim 35, wherein: The response indicates that the stimulus did not cause the human subject to direct the spatial attention to the stimulus, wherein updating the prediction model includes updating the model to provide a significance level that is higher than the significance level determined in the significance level of the determined stimulus under the same conditions.

39. A non-transitory computer-readable medium, wherein: The medium retains program instructions that, when read by a processor, cause the processor to perform the following operations: obtaining the current cognitive state of a human subject; determining a target cognitive state of a human subject, wherein the target cognitive state is different from the current cognitive state of the human subject; determining a salience level of the stimulus based on the current cognitive state and the target cognitive state, wherein the salience level is configured to cause the human subject to direct spatial attention to the stimulus, wherein the salience level is configured to cause the stimulus to be seamless to the current cognitive state of the given human subject; Outputting stimuli at a salience level such that they can be perceived by human subjects; wherein the stimulus is presented as a visual stimulus perceived by the peripheral vision of a human subject, the visual stimulus comprising a pattern of one or more visual markers appearing in a specific direction, wherein the appearance pattern of the one or more visual markers is configured to cause the human subject to direct the spatial attention to the target.

40. The non-transitory computer readable medium of claim 39, wherein: The non-transitory computer-readable medium is at least partially loaded into a storage unit of a partially autonomous vehicle, wherein the human subject is a driver of the partially autonomous vehicle.

41. The non-transitory computer readable medium of claim 40, wherein: The target cognitive state is associated with a hazard faced by the partially automated vehicle.

42. The non-transitory computer readable medium of claim 40, wherein: Determining the significance level of the stimulus is performed based on a model, wherein the model is updated based on the driver's identified responses to the plurality of stimuli.

43. The non-transitory computer readable medium of claim 40, wherein: Determining the significance level of the incentive is performed based on the driver's cognitive state, the assessed risk level, and the assessed urgency.

44. The non-transitory computer readable medium of claim 39, wherein: The target cognitive state includes directing the spatial attention to a target, wherein the stimulus is associated with a location of the target.

45. The non-transitory computer readable medium of claim 44, wherein: The stimulus is a visual stimulus that is presented in a relative position in the field of view of a human subject proximate to a target relative to the field of view.

46. ​​The non-transitory computer readable medium of claim 43, wherein: The stimulus is an audio stimulus configured to be perceived by a human subject having an associated relative position, wherein the relative position is relative to the human subject corresponding to a position of a target.

47. The non-transitory computer readable medium of claim 39, the stimulus being a visual stimulus, wherein: The significance level is at least one of the following: Visually stimulating dimensions, Visually motivated opacity levels, visually stimulating brightness levels, and The duration for which the visual stimulus is presented.

48. The non-transitory computer readable medium of claim 39, the stimulus being an audio stimulus, wherein: The significance level is at least one of the following: The volume level of the audio stimulus, The frequency spectrum of the audio stimulus, a measure of the similarity of the audio stimulus relative to the background noise, and Provides the duration of the audio stimulus.

49. The non-transitory computer readable medium of claim 39, wherein: The target cognitive state is minimized by the following function: f(c)=α∙Risk(c)-β∙WellBeing(c), where (c) is the cognitive state, where Risk(c) is a function of risk relative to the cognitive state, where WellBeing(c) is a function of the well-being of a human subject given the cognitive state, and where α and β are coefficients.

50. The non-transitory computer readable medium of claim 39, wherein: Determining the significance level of the incentive is based on the well-being of the human subject, wherein the well-being of the human subject is assessed using at least one of: facial expressions of human subjects; Eye saccades of human subjects; One or more body postures of the human subject; and One or more changes in the body temperature of a human subject.

51. The non-transitory computer readable medium of claim 39, wherein said determining a significance level of a stimulus is performed using a predictive model, wherein: The prediction model is configured to assess a saliency level of a stimulus so as to provide seamless stimulation while directing the spatial attention to the stimulus, wherein the prediction model provides the assessed saliency level based on cognitive state, target cognitive state, and characteristics of the human subject.

52. The non-transitory computer readable medium of claim 51, wherein: The predictive model is trained based on crowdsourced information about different people driving different vehicles.

53. The non-transitory computer readable medium of claim 39, wherein: The determining of the significance level of the incentive is performed based on a predictive model; Wherein, the processor is configured to execute: Identify responses of human subjects to stimuli; Updates the predictive model to improve determination of significance levels for future uses of the predictive model.

54. The non-transitory computer readable medium of claim 53, wherein: The identifying a response of the human subject to a stimulus includes obtaining a new cognitive state of the human subject, wherein the updating the prediction model is based on a difference between the new cognitive state and a target cognitive state.

55. The non-transitory computer readable medium of claim 53, wherein: The response indicates that the stimulus is not seamless to the human subject, wherein updating the prediction model includes updating the model to provide a lower significance level than a significance level determined in the determining a significance level of the stimulus under the same conditions.

56. The non-transitory computer-readable medium of claim 53, the response indicating that the stimulus did not cause the human subject to direct the spatial attention to the stimulus, wherein: The updating of the prediction model includes updating the model to provide a significance level that is higher than a significance level determined in the determining a significance level of the incentive under the same conditions.

Images