Support system using relevance-dependent stimulus modulation

Abstract

To provide a support system using stimulation modulation depending on relevance.SOLUTION: A method for supporting a person to execute a task in a dynamic environment comprises the steps of: acquiring data regarding the dynamic environment; and predicting at least one behavior of an object or event in the dynamic environment on the basis of the acquired data. The method progresses by determining a relevance measurement value of the at least one behavior of the object or event on the basis of prediction. Next, a modulation signal is generated, and changes at least one signal parameter as a function of the determined relevance. Next, at least one direction stimulation signal is modulated by using the modulation signal. The at least one modulated direction stimulation signal further depends on a direction of the at least one behavior to be predicted of the object or event to a supported person. Next, the at least one modulated stimulation signal is output to the supported person.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an assistance system for assisting humans in mobility-related tasks, for example an assistance system for a vehicle. The present invention relates to an assistance system, an assistance method, and a human-machine interface for the assistance system and method. [Background technology]

[0002] Assistance systems for mobility-related tasks, such as assisting an operator in traffic navigation in the automotive, aviation or marine domains, are well established. Assistance systems can positively contribute to the operator's situation assessment, to the operator's navigation skills and therefore to the safety in the mobile environment in which the assisted person operates.

[0003] Timely perception of task-relevant and safety-relevant information is not always guaranteed during mobility tasks in an environment that includes multiple factors that may focus or affect a person's attention. For example, if a person is driving his or her own vehicle ("self-vehicle") while simultaneously listening to music played on the infotainment system of the vehicle, the music may interfere with sounds from the environment that contain relevant information for performing the driving task. Therefore, it is more difficult for a person to perceive this sound together with the relevant information.

[0004] Especially in a highly dynamic environment such as the current road traffic environment, a complex situation arises with multiple options for the next action to be taken by a person driving an ego vehicle. A person may have difficulty selecting one specific action that maximizes utility from multiple possible actions if all available relevant information is not considered during the selection process. For example, when a vehicle preceding the ego vehicle on a road with multiple lanes in one direction slows down, the person driving the ego vehicle may have to choose between slowing down the ego vehicle, possibly sacrificing energy efficiency, or changing into another available lane among the further lanes. The choice by the person depends on information about the current and predicted occupancy of nearby lanes in the near future. However, if the person does not obtain information about currently available lanes fast enough due to the dynamically changing traffic scenario in the dynamic environment, the currently existing gap that allows the ego vehicle to change lanes into the nearby lane may no longer be available at the time of decision.

[0005] Regarding the automotive traffic environment, there are currently approaches to enhance existing vehicles with the addition of wearable devices, which can even increase vehicle safety.

[0006] US Patent Application Publication No. 2017 / 0156000 discloses a wearable earphone configured to wirelessly connect to an ego-vehicle. Once connected to the vehicle, the earphone enters a driving mode. In the driving mode, the earphone senses ambient sounds using a microphone associated with the earphone and reproduces the sensed ambient sounds using its speaker. The earphone can combine an audio stream including ambient sounds from inside and outside the vehicle cabin with further audio streams from an entertainment or infotainment system and audio messages associated with vehicle status information or vehicle alert conditions. An intelligent control system can modify audio characteristics of the reproduced audio stream, in particular reducing or amplifying the amplitude of the ambient sounds or parts thereof. Modifying the audio characteristics of the reproduced audio stream can rely on detecting a particular action of the driver of the vehicle or providing audio cues regarding the presence or absence of other vehicles in the ego-vehicle's environment.

[0007] US2017 / 0156000 is silent on risk assessment of traffic scenarios. The earphones representing the focus of US2017 / 0156000 do not provide the ability to provide directional cues to guide the assisted person in a specific direction. The assisted person's focus of attention, the assisted person's alertness, historical attention patterns or other possible targets of traffic participants are all ignored. In short, US2017 / 0156000 provides limited improvement of situational awareness, both regarding the predicted development of a dynamic scene and limited to only audible information. In essence, US2017 / 0156000 attempts to overcome the disadvantages and dangers of wearing earphones in the own vehicle.

[0008] US Patent No. 9,650,041 discloses a method for implementing a predictive human-machine interface to prevent vehicle accidents based on detecting a driver's gaze vector and detecting the positions of nearby objects in the vehicle's environment and their velocities. The method determines a vehicle accident risk under a detected surrounding environment and a driver's situation awareness of the detected surrounding environment. The method can activate a vehicle alert based on the determined vehicle accident risk and the driver's situation awareness. The method can adjust a type of vehicle alert based on the determined vehicle accident risk, the driver's situation awareness, based on a determined level of driver experience of a driver of the host vehicle, and further based on weather and traffic conditions. Adjusting the type of vehicle alert includes selecting a predefined alert type or determining whether to provide only a minimum number of help messages or to provide all help messages to the assisted driver. The alert type can distinguish between driver-side alerts and passenger-side alerts. The method can at least partially lock out an infotainment system of the host vehicle in association with adjusting the vehicle alerts and the driver experience level. No. 9,650,041 discloses a human-machine interface in combination with accident prevention systems (driver assistance systems) such as lane keeping assist systems, collision mitigation braking systems, and lane change and blind spot indication systems, etc. The accident prevention systems can exercise active control of the vehicle when a determined vehicle accident risk requires immediate action.

[0009] The method of U.S. Patent No. 9,650,041 has the disadvantage that it allows for binary decisions such as whether to display an alert, whether to select a specific alert type, whether to lock out the infotainment system, whether to detect a vehicle in an alert zone, etc. Supporting the assisted person based on such binary decisions and alerts may lead to a reduced situational awareness of the assisted person ("over-reliance on the assistance system") or a reduced acceptance of the assistance system by the assisted person ("annoying alerts"). Summary of the Invention [Problem to be solved by the invention]

[0010] It remains the task of assisting people to perform tasks in dynamic environments by helping the assistee to focus their attention on the scene relevant to the task and providing assistance according to their risk potential regarding the task, which would result in a high acceptance of the assistance from the person's point of view. [Means for solving the problem]

[0011] A method for assisting a person according to a first aspect, an assistance system according to a second aspect, and a computer program according to a third aspect solve a problem.

[0012] The dependent claims define further advantageous embodiments of the invention.

[0013] In a first aspect, a method for assisting a person to perform a task in a dynamic environment includes acquiring data on the dynamic environment and predicting at least one behavior of an object or event in the dynamic environment based on the acquired data. The method proceeds to a step of determining a relevance measure of the at least one behavior of the object or event based on the prediction. A modulation signal is generated, the modulation signal modifying at least one signal parameter as a function, preferably a continuous function, of the determined relevance. Then, at least one directional stimulus signal is modulated with the generated modulation signal, the at least one modulated directional stimulus signal further depending on a direction of the predicted at least one behavior of the object or event relative to the assisted person. Then, the at least one modulated stimulus signal is output to the assisted person.

[0014] In a second aspect, an assistance system for assisting a person to perform a task in a dynamic environment comprises an acquisition interface configured to acquire data regarding the dynamic environment. A processor is configured to predict at least one behavior of an object or event in the dynamic environment based on the acquired data, and to determine a relevance measure of the at least one behavior of the object or event based on the prediction. The processor generates a modulation signal, the modulation signal varying at least one signal parameter as a continuous function of the determined relevance. The processor is then configured to modulate at least one directional stimulus signal with the modulation signal, the at least one modulated directional stimulus signal further depending on a direction of the predicted at least one behavior of the object or event relative to the assisted person. The system further comprises at least one actuator configured to output the at least one modulated stimulus signal to the assisted person.

[0015] In a third aspect, the present invention relates to a computer program comprising program code means for performing the steps according to one of the embodiments of the method for assisting a person, when the computer program is run on a computer or digital signal processor.

[0016] A third aspect may relate to a computer program product comprising program code means stored on a machine readable medium for performing the steps according to one of the embodiments of the method for assisting a person, when the program is run on a computer or digital signal processor.

[0017] A third aspect may relate to a non-transitory computer-readable storage medium embodying a program of machine-readable instructions executable by a digital processing device to cause the digital processing device to perform operations according to an embodiment of a method for assisting a human.

[0018] The discussion of the embodiments of the present invention refers to the accompanying drawings. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 shows a flow chart for a support method of a method embodiment. [Diagram 2] FIG. 2 illustrates the application of the invention in a road traffic scenario. [Diagram 3] FIG. 1 illustrates a risk-dependent stimulus modulation curve and a risk-dependent stimulus modulation reversal curve, according to an embodiment. [Figure 4] FIG. 13 illustrates risk-dependent stimulus modulation reversal curves for various initial risks, according to an embodiment. [Diagram 5] FIG. 2 shows an overview of structural elements of a system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The drawings, wherever possible, use the same reference numbers to designate identical or corresponding elements to facilitate understanding, and for the sake of brevity, discussion of the same reference numbers in different drawings will be omitted.

[0021] The present disclosure combines several advantageous effects with respect to known approaches: The influence of factors extraneous to or potentially distracting from the task can be reduced.

[0022] It can increase the salience of task-relevant factors in the environment.

[0023] To optimize task-appropriate assistance for the supportee, information about the supportee's attention and attention history can be taken into account.

[0024] It is possible to take into account information about possible targets of other agents in a dynamic environment in order to optimize assistance to the person according to an overall safety or efficiency target.

[0025] Although means of drawing a person's attention to possible sources of danger are known in the art, the present method improves over known means by reducing the salience of stimuli to the supportee that are potentially distracting from the (main) task. Due to targeted signal modulation, associated signal amplification and directional correspondence, the supportee can improve his / her understanding of surrounding objects, their behavior and reference to predicted events in a dynamic environment. Furthermore, the support is maintained beyond the duration of the task distractor, addressing the long-term demands of the supportee. Information about the supportee's attention is taken into account to optimize the assistance provided.

[0026] The most direct impact of the method on performance measurement is to support the recipient in increasing the speed of information perception in a dynamic environment, as well as to improve comprehension and facilitate corresponding responses. This can be achieved by providing easy access to relevant information in a dynamic environment.

[0027] Furthermore, the features driving the assistance can support a wide range of task goals beyond direct individual utility and can trigger positive secondary effects of traffic regulation.

[0028] A method according to an embodiment includes determining a status of a supported person and adapting a risk-dependent stimulation modulation based on the determined status.

[0029] The determined state of the supported person may include at least one of an attention state of the supported person, an attention history of the supported person, and an alertness level of the supported person.

[0030] The at least one stimulation signal may include at least one of an audio signal, a visual signal, and a tactile signal.

[0031] The at least one stimulation signal may include an artificial stimulation signal, in particular a recorded sound signal.

[0032] The at least one signal parameter may be one of a stimulation amplitude, a stimulation frequency, a stimulation repetition rate, and a perceived signal direction.

[0033] The perceived signal direction of the directional stimulus signal includes the angular direction toward the spatial source of the directional stimulus signal as perceived by the assisted person, or toward an anticipated behavior or event in the environment relative to the assisted person.

[0034] The terms stimulation amplitude, stimulation frequency, stimulation repetition rate and perceived signal direction encompass not only discrete values ​​of each parameter, but also ranges of values ​​for each parameter.

[0035] In one embodiment, the at least one stimulus signal includes at least one of an audio signal stream and the at least one signal parameter is one of an audio volume and an audio signal frequency band.

[0036] A method according to an embodiment includes detecting a further stimulus perceivable by the assisted person and including the detected further stimulus in the at least one directional stimulus signal before modulating the at least one directional stimulus signal.

[0037] Detecting the further stimuli perceptible by the assisted person may include stimuli that are pre-determined in advance, for example during the design phase of the assistance system, and are assumed to be perceptible by the assisted person in a current scenario, for example in a dynamic environment. In one example, the method may assume that the assisted person, for example the driver of the vehicle, can hear the audio output by the vehicle radio, and the method may modulate the stimulus signal including the audio output by the vehicle radio without actually checking whether the assisted person can actually perceive the audio signal. Alternatively or in addition, the further stimuli perceptible by the assisted person may include stimuli including, for example, the audio signal output of the radio, sounds recorded in a real situation, for example from another vehicle driving in the assisted person's environment during the execution time of the assistance method.

[0038] Detecting the further stimuli perceivable by the assisted person is the basis for reducing the impact of potentially task-irrelevant or distracting stimuli on the assisted person. Prior to modulating the at least one directional stimulus signal, the detected further stimuli are included in the at least one directional stimulus signal, which then supports focusing the assisted person's attention on task-relevant elements by increasing the salience of the relevant stimuli with respect to the predicted behavior of the object or event.

[0039] In the method according to the embodiment, the continuous function continuously increases the salience of at least one stimulation signal perceived by the assisted person when the determined relevance increases. Alternatively or additionally, the continuous function continuously decreases the perceived distance to the predicted behavior of the object or event in at least one directional stimulation signal perceived by the assisted person when the determined relevance increases.

[0040] Furthermore, the continuous function may continuously decrease the salience of the at least one stimulation signal perceived by the assisted person in the event of a decrease in the determined relevance.

[0041] This reduces the distraction of the supportee as the relevance decreases, allowing the supportee to focus their attention on other elements in the environment, while other relevant elements become more salient.

[0042] Due to the determined risk, returning the infotainment sound level to pre-modulation can, for example, be offset by a certain time delay or can follow a recovery curve of the modulated signal having a low gradient (flat slope) of modulation strength with respect to relevance.

[0043] Delaying the return to normal levels, particularly the particular level prior to communication of the determined risk, provides the assisted person with some time to readjust to the resolved situation, including risk as a particular example of relevance.

[0044] According to an embodiment, the value of the slope of the continuous function in the case of a decrease in the salience of the at least one stimulation signal over time is smaller than the value of the slope of the continuous function for an increase in the salience of the at least one stimulation signal over time.

[0045] This particular example of different gradients for increasing saliency on the one hand and decreasing saliency on the other hand ensures that the assisted person can direct his / her attention early to those aspects in the environment that show high relevance. Moreover, the method instructs the assisted person to maintain a sufficient degree of attention to these aspects even when the relevance starts to decrease again. Thus, the method instructs the assisted person to make his / her own judgment about the relevance of the predicted behavior of the object or the predicted event.

[0046] The modulation of the at least one directional stimulus signal in the case of a decrease in relevance causes a value of the slope of the modulation signal to increase with decreasing relevance.

[0047] In a further embodiment of the method, the modulation of the at least one directional stimulus signal in the case of a decrease in the degree of relevance comprises a smaller value of the slope of the modulation signal the higher the initial value of the degree of relevance.

[0048] According to a further advantageous embodiment of the method, the determined relevance is a function of at least one task goal, the at least one task goal comprising at least one of road safety, comfort of the assisted person, status of the assisted person, and a task goal of another person.

[0049] The determined relevance may be a risk metric associated with at least one predicted behavior of the object or predicted event.

[0050] This specification uses the term salience to refer to a salient state or condition. Salience specifically refers to the relative importance or prominence of a portion of a sign. When considered in other contexts, the salience of a particular sign helps an individual quickly rank large amounts of information by importance, thereby paying attention to what is most important. This process keeps an individual from being overwhelmed by information overload.

[0051] Salience (or saliency) thus describes how perceptually prominent an item is compared to its surroundings. Factors that commonly affect saliency are contrasts, such as a bright spot in a dark environment, or a fast moving object among slower moving objects, but higher level factors such as personal relevance can also affect saliency.

[0052] An earcon refers to a brief, characteristic sound that represents a particular event or conveys information. In particular, an earcon is the auditory equivalent of an icon. A characteristic sound that represents a particular piece of information, such as a computer beep signaling an error. In this context, earcons refer to such artificial sounds, as well as auditory icons that are based on actual recordings of the characteristic sounds.

[0053] The elements in the environment or the relevant elements refer to objects, structures in the environment and the current scenario that affect the decisions and actions of the assisted person performing the task. Elements can be stationary (lane markers, traffic lights) or moving (vehicles, pedestrians), for example in road traffic applications. Elements can be predicted events in the environment (e.g. predicted collisions).

[0054] The term perceivability refers to the degree to which an entity can be perceived by the assisted party. Typically, perceivability is determined by the degree to which the perceiving entity's sensors can be triggered.

[0055] In this context, relevance refers to risk. Relevance can be considered to be proportional to risk or a function of both risk and driver state.

[0056] Fig. 1 shows a flow chart for an assistance method of an embodiment. The discussion of Fig. 1 shows an assistance method applied in a dynamic environment (road traffic environment) by an ego-vehicle 1 operated by a assisted person (driver) who benefits from the support of the assistance method.

[0057] The method for assisting a human operator of an ego-vehicle 1 in carrying out a task in a dynamic environment comprises a step S1 of acquiring sensor data.

[0058] The method acquires data in a dynamic environment using sensors located on an ego-vehicle 1 operated by the assisted person in a road traffic environment. The term sensor includes built-in sensors operating on radar principle, lidar principle or camera principle, but also covers x-to-vehicle communication for acquiring data in a dynamic environment.

[0059] Based on the acquired data, the method proceeds in step S2 by predicting at least one behavior of an object or event in the dynamic environment.

[0060] In step S3, the method proceeds by determining a relevance measure of at least one behavior of the object or event based on the prediction.

[0061] Step S3 is followed by step S4, in which the method determines whether the determined relevance measure exceeds at least one predetermined threshold level. If it does, the method proceeds to step S5, determining the modulation signal. If, in step S9, the determined relevance measure does not exceed at least one predetermined threshold level, the method concludes that the predicted scene elements of the current scene in the dynamic environment do not require providing assistance to the assisted person.

[0062] In step S5, the method obtains at least one directional stimulus signal.

[0063] Further, in step S6, the method generates a modulated signal. The modulated signal varies at least one signal parameter as a continuous function of the determined relevance. Step S6 proceeds by modulating at least one directional stimulus signal with the modulated signal to generate at least one modulated directional stimulus signal. The resulting at least one modulated directional stimulus signal depends on the direction of at least one predicted behavior of the object or event relative to the assisted person.

[0064] In step S7, the assistance method provides the generated modulated directional stimulus to at least one actuator. In step S8, the at least one actuator outputs at least one modulated stimulus signal to the assisted person, and proceeds to end the current processing cycle of the assistance method.

[0065] The assistance method makes it possible to reduce the influence of factors that are extraneous to the task or that potentially distract from the task performed by the assistance recipient.

[0066] Thus, the method assists the assisted person in performing a primary task, e.g., operating the own vehicle 1 in a traffic environment, while reducing distractions resulting from secondary tasks, e.g., following information provided by an on-board infotainment system. The assistance method achieves this by increasing the relative perceptibility or salience of factors by reducing the perceptibility or salience of other interfering factors, in particular those that the assistance system 20 judges to be irrelevant or distracting.

[0067] For example, sounds produced by another vehicle through its engine and road / air friction are easier to notice and identify when the radio in your own vehicle is turned off than when it is turned on, especially at a high volume.

[0068] Thus, one part of the invention consists of reducing aspects that affect the perceptibility or salience of task-irrelevant, distracting or interfering factors in proportion to the need for the user to pay attention to different factors.

[0069] With regard to the application field of driver assistance in a traffic environment, the assisted person (driver) may be exposed to a wide variety of potentially distracting sounds, such as sounds from the entertainment system of the own vehicle, a telephone call received via a wireless telephone, or a conversation with another occupant in the own vehicle 1. When the assistance system 20 predicts an approaching collision risk or a navigation option, the attention of the assisted person should be focused as much and as quickly as possible on the traffic environment, e.g. the road and other traffic participants. In this case, the assistance method reduces the perceptibility or saliency of distracting sounds affecting the assisted person by modifying certain characteristics of the sounds in proportion to the perceptibility or saliency of the sounds conveying the importance of the predicted event. Such a certain characteristic of the at least one directional stimulus signal modified by the modulation signal can be the volume or the frequency spectrum of the at least one directional stimulus signal.

[0070] For example, as the determined relevance measure, or in other words the risk of a potential hazard, increases, the volume of the infotainment system decreases.

[0071] Alternatively, to prevent the assisted person from attempting to counter such reduction by manually changing the volume of the infotainment system, the volume can remain constant or at least at an audible level, but the perceived presence of the infotainment sound is reduced by reducing certain frequency components in the frequency spectrum of the infotainment sound signal. An embodiment can implement this by low-pass filtering the infotainment sound signal, the cut-off frequency of the low-pass filter being dependent on the determined relevance measure. The low-pass filtering of the infotainment sound signal, which is a specific example of modulation of the directional stimulus signal, generates a perception of distance for the assisted person related to the determined relevance measure.

[0072] Once the predicted event associated with the risk no longer exists or is within the predetermined limits of the assistance system, the potentially distracting infotainment sound signal may return to the level that was present before the assistance system 20 began modulating it.

[0073] To give the assisted person some time to readjust to the resolved situation involving risk, returning the infotainment sound level to pre-modulation due to the determined risk can, for example, be offset by a certain time delay or can follow a recovery curve of the modulation signal with a low gradient (flat slope) of modulation intensity versus risk.

[0074] Additionally or alternatively, the assistance system 20 can vary the curve of the modulation signal for each speaker of the infotainment system 23 individually. For example, depending on how well the respective speaker is spatially aligned with the direction of the predicted element, e.g., risk source, the modulation can follow a fast increase, e.g., have a high slope (steep incline) when close to the same direction, or follow a small slope when having a larger directional difference to the risk element in the environment from the viewpoint of the assistance recipient. The assistance system 20 accordingly provides an early indication of the relevant direction toward the predicted risk element in the environment by a relatively fast local change of the stimulation characteristic of the at least one modulated directional stimulation signal from the at least one unmodulated directional stimulation signal.

[0075] Additionally or alternatively, the assistance method makes it possible to increase the salience and therefore the influence of task-related factors or factors that focus attention on the task performed by the assisted person in the environment.

[0076] In combination with the reduction of the influence of distracting or irrelevant factors, the assistance system 20 can increase the relative perceptibility or salience of relevant elements with potential risks in the current scenario, for example by directly amplifying the signal components of the at least one directional stimulus signal indicative of the respective relevant element. If sound is used as a medium for the at least one directional stimulus signal, the assistance method can achieve this by recording the sounds originating from the relevant elements in the environment with a microphone sensor. The assistance method can then amplify the recorded sounds based on the modulated signal and output the amplified sounds via speakers of an infotainment system arranged inside the ego-vehicle 1 in the particular application field of driver assistance in road traffic environments.

[0077] Additionally or alternatively, the assistance method may amplify at least one directional stimulus signal in a portion of the frequency spectrum of at least one particular directional stimulus signal that corresponds to a portion of the frequency spectrum of the directional stimulus signal that is reduced due to representing undesirable distractions (noise) from the task and anticipated elements in the environment presenting risk.

[0078] In particular, when a sound signal is regarded as at least one directional stimulus signal, the assistance method can modulate based on modulation signal characteristic parameters of the at least one directional stimulus signal, such as pitch, sound quality, timing and perceived spatial location, to support the assisted person in separating relevant elements from irrelevant elements in the environment.

[0079] Additionally or alternatively, the assistance method may use at least one recorded artificial stimulus signal indicative of the respective element representing a risk. The at least one artificial stimulus signal in the application field of sound signals may be realized by using earcons, which are characteristic of a class of related elements to which the determined element is classified to belong. If the predicted element includes an object such as a vehicle, the earcons may be a recorded sound file of engine noise for the vehicle. If the predicted element includes an object such as a pedestrian, the earcons may be a sound file of pedestrian footsteps.

[0080] Additionally or alternatively, the assistance method may generate and output at least one directional stimulus signal by varying the output balance between different speakers of the infotainment system 23 of the host vehicle 1 arranged in a stereo or surround setting, such that the apparent position of the sound source corresponds to the direction relative to the assisted person of a relevant element representing a risk in the environment.

[0081] When predicting multiple relevant elements in the environment that would result in being classified as separate earcons, alternative strategies can also be applied to avoid sensory overload for the assisted person.

[0082] When multiple relevant elements as risk sources classified as separate earcons are oriented in roughly the same direction ("closely aligned"), the assistance method can use a single representative sound instead of a mixture of the respective multiple sounds associated with the multiple relevant elements.

[0083] In cases where the orientations of multiple relevant elements representing multiple simultaneous risk sources vary over a wide angle ("misaligned"), only the most relevant subset of the set of earcons can be represented by a representative sound without the directionality of the at least one directional stimulus signal. Alternatively, the assistance method can output a representative sound for a single most relevant orientation of the multiple relevant elements.

[0084] Additionally or alternatively, the assistance method may modulate at least one directional stimulus signal to propel the assisted person towards a predefined action, such as reducing the speed v1 of the ego vehicle 1. Examples of directional stimulus signals that achieve this target include artificial engine sounds suggestive of an accelerating vehicle, a general alarm sound, sounds similar to the squealing of vehicle tires, and stimuli in the nature of crashing sounds.

[0085] Additionally or alternatively, the method includes utilizing sounds already present in the own vehicle 1, e.g., music provided by the infotainment system 23, to direct the attention of the assisted person in the direction of relevant elements representing potential risks in the environment. Instead of or in addition to modulating the characteristics (parameters or properties) that affect the salience of at least one directional stimulus signal, virtual spatial sound sources already present in the own vehicle 1 are adapted to align with the direction of relevant elements representing risks by modulation with a modulating signal. This results in a shift of spatial audible elements to which the assisted person is likely already paying attention. In particular, the assistance method may perform such a spatial shift for a portion of the frequency spectrum of at least one directional stimulus signal. Thereby, the assistance method generates the effect of a simulated change in position of some equipment, e.g., music output by the infotainment system 23.

[0086] Additionally or alternatively, by combining sound data acquired by a microphone sensor recording sounds from within the own vehicle 1, even the position of the occupant can be shifted from the position of the assisted person to a virtual position in the same direction as the direction towards the relevant element representing the risk, by a corresponding speaker output of at least one modulated directional stimulus signal.

[0087] Alternatively, the assistance method can utilize simulated sound sources in the ego-vehicle 1 to shift their apparent position in the opposite direction of the relevant element representing the risk, and thus away from the directional stimulus representing the relevant element. This brings the advantage of promoting the separation of attention between the relevant stimuli and irrelevant or at least less relevant stimuli from the perspective of the supported person. The assistance method creates the intuitive effect of a relevant event displacing irrelevant elements in the environment until the situation is resolved. To return to the previous situation, the supported person is required to ensure that he or she is paying sufficient attention to the relevant element representing the risk in order to reduce the associated relevance measure.

[0088] Alternatively, instead of aligning the virtual stimulation position or orientation from the viewpoint of the supportee with the current position or orientation of the respective relevant element representing the risk, a predicted (anticipated) future position or orientation can be used instead of at least one modulated directional stimulation signal. The prediction of the future development can provide, for example, a future position for a certain time, for example, 0.5 to 2 seconds ahead. This feature is particularly advantageous in highly dynamic environments. Furthermore, situations involving delays between signal generation and output on the one hand, and between perception and understanding by the supportee, may result in the conveyed position information being out of date at the time of the user's own perception.

[0089] Alternatively or additionally, modulating the at least one directional stimulation signal can include amplification using multi-modal signaling. For example, the method can rely on haptic actuators to act as additional cues to enhance information and potentially improve auditory perception by the aided person.

[0090] Alternatively or additionally, the tactile stimulation involves an actuator that tightens the seat belt depending on the risk.

[0091] Alternatively or in addition, visual cues using LED arrays, HUD icons or features in a meter display located within the host vehicle 1 can supplement the at least one modulated directional stimulus signal to improve the assistant's situational awareness based on the at least one directional stimulus signal.

[0092] Multimodal cues can help the assisted person to distinguish between multiple reasons for the reduction of task-irrelevant stimuli. For example, if a change in sound characteristics, such as the volume or frequency spectrum of the infotainment system 23, can also be caused by task-irrelevant events, such as traffic news, navigation instructions, or the occupant changing the sound settings, additional input from different modalities can make it clear to the user that the primary task, such as risk avoidance, is the cause of the signal change. This improved understanding helps prevent erroneous task prioritization, such as the assisted person trying to change the sound settings of the infotainment system 23 instead of focusing on the current scene in the environment. If the assisted person focuses on the sound settings instead of the scene in front of them when the volume is lowered or another sound characteristic changes, this person will increase the risk instead of paying more attention. A stimulus signal controlling the further tightening of the seat belt makes it easier to interpret the reason for the volume drop of the sound signal output by the infotainment system 23, thereby driving the assisted person towards the correct action in response.

[0093] Another approach to prevent focus on the sound settings is to use actions taken by the assisted person to manipulate the sound settings, e.g., volume knob rotation, voice commands, as triggers for accelerating the modulation of the modulated signal, which provides additional feedback inside the vehicle to support the understanding of the support of the modified sound signal.

[0094] The assistance method can achieve an improvement in the sensitivity of the support recipient to the increase in the saliency of the stimuli pointing to the relevant element by the temporal separation between the irrelevant "noise" saliency and the task-relevant "signal" saliency. The reduction of the "noise" saliency precedes the increase in the "signal" saliency in time, rather than simultaneously. This creates a short silent period of the modulated characteristics of the at least one directional stimulus signal. This silent period increases the contrast of the at least one modulated stimulus signal with respect to the at least one unmodulated stimulus signal. This is particularly advantageous when the predetermined threshold for the decision of whether to assist the support recipient is implemented at a time immediately after the support recipient still has time to respond, at a low cost. When the time constraints and associated risks increase, the modulation of the at least one directional stimulus signal can occur at least one of a faster rate and with more temporal overlap between the at least one directional stimulus signal and the at least one modulated directional stimulus signal.

[0095] Furthermore, by evaluating what level of modulation is required for at least one directional stimulus signal, the assistance method can take into account the actual soundscape near the ears of the assisted person.

[0096] By integrating microphone sensors, preferably stereo microphones in the headrest and / or the ceiling above the driver's head, it is possible to estimate which sounds actually reach the assisted person. The modulation signal can then be adapted to optimize the estimated perceived sound, rather than simply changing the parameters of at least one directional stimulus signal. This allows the assistance method to take into account sound sources that are not directly controlled by the assistance system. Furthermore, the unique noise of the vehicle and traffic-dependent noise variations are taken into account. Adding a feedback loop to the processing further allows active noise cancellation techniques to be included in the processing of the assistance method.

[0097] The assistance method may further take into account information regarding the attention of the assisted person, the attention history of the assisted person, and the alertness of the assisted person to optimize the assistance provided by the at least one modulated directional stimulation signal.

[0098] The modulation signal can be generated based on at least one of information about the attention state of the support recipient and attention history of the support recipient. The sensor of the support system 20 enables monitoring of the support recipient using a process of eye tracking and head tracking, measuring skin conductance and heart rate to generate a model of the support recipient's attention.

[0099] A assisted person whose gaze direction substantially deviates from a relevant direction of an object in the environment, as determined during prediction, may benefit from stimulation in at least one modulated directional stimulation signal that gradually draws attention away from the undesirable direction of the determined actual gaze direction and towards the relevant direction of the object.

[0100] Alternatively, modulation with the modulating signal can be directly a function of multiple variables such as the apparent distraction from the relevant direction of the subject and the importance of relevant elements in the environment.

[0101] Information about the attention state of the supportee and the attention history of the supportee can be used to determine which elements represent the most relevant and therefore useful for the supportee to observe in order to adequately deal with the current scenario.

[0102] The attention history can take into account the generation of at least one modulated directional stimulus signal where temporal integration may be contingent on specific scenario characteristics, e.g., longer for a slowly changing environment and shorter in a highly dynamic environment with fast changes in the scenario.

[0103] The confidence level regarding the assistance recipient's situation awareness, determined based on attention and attention history, can be communicated directly to the assistance recipient by the actuator. Such a feature can be beneficial in ensuring understanding of the various factors that may contribute to the signal changes.

[0104] In addition to monitoring the assisted person, other people, such as passengers in the own vehicle 1, can also be monitored to determine the most suitable modulation level for the modulated signal. For example, if the passenger's behavior clearly indicates that the assisted person as the driver of the own vehicle 1 may be distracting, it may be appropriate to generate a modulated signal that includes high salience scene-enhancing stimuli. If the passenger is more passive or even supposed to be asleep, a more subtle guide through basic noise reduction may be sufficient in the modulated signal.

[0105] Furthermore, occupants whose behavior indicates concern regarding potential risks in a dynamic environment may be supported to include this concern by an amplified signal of the relevant element or of the direction of the inferred risk in at least one modulated directional stimulus signal.

[0106] For attention-based modulation signals, asymmetric resilience with appropriate gradients is also advantageous. If monitoring of the aided person's gaze pattern or heart rate indicates increased alertness or uncertainty about the situation, increased salience can be maintained for directions toward other vehicles 2 in the environment contained in at least one modulated directional stimulus signal until the aided method determines that the aided person has regained his or her certainty.

[0107] By obtaining information about expected or communicated targets of other agents in a dynamic environment, it becomes further possible to optimize assistance to the supported party according to global safety or task-as-efficiency targets.

[0108] In addition to basing the modulation signal exclusively on factors directly relevant to the assisted person, it can take into account known or assumed goals of other agents acting in the dynamic environment, which can trigger actions that have positive secondary effects that indirectly benefit a larger group of agents beyond the assisted person as the sole beneficiary of the assistance system.

[0109] An example in the field of driving assistance uses the inferred intention of a driver of another vehicle 2 based on acquired context information, the driver's individual actions, and experiences derived from recorded historical information. For example, a driver of a vehicle 2 located on a highway on-ramp may intend to change lanes in the near future, and in another example, a driver of a vehicle 2 slowing down noticeably before a highway exit may intend to exit the highway.

[0110] When the assistance system of the ego vehicle 1, by monitoring the environment, recognizes that current traffic conditions are impeding the ability of another traffic participant to fulfill such an assumed intention, it provides a modulated directional assistance signal to the assisted person operating the ego vehicle 1, conveying the information that the other vehicle 2 may benefit from a certain behavior of the assisted person. The assisted person can then act accordingly to enable the other traffic participant to fulfill its target, for example by slowing down the ego vehicle 1.

[0111] Fig. 2 illustrates the application of the invention in a road traffic environment. In particular, Fig. 2 serves to illustrate the risk-dependent modulation of spatial sound characteristics in an ego-vehicle 1 operated by a supportee.

[0112] An ego vehicle 1 is traveling along a segment of a road and is shown for three different times t1, t2 and t3. The ego vehicle 1 has a speed v1.

[0113] The musical notes shown inside the ego-vehicle 1 represent infotainment sounds 3 output by speakers of the infotainment system of the ego-vehicle 1. The infotainment system is adapted to output sounds via speakers arranged in a prescribed manner inside the ego-vehicle 1, such that the assisted person driving the ego-vehicle 1 from the driver's seat perceives sounds emanating from a particular direction. This effect can be achieved by arranging multiple speakers around the interior of the ego-vehicle 1 and adapting the sound streams of the multiple specific speakers in the processor accordingly.

[0114] At time t1, processing of data in the environment of the vehicle 1 acquired by the assistance system 20 reveals that no other objects or events have been determined that are associated with a high degree of relevance, in particular those that represent a risk to the vehicle 1 which is moving smoothly.

[0115] Fast forward to time t2, the ego-vehicle 1 is traveling at speed v1 along a road segment. Driving in front of the ego-vehicle 1 is another vehicle 2 with speed v2. At time t2, the vehicle 2 is at a distance d1 in front of the ego-vehicle 1 in the driving direction. The speed v2 is less than the speed v1. The assistance system 20, for example a driving assistance system implementing the functions of a collision mitigation and braking system, predicts future potential relevance events of the ego-vehicle 1 accordingly. In the surrounding environment shown in Fig. 2, the assistance system 20 predicts a possible collision of the ego-vehicle 1 with the vehicle 2 in front of the ego-vehicle 1 in the driving direction. The assistance system determines a relevance measure, in particular a risk associated with the determined possible collision.

[0116] At time t3, vehicle 2 is at a distance d2 in front of the ego-vehicle 1 in the driving direction. The speed v2 is still less than the speed v1. The distance d2 between the ego-vehicle 1 and vehicle 2 has decreased from the distance d1 at t2 to the distance d1 at t3. In response, the assistance system 20 predicts a potential collision event as a future relevant event for the ego-vehicle 1. At t3, the risk associated with the predicted collision event is increased compared to the risk of the predicted collision event at t2.

[0117] FIG. 2 illustrates risk-related sounds using a stylized sound frequency spectrum bar 4.

[0118] The risk associated with the predicted frontal collision increases from time t1, beyond time t2, to time t3. As the risk associated with the predicted collision increases from t2 to t3, the apparent location of the infotainment sound 3 perceived by the assisted person is shifted toward the rear of the host vehicle 1, as shown in FIG. 2, when compared to the infotainment sound 3 from t1, t2, and t3.

[0119] Additionally or alternatively, the assistance system 20 generates and outputs a sound representative of the risk associated with the predicted crash event, as if it were emanating from a direction ahead of the ego-vehicle 1. At time t2, the sound 4 spreads spatially over an arc in front of the ego-vehicle 1 with a range 5.1 of the sound frequency spectrum bar 4 representing the sound 4. From t2 to t3, the sound represented by the sound frequency spectrum bar 4 gradually concentrates into a narrower range 5.2 of directions, thus indicating a converging direction from the ego-vehicle 1 towards the potential crash event ahead. Additionally or alternatively, the sound represented by the sound frequency spectrum bar 4 is further amplified as indicated by the increasing height of the sound frequency spectrum bar 4 from time 2 to time 3, as the determined risk also increases from time t2 to t3.

[0120] Thus, the assisted person located in the driver's seat of the host vehicle 1 will perceive the infotainment sound output by the infotainment system with reduced saliency or presence in the direction toward the front of the host vehicle 1 from t1, past t2, to t3. At the same time, the risk-related sound represented by the sound frequency spectrum bar 4 output by the infotainment system is output with increasing saliency (or presence and focus) in the direction toward the front of the host vehicle 1 from t2, past t2, to t3. The modified sound distribution in the host vehicle 1 from t1, past t2, to t3 attracts the attention of the assisted person with increasing saliency toward the front of the host vehicle 1, thereby enabling the assisted person to behave appropriately to mitigate the predicted collision by appropriate action. For example, the assisted person can initiate a lane change or reduce the speed v1 of the host vehicle 1.

[0121] Fig. 3 shows a risk-dependent stimulation modulation curve according to one embodiment. Fig. 3 shows that the assistance system 20 can behave asymmetrically in terms of the way the stimulation is modulated in response to an increase in risk on the one hand and a decrease in risk by the modulation signal on the other hand. In this context, the relevance measure describes the risk for a particular predicted behavior of an object or event in a dynamic environment and is denoted as such for the following discussion.

[0122] In the diagram of Fig. 3, the x-axis (abscissa) shows the relevance measurement of the predicted behavior of an object or event. The x-axis of Fig. 3 in particular shows the determined value of the risk. On the y-axis of Fig. 3 the value of the modulation intensity is shown. The modulation intensity can be interpreted as a measurement of the value of the stimulus generated and output by the assistance system 20.

[0123] In Fig. 3, curve 7 describes the intensity of the stimulation modulation as a function of risk, for risk that increases over time. The indicated intensity of the stimulation modulation according to curve 7 increases with a constant slope in the case of increasing risk.

[0124] FIG. 3 further shows that the curve 7 describes the intensity of the stimulation modulation as a function of risk, for a risk that decreases over time. The depicted intensity of the stimulation modulation according to the curve 7 decreases with a constant slope in the case of a decrease in risk. Preferably, the slope of the decrease in risk has a smaller absolute value than the slope of the curve for an increase in risk. Thus, the assistance system 20 behaves asymmetrically in terms of the manner in which at least one directional stimulation signal is modulated in response to an increase and a decrease in risk, respectively.

[0125] It should be noted that, as discussed with FIG. 4, the relationship between modulation strength and risk is not limited to a proportional dependence.

[0126] The asymmetrical curves 7, 8 of the modulation intensity have the effect that the modulation signal in response to an increase in risk results in a rapid increase in the intensity of the stimulation of the supported person by the at least one directional stimulation signal in order to support the supported person's perception as early as possible, whereas the decrease in the intensity of the at least one modulated directional stimulation signal generated with the modulation signal in response to a decrease in risk follows the decrease in risk with a lower slope.

[0127] The effect of this approach of the asymmetric behavior of the assistance system 20 for risk increase and risk decrease allows the assisted person to continue to perceive the current development of the scene in the environment and to ensure that the risk in the environment has been properly resolved. The assisted driver may leave other immediate potential risks unfolding while focusing on resolving one issue. By the time the first risk is resolved, such secondary risks may become more severe and require immediate attention. The assisted person, such as a driver, can operate under such assumptions and thus maintain a higher level of alertness towards the environment after the initial risk is resolved. The asymmetry of the assistance system 20 in handling risk increase and decrease allows the assisted person to assess the increasing risk as early as possible and to pay his / her attention for a period of time to ensure that the unfolding scene situation is safe again.

[0128] Alternatively or additionally, the slope of the curve 8 and thus the speed of recovery of the at least one directional stimulus signal to an unmodulated value can depend on the previous or initial risk level before the risk level starts to decrease. The higher the determined initial value of the risk before the risk level starts to decrease, the slower the return to the at least one unmodulated directional stimulus signal, e.g. an unmodulated infotainment system sound output by the infotainment system 23 of the host vehicle 1. In this context, FIG. 4 shows the relationship between the modulation strength and the risk of reversal of the stimulus modulation in response to a risk reduction as a function of the previous risk level (initial risk level).

[0129] FIG. 4 illustrates risk-dependent stimulus modulation reversal according to one embodiment.

[0130] In Fig. 4, the x-axis represents time. On the y-axis arranged on the left, the values ​​of the modulation intensity of the modulating signal are shown. The y-axis arranged on the right side of Fig. 4 allows the value of the risk to be read out.

[0131] FIG. 4 shows the modulation intensity curves 9, 10, 11, 12 at solid lines 9.1, 10.1, 11.1, 12.1 for risk reversal and the gradual drop in risk at dashed lines 9.2, 10.2, 11.2, 12.2 for four different initial risk levels 9.3, 10.3, 11.3 and 12.3 over time. The greater the initial risk level 9.3, 10.3, 11.3, 12.3 before risk reduction (risk reduction), the lower the absolute value of the slope of the reversal in the first stimulation modulation. For smaller initial risks, the reversal of modulation intensity occurs at a faster rate, approaching the rate of the corresponding reduction in risk.

[0132] Correspondingly, the modulation strength that is gradually decreased due to the decrease in risk may decrease slower than the actual decrease required by the risk. If the risk ends its decrease and even starts to increase again, the assistance system can implement, for example, one of the following strategies:

[0133] In a first scenario, the reduction in modulation intensity may be terminated upon the determination of an increase in risk until the determined increase in risk begins to align with the corresponding modulation intensity at the level at which the reduction in modulation intensity ended. When the modulation intensity corresponding to the increase in risk reaches and exceeds the level of modulation intensity at which the reduction in modulation intensity ended, the modulation intensity begins to increase again following the determined increase in risk ("wait until risk catches up with modulation intensity").

[0134] In a second alternative scenario, the actual risk stops its increase. The second scenario can also include the actual risk starting to decrease again after having previously ceased decreasing. In this scenario, the decrease in modulation intensity can proceed to continue until it is aligned with the corresponding modulation intensity of the constant actual risk, or until it reaches a modulation intensity value of 0 (zero), in which case the determined actual risk disappears completely.

[0135] 5 provides an overview of the structural elements of the assistance system 20 according to one embodiment. In the context of FIG. 5, the assisted person is the driver of the ego vehicle 1.

[0136] In particular, Fig. 5 displays the main units (modules, means) of the driver assistance system modulating the saliency of the sounds output by the infotainment system 23 inside the ego-vehicle 1 equipped with the driver assistance system 20. The sounds output by the infotainment system 23 include infotainment sounds 3, e.g. at least one of a radio audio stream, a music stream, and an audio stream provided by a navigation system not shown in Fig. 5. The sounds output by the infotainment system 23 further include multi-modal stimuli related to traffic elements in the environment outside the ego-vehicle 1.

[0137] The driver assistance system 20 includes an acquisition unit 22. The driver assistance system 20 acquires data on the environment of the host vehicle 1 via the acquisition unit 22. This environment is a dynamic environment characterized by changing traffic scenarios in which the host vehicle 1 and the driver of the host vehicle 1 corresponding to the assisted person have to navigate.

[0138] The acquired data may include sensor data acquired by at least one sensor (sensor device) mounted on the host vehicle 1 and sensing the environment of the host vehicle 1. The sensors may include at least one of active and passive sensors, e.g., camera sensor 21.1, radar sensor 21.2, lidar sensor 21.2, acoustic sensor (microphone), ultrasonic sensor, temperature sensor, in any combination.

[0139] The acquisition unit 22 may further acquire data including sensor data acquired by at least one sensor (sensor device) mounted on the host vehicle 1 and sensing the interior of the host vehicle 1. The sensors acquiring the sensor data from the interior of the host vehicle 1 may include at least one of active and passive sensors, e.g. a camera sensor 21.4, an acoustic sensor 21.5 (microphone), a temperature sensor, in any combination.

[0140] The acquiring unit 22 may further acquire data including map data from a map data provider 21.6.

[0141] Additionally or alternatively, the acquisition unit 22 acquires data including traffic data from a traffic data provider 21.7.

[0142] Additionally or alternatively, the acquisition unit 22 acquires data including x-to-vehicle data via a vehicle communication module 21.8. The vehicle communication module 21.8 can, for example, enable acquisition from other vehicles 2 in the environment of the ego-vehicle 1 (vehicle-to-vehicle communication, but also from traffic infrastructure devices capable of infrastructure-to-vehicle communication). The vehicle communication module 21.8 can include at least one wireless communication module operating according to a particular cellular communication standard, such as, for example, LTE, LTE Advanced, and 5G.

[0143] Additionally or alternatively, the acquisition unit 22 acquires sound data including at least one sound data stream from the infotainment system 23 .

[0144] The assistance system 20 of Fig. 5 comprises a processor 25 (electronic control unit, electronic data processing unit) which may comprise a number of processing units located in the ego vehicle 1 or located spatially separated from the ego vehicle 1 and communicating with the ego vehicle 1, for example via a communication unit 21.8, implementing a distributed processing architecture.

[0145] The processor 25 stores data in and reads data from the memory unit 24. The memory 24 in Fig. 5 represents a data storage function for recording application data, e.g. an audio data stream in the form of an audio file, or the program code of the driver assistance system 20, or data generated and stored during the execution of the program code, and thus also functions as a RAM or ROM memory.

[0146] The assistance system 20 may further include a user interface 27. The user interface 27 allows the assistance recipient to provide input data to the data assistance system 20 and receive data output from the assistance system 20. The user interface 27 may include at least one display that simultaneously serves as a data input means, for example a touch screen. As an alternative to the automatic-only assistance method, via the user interface 27 the assistance recipient may manually indicate a degree of confidence in his or her ability to handle a current scenario in a dynamic environment. Alternatively, the assistance recipient may explicitly request assistance from the assistance system 20.

[0147] In particular, the processor 25 is configured to carry out the steps of the assistance method, as discussed with reference to FIG.

[0148] The processor 25 provides output data to an output driver 28. The output data provided to the output driver includes at least one modulated directional stimulus signal.

[0149] The output driver 28 outputs at least one modulated directional stimulus signal to a number of actuators 26.1, 26.2, 26.3 for outputting the at least one modulated directional stimulus signal to the assisted person.

[0150] Additionally or alternatively, the output driver 28 provides at least one modulated directional stimulus signal to the infotainment system 23. The infotainment system 23 may include multiple loudspeakers, not explicitly shown in FIG. 5, arranged inside the host vehicle 1, in particular in the passenger compartment. Using the loudspeakers of the infotainment system 23 requires a set of loudspeakers capable of providing the assisted person with acoustic stimulus signals that the assisted person can distinguish with respect to spatial origin and thus the direction in which the acoustic stimulus arrives at the assisted person's position. Thus, the present invention can expand the functionality of the vehicle infotainment system to fulfill or complement the functionality of the assistance system.

[0151] Actuators 26.1, 26.2, 26.3 may include an arrangement of actuators positioned and adapted to provide directional stimulation to the recipient using one or even more sensory modalities such as sound, airflow, the recipient's sense of touch, etc., to name a few possible implementations of actuators 26.1, 26.2, 26.3.

[0152] In the event of an increased risk of a head-on collision caused by a sudden deceleration of another vehicle 2 in front of the own vehicle 1, the data obtained from the on-board sensors 21.1, 21.2, 21.3 providing data on monitoring distances d1, d2 and estimating the speed v2 of surrounding objects, in particular the vehicle 2, are determined (classified) by the processor 25 as being relevant to the assisted person performing the task of operating the own vehicle (1).

[0153] The processor 25 may perform classifications based on the data obtained, including comparing distance-dependent and relative speed-dependent risk measures, such as headway, to thresholds.

[0154] The processor 25 can supplement the risk determined from the classification with information from other sources, for example map data, traffic density data, local driving profile statistics, and further data obtained via x-to-vehicle communications.

[0155] Data containing local driving profile statistics can convey information such as, for example, "how fast do vehicles typically drive here?"

[0156] Based on the classification, the processor 25 modulates a parameter of at least one directional stimulus signal, e.g., representative of a salience-related characteristic of the infotainment sound, whereby the modulation increases continuously with the value (magnitude) of the determined risk measure.

[0157] The modulation performed by the processor 25 may include applying a spectral filter, such as a low-pass / high-cut filter, to the at least one directional stimulus signal. A variable filter cut-off frequency or a mix of filtered and unfiltered sound components of the at least one directional stimulus signal may be set as a function of the determined risk measure. Thus, in the case of a low-pass filter, high frequency components of the infotainment sound components from the front direction to the assisted person are gradually removed as the risk measure associated with a frontal collision increases.

[0158] Additionally or alternatively, modulating the at least one directional stimulus signal includes at least one of limiting and equalizing the amplification range of the infotainment sound spectrum of the infotainment signal. This modulation takes into account that loudness spikes also affect the salience of the sound. The undesirable effect of attention-grabbing sounds, such as the ringing of a mobile phone, on the attention of the assisted person is effectively reduced.

[0159] Additionally or alternatively, the modulation includes adapting the apparent source of the infotainment sound such that it gradually shifts toward a direction opposite to a direction of the determined at least one behavior of an object or event in the environment as the value of the associated risk measure increases. If the predicted event is a frontal collision, the infotainment sound can be shifted toward the rear of the host vehicle 1.

[0160] Additionally or alternatively, the shift in the apparent source of the infotainment sound may be perpendicular to the plane of motion in which the host vehicle 1 is moving, for example moving downward or upward relative to the plane of motion depending on a change in the associated risk measurement.

[0161] Additionally, the processor 25 can generate or extend stimuli that induce the assisted person to focus his / her attention in a relevant direction in the environment. The processor 25 can generate these stimuli simultaneously or in close succession and output them via the actuators 26.1, 26.2, 26.3 and / or the infotainment system 23.

[0162] The sensors 21.1, 21.2, 21.3 may include microphones located outside the ego-vehicle 1 to capture environmental sounds from the external environment, in particular sounds originating from the direction of the predicted behavior of an object or a predicted event, and thus originating from the spatial direction of the risk from the viewpoint of the ego-vehicle. The processor 25 may control the reproduction of this sound using speakers of the infotainment system 23 inside the ego-vehicle 1. In particular, the processor 25 may control the reproduction of the sound utilizing frequency components of the at least one modulated directional stimulus signal that are filtered out for the infotainment sound when modulating the at least one directional stimulus signal. The processor 25 may adjust the relative volume of the individual speakers of the infotainment system 23 inside the ego-vehicle 1 to generate virtual sound sources perceived by the assisted person that correspond to the direction of each risk.

[0163] Furthermore, the processor 25 can control at least one modulated directional stimulus signal depending on the determined risk level with a varying accuracy of the sound perceived by the support recipient. By varying the accuracy of the sound perceived by the support recipient, it can be ensured that the apparent position of the output sound is more precisely aligned with the direction of the predicted risk. The modulation signal can modulate the at least one directional stimulus signal such that the greater the risk measured or classified, the smaller the angular region of the output sound is concentrated in the direction of the predicted risk. Alternatively, if the risk measured or classified is smaller, the output sound is concentrated in a wider angular region in the direction of the predicted risk. By making this accuracy of the direction of the output sound dependent, for example, on the value in the relevance measure of the determined risk, rather than always on the highest possible level, negative satisfaction or attention tunneling effects on the support recipient are avoided. Such negative satisfaction or attention tunneling effects would result in early restriction of the support recipient's attention to a narrow region already at low values ​​of the relevance measure or risk level.

[0164] Additionally or alternatively, the processor 25 may control the modulation of the at least one directional stimulation signal such that the output volume of the at least one output of the modulated directional stimulation signal increases continuously with increasing relevance measure or risk level until it reaches a predetermined maximum value.

[0165] The processor 25 of the assistance system 20 may include an ambient sound expansion module adapted to generate artificial sounds. The artificial sounds may be stored in the memory 24 as artificial sound files. The generated artificial sound files are semantically associated with a given behavior of an object or event and thus with a particular source or source category of risk. Generating and storing artificial sound files makes it possible to overcome the problem that other traffic participants may not necessarily generate sounds that the sensors 21.1, 21.2, 21.3 can capture, especially over large distances in potentially noisy environments.

[0166] Generating the artificial sound file may include in a first step transferring data on a particular risk source acquired by the sensors 21.1, 21.2, 21.3 or obtained using x-to-vehicle communication to a classification module, thereby associating the particular risk source with a particular risk category, or both. For each risk category, the assistance system 20 may play a matching artificial sound, such as the engine sound of the vehicle 2, or the footsteps of a pedestrian. The artificial sound file may form part of a set of artificial sound files optimized to include attention-capturing elements whose salience increases with increasing risk. The artificial capturing elements may include, for example, the onset of a loud sound. Another example of an attention-capturing element is a sound characteristic that contrasts with the sound characteristics of a less relevant sound, for example an emphasis on the infotainment sound output by the infotainment system 23. The sound characteristics may include sound parameters such as, for example, pitch, sound quality or spatial location. The artificial sound file selected by the processor 25 may be periodically replayed while a risk is determined to be present. The loud onset of the artificial sound file can be varied from iteration to iteration depending on whether the relevance measure or risk value increases (loudness increase), decreases (no onset amplification), or remains constant (no amplification or iteratively reduced onset amplification).

[0167] With respect to the recorded ambient sounds, artificial sounds are generated and output in at least one modulated directional stimulus signal, directed in the direction of the respective risk source, to generate an illusion for the assisted person at a volume level determined based on the risk measurement, for example proportional to the determined risk level.

[0168] The assistance system 20 may include one or more actuators 26.1, 26.2, 26.3. The actuators 26.1, 26.2, 26.3 may be adapted to provide visual, tactile, vestibular or olfactory stimuli to the assisted person, thus assisting the assisted person to distinguish between risk-related sounds and other, for example infotainment-related, sounds. By using the stimuli, in particular directional stimuli, output by the actuators 26.1, 26.2, 26.3, the assistance system may also exploit the multimodal sensory processing abilities of humans. Thus, the processor 25 may augment the output of at least one modulated directional stimulus signal conveying information about the predicted behavior of an object or a predicted event via the infotainment system 23 by using visual, tactile, vestibular or olfactory stimuli based on the modulated signal.

[0169] The actuators 26.1, 26.2, 26.3 may include haptic actuators integrated in at least one of the steering wheel, the seat and the seat belt of the ego-vehicle 1 and controlled by the processor 25 via an output driver 28 capable of enhancing the auditory perception of the respective stimulation signals output by the actuators 26.1, 26.2, 26.3. The processor 25 may perform the conversion of a subset of the audio frequency spectrum into vibrations. Examples of this kind of haptic actuators are the Fujitsu Ontenna or the Neosensory Buzz. Furthermore, the characteristics of the haptic stimulation by the haptic actuators 26.1, 26.2, 26.3, such as the intensity of the stimulation, may be modulated according to the respective value of the relevance measure or the risk level.

[0170] Alternatively or additionally, the actuators 26.1, 26.2, 26.3 may include a ventilation system of the host vehicle 1. The processor 25 may control the ventilation system to output an airflow toward the assisted person originating from a direction toward the predicted behavior or predicted event of the object with an associated relevance measure or risk level of the airflow. The strength of the airflow may represent a characteristic parameter that is modulated based on the respective value of the relevance measure.

[0171] Alternatively or additionally, the actuators 26.1, 26.2, 26.3 may comprise devices adapted to disperse odor molecules within the ego-vehicle 1 having properties that increase attention in high-risk situations.

[0172] Alternatively or in addition, actuators 26.1, 26.2, 26.3 may include visual elements in the dashboard, a head-up display, a light strip on the center console of the host vehicle 1, which may further point or highlight a direction towards predicted risk elements in the host vehicle's environment.

[0173] Alternatively or in addition, actuators 26.1, 26.2, 26.3 may augment at least one modulated directional stimulation signal by outputting a series of words, such as the name of the person being assisted, or other semantically related sound signals, to increase the attention of the person being assisted.

[0174] The sensor that acquires the sensor data from inside the vehicle 1 can include at least one of active and passive sensors, such as a camera sensor 21.4, an acoustic sensor 21.5 (microphone). This allows the processor 25 to perform a process that makes the slope of the modulation signal not only depend on the determined value of the relevance measure, but also, optionally, on information about the support recipient. In particular, the assistance system 20 can determine the attention state of the support recipient, the alertness of the support recipient, and the attention history of the support recipient.

[0175] The camera sensor 21.4 can provide image data that allows monitoring at least one of the head and eye movements of the supported person. The processor 25 can be configured to estimate which aspects of the current scene in the environment the supported person perceives. The processor 25 determines the gaze pattern and fixation duration of the supported person based on the image data and refines probability estimates of the perception of different elements in the environment, in particular the predicted behavior of objects and predicted events involving these objects.

[0176] Sensors acquiring sensor data from inside the vehicle 1 may include sensors for determining pupil dilation, heart rate and other physiological measurements such as skin conductance that can estimate the alertness of the person being assisted.

[0177] The processor 25 can use the determined information on the attention of the support recipient and the alertness of the support recipient to adjust the assistance provided to the support recipient to the estimated actual demand according to the determined scene context and the stored data on the human behavior pattern. The assistance system 20 can further increase the stimulation modulation by the modulation signal whenever the assistance system 20 determines the alertness of the support recipient or whenever the determined attention of the support recipient to the task is below a threshold corresponding to a predetermined minimum attention or alertness level required.

[0178] The determined attention level or alertness of the support recipient can further be used by the support system 20 to adapt the slope of the modulation signal after the relevance measurement or risk level is significantly reduced. For example, if the determined alertness of the support recipient reaches a normal level well before the reduction of the modulation signal after the risk is eliminated, the slope of the modulation signal, and therefore the slope of the recovery, can be increased.

[0179] Alternatively, the assistance system 20 can determine the individual needs of the support recipient to require maintaining a high modulation level of the modulation signal unless the support recipient shows a sufficiently high level of alertness. This process results in an enhanced safety margin in the support for the support recipient.

[0180] The assistance system 20 may include a user interface 27. The user interface 27 may allow manual modulation by the assisted person, for example, by using button presses or voice commands with the user interface 27.

[0181] The description of the embodiment relies on application scenarios in the field of automatic assistance systems. Driver assistance systems present one particularly useful application field of the present invention. The present invention is applicable to other mobility-related scenarios, where multiple elements of the scenario in the environment differ in the associated relevance of the main target of the assisted person, but are comparable to the attention of the assisted person. By applying the assistance method according to the attached claims, safety is increased by improving the ability of the assisted person to perceive the environment when considered necessary, without hindering or impairing the autonomy of the assisted person. This increases the acceptance of the assistance method by the assisted person.

[0182] The application areas therefore include not only driver assistance in road traffic, but also navigator assistance in maritime or pilot assistance in air transport scenarios.

[0183] A further advantageous application scenario of the present invention relates to assisting in the remote control of mobile vehicles in ground-based, air-based or sea-based scenarios.

Claims

1. 1. A method for assisting a assisted person to perform a task in a dynamic environment, comprising: acquiring data relating to said dynamic environment; predicting at least one behavior of objects or events in the dynamic environment based on the acquired data; determining a relevance measure of the at least one behavior of the object or event based on the prediction; generating a modulated signal, the modulated signal varying at least one signal parameter as a continuous function of the determined relevance; modulating at least one directional stimulus signal with the modulation signal, the at least one modulated directional stimulus signal further depending on a direction of the at least one predicted behavior of the object or event relative to the assisted person; The method comprises: detecting a stimulus perceivable by the assisted person and including the detected stimulus in the at least one directional stimulus signal prior to modulating the at least one directional stimulus signal; The continuous function continuously reduces the salience of the at least one directional stimulus signal perceived by the supportee in the case of a decrease in the determined relevance; a value of the slope of the continuous function in the case of a decrease in the salience of the at least one directional stimulus signal over time is less than the value of the slope of the continuous function in the case of an increase in the salience of the at least one directional stimulus signal over time; outputting the at least one modulated directional stimulus signal to the assisted person; A method comprising:

2. The method of claim 1 , further comprising determining a status of the recipient and adapting risk-dependent stimulation modulation based on the determined status.

3. The method of claim 2 , wherein the determined state of the supported person includes at least one of an attention state of the supported person, an attention history of the supported person, and a level of alertness of the supported person.

4. The method of claim 1 , wherein the at least one directional stimulus signal is an artificial stimulus signal comprising a recorded sound signal.

5. The method of claim 1 , wherein the at least one signal parameter is one of a stimulus amplitude, a stimulus frequency, a stimulus repetition rate, and a perceived signal direction.

6. The method of claim 1 , wherein the at least one directional stimulus signal includes at least one of an audio signal stream, and the at least one signal parameter is one of an audio volume and an audio signal frequency band.

7. The continuous function continuously increases the salience of the at least one directional stimulus signal perceived by the supportee when the determined relevance increases, or The method of claim 1 , wherein the continuous function continuously decreases the perceived distance to the at least one predicted behavior of the object or event in the at least one directional stimulus signal perceived by the assisted person when the determined relevance increases.

8. The method of claim 1 , wherein the modulation of the at least one directional stimulus signal in the case of a decrease in relevance comprises increasing a value of a slope of the modulation signal with the decrease in relevance.

9. The method of claim 1 , wherein the modulation of the at least one directional stimulus signal in the case of a decrease in the relevance comprises a smaller value of the slope of the modulation signal the higher the initial value of the relevance.

10. the determined relevance is a function of at least one task goal; The method of claim 1 , wherein the at least one task goal includes at least one of traffic safety, comfort of the assisted person, a status of the assisted person, and a task goal of another person.

11. The method of claim 1 , wherein the determined relevance is a risk metric associated with the predicted at least one behavior of the object or the predicted event.

12. A support system for supporting a supportee to perform a task in a dynamic environment, comprising: an acquisition interface configured to acquire data related to the dynamic environment; a processor; The processor predicts at least one behavior of an object or event in the dynamic environment based on the acquired data; determining a relevance measure of the at least one behavior of the object or event based on the prediction; generating a modulated signal, the modulated signal varying at least one signal parameter as a continuous function of the determined relevance; modulating at least one directional stimulus signal with the modulation signal, the at least one modulated directional stimulus signal being further dependent on a direction of the at least one predicted behavior of the object or event relative to the assisted person; It is configured as follows: The processor is configured to include a stimulus perceivable by the assisted person in the at least one directional stimulus signal before modulating the at least one directional stimulus signal; The continuous function continuously reduces the salience of the at least one directional stimulus signal perceived by the supportee in the case of a decrease in the determined relevance; a value of the slope of the continuous function in the case of a decrease in the salience of the at least one directional stimulus signal over time is less than the value of the slope of the continuous function in the case of an increase in the salience of the at least one directional stimulus signal over time; The assistance system further comprises at least one actuator configured to output the at least one modulated directional stimulus signal to the assistance recipient; Support system.

13. A computer program comprising program code means for carrying out the method according to any one of claims 1 to 11 when said program is executed by an assistance system according to claim 12.

Images